Kafka

Apache Kakfa is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

helm repo add ...
helm repo update

Install Kakfa.

Plane

plane up datalayer-kafka

Helm

cat << 'EOF' > /tmp/values.yaml
...
EOF
export RELEASE=datalayer-kafka
export NAMESPACE=datalayer-kafka
helm upgrade \
    --install $RELEASE \
    apache/kafka \
    --create-namespace \
    --namespace $NAMESPACE \
    --values /tmp/values.yaml \
    --timeout 10m

Check the availability of the Kafka Pods.

kubectl get pods -n datalayer-kafka

# NAME                                 READY   STATUS      RESTARTS   AGE
...

To interact with Kafka, you first need to port-forward.

plane pf-kafka

To connect from you host to the Kafka service, add the following entries in your /etc/hosts file.

# /etc/hosts
...

You can check the connection to Kafka with the following Python code (you need kafka-client).

# Producer.
...

# Client.
...

Tear down Kafka if needed.

Plane

plane down datalayer-kafka

Helm

export RELEASE=datalayer-kafka
export NAMESPACE=datalayer-kafka
helm delete $RELEASE --namespace $NAMESPACE