Kafka

Apache Kakfa is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Define the pub sub engine as kafka.

DATALAYER_PUB_SUB_ENGINE=kafka

Install Kakfa with the Strimzi Operator.

Plane

plane up datalayer-kafka

Helm

cat << 'EOF' > /tmp/values.yaml
...
EOF
export RELEASE=datalayer-kafka
export NAMESPACE=datalayer-kafka
...

Check the availability of the Kafka Pods.

kubectl get pods -n datalayer-kafka

# NAME                                        READY   STATUS    RESTARTS   AGE
# datalayer-kafka-datalayer-kafka-node-pool-0        1/1     Running   0          60s
# datalayer-kafka-datalayer-kafka-node-pool-1        1/1     Running   0          60s
# datalayer-kafka-entity-operator-76d4886bdc-mpkgr   2/2     Running   0          55s
# datalayer-kafka-kafka-exporter-6bb957896f-9lvns    1/1     Running   0          34s
# strimzi-cluster-operator-597fd4ff5b-d579v          1/1     Running   0          79s

To interact with Kafka, you first need to port-forward.

plane pf-kafka

You can check the connection to Kafka with the following Python code (you need kafka-client).

# Producer.
...

# Client.
...

Tear down Kafka if needed.

Plane

plane down datalayer-kafka

Helm

export RELEASE=datalayer-kafka
export NAMESPACE=datalayer-kafka
helm delete $RELEASE --namespace $NAMESPACE