Skip to main content

Services

The following requirements are needed to deploy the Datalayer services.

☸️ A Kubernetes cluster with administrator access.

🐳 Docker images available in a Docker Repository.

🪖 Helm charts available in a OCI Repository.

🫧 Configuration and Environment Variables.

🔒 Secrets available in a Vault.

🐳 Docker Images​

You should have access to a Docker Registry with the Docker Images available.

If needed, you can build your own images and push them to a Docker Registry where you have write access to.

Therefor, you need docker with buildx on your machine.

# Setup for Ubuntu.
sudo apt update
sudo apt install docker.io docker-buildx
sudo usermod -aG docker $USER # Logoff/Login to be able to access the Docker process from your user.

Some Docker registries enforce you to create the repositories in advance. If this is the case, ensure you have the following repositories pre-created.

  • datalayer/iam
  • datalayer/ingress-nginx-controller
  • datalayer/jupyter
  • datalayer/jupyter-python
  • datalayer/manager
  • datalayer/operator
  • datalayer/operator-companion
  • datalayer/spacer
  • datalayer/status

You are now ready to build and push all the Docker Images with the following commands.

# Build and push all the Docker Images.
plane docker-build
plane docker-push

Alternatively, you can have more control building the images of our choice.

# Build and push a selection of Docker Images.
for d in datalayer-iam datalayer-operator datalayer-operator-companion datalayer-jupyter jupyter-python
do
  echo Building Docker Image $d
  cd $PLANE_HOME/etc/dockerfiles/$d
  make build
  echo Pushing Docker Image $d
  cd $PLANE_HOME/etc/dockerfiles/$d
  make push
done

On top of the Datalayer Services, you also need the Docker Images for the Jupyter Kernels. The base jupyter-python Kernel Docker Image is included in the previous step. You can extend the base ones to create custom ones that fit your requirements.

Registry Secret​

for ns in datalayer-api datalayer-jupyter datalayer-router datalayer-nginx datalayer-traefik
do
  echo Deleting reg-creds secret in namespace $ns
  kubectl delete secret reg-creds -n $ns
  echo Creating reg-creds secret in namespace $ns
  kubectl create secret \
    docker-registry reg-creds \
    -n $ns \
    --docker-server=$DATALAYER_DOCKER_REGISTRY_HOST \
    --docker-username=$DATALAYER_DOCKER_REGISTRY_USERNAME \
    --docker-password=$DATALAYER_DOCKER_REGISTRY_PASSWORD
  kubectl get secret reg-creds -n $ns -o jsonpath="{.data.\.dockerconfigjson}"
done

Pre-pull the Docker Images​

Pre-pull the Docker Images on the Kubernetes cluster nodes.

# Prepull for the CPU nodes.
plane k8s-prepull-cpu
kubectl delete daemonset images-prepuller-cpu -n datalayer-jupyter
# Prepull for the GPU CUDA nodes.
plane k8s-prepull-gpu-cuda
kubectl delete daemonset images-prepuller-gpu-cuda -n datalayer-jupyter

Services​

You are now ready to deploy the services one by one with plane as described in the following sections.

Plane is using behind the scene the Helm charts. Once the services are deployed, you can manage them.

# List the Datalayer services.
plane list

The API are available as Swagger OpenAPI specs.