Skip to main content
Below is an example of the installation process on bare-metal nodes in AWS

​
Terraform

​
Download and install Terraform

​
Download and unzip the examples scripts

<<EXAMPLES_LINK>>

​
Acquire AWS IAM credentials and create a credentials file

Visit the AWS IAM console and create a access key and secret key and save these values. NOTE Replace <Your AWS ACCESS KEY ID> and <Your AWS SECRET ACCESS KEY> with the values acquired from the AWS IAM console.

​
Load your AWS IAM credentials

NOTE - Add -migrate-state if moving state from local to s3

​
Initialize the rke2 Terraform module

​
Issue a Terraform apply command to bring the stack up

From within the terraform directory located in the git repository

​
Ansible

​
Run Ansible to provision RKE2 software

Go to Ansible directory
Install Ansible Galaxy role lablabs.rke2
Install Ansible Galaxy collection kubernetes-core
Verify that the nodes are ready
Configure NVMEs
Run RKE2 playbook ../ansible/rke2_playbook
Output:

​
Prepare cert manager and CAs

Install and configure cert manager and create certificate
Output:
Local storage
Install local storage driver
Storage class
Contents:

​
MinIO prep

Install operator
Create namespace
Install directpv plugin
Output:
Create directpv volumes, if necessary
Output:
  • If response is “No drives are available to initialize” then don’t do the init! *

​
Install MinIO operator

​
Install MinIO tenant

Note: minio-values.yaml is a modified version of https://docs.min.io/community/minio-object-store/reference/tenant-chart-values.html

​
Apply certs for MinIO

​
Configure MinIO bucket and key

  • Make sure you are also port forwarding via ssh so you can get to the UI from your local host.
  • Log in using the username and password from the provisioning host’s environemnt variables:
    • MINIO_ACCESS_KEY
    • MINIO_SECRET_KEY
  • Create bucket (https://localhost:9443/browser/add-bucket) Use value of environment variable S3_BUCKET_NAME
  • Create access key (https://localhost:9443/access-keys/new-account) Use values of environment variables:
    • S3_ACCESS_KEY_ID
    • S3_SECRET_ACCESS_KEY
Label nodes in prep for ClickHouse Keeper and Server pods

​
ClickHouse Installation Notes on Bare metal

This document outlines the steps and commands for installing ClickHouse on a Bare-metal deployment. It is assumed that you have a Kubernetes cluster built and access to a container registry.

​
Helm Chart Configuration (clickhouse-values.yaml)

This section outlines the structure and key parameters for your clickhouse values file, which will be used by Helm.

​
Quick deploy

Scripts can be found here: <<EXAMPLES_LINK>>

​
Details

1. Helm Variables Script (1_set-helm-vars.sh)
This script defines all the environment variables needed for the Helm deployment.
2. Install ClickHouse Operator (2_helm_install_operator.sh)
This script installs or upgrades the ClickHouse Operator using Helm, configuring it with the necessary image repository, feature flags, and availability zones.
If error such as this, then you need to log into the Helm registry:
3. Install ClickHouse Cluster (3_helm_install_clickhouse.sh)
Optional: If you don’t have a local storage driver and plan to use one, here is an example of how to set one up with nvme.
  • Install local storage driver
  • Storage class
Contents:
Run 3_helm_install_clickhouse.sh
This script automates the Helm installation/upgrade process for the main ClickHouse cluster, sourcing variables and preparing the values file.
4. Fix Keeper StatefulSet (4_fix_keeper_sts.sh)
This script is used to patch the Kubernetes secret and the StatefulSet for the ClickHouse Keeper, which can be crucial for resolving certain access issues related to image pull secrets or environment variables. It ensures the keeper pods can correctly access necessary resources.
sts_keeper_patch_template.yaml
This YAML template is used by the 5_fix_keeper_sts.sh script to patch the ClickHouse Keeper StatefulSet. It adds imagePullSecrets and injects AWS access key and secret key as environment variables into the keeper container.
5. Fix Server PVs (5_fix_server_pvs.sh)
This script addresses issues where ClickHouse server pods might be stuck in a 0/1 ready state due to Persistent Volume (PV) topology label mismatches. It updates deprecated failure-domain.beta.kubernetes.io/ labels to topology.kubernetes.io/region and topology.kubernetes.io/zone, ensuring proper scheduling and functionality. It also attempts to apply image pull secrets and deletes server pods to trigger a re-creation with the correct configurations.
sts_server_patch.sh
This script is designed to patch the StatefulSets of ClickHouse server pods. It ensures that the necessary image pull secret is applied to the server StatefulSets, which is critical for pulling container images from private registries like IBM Cloud Container Registry. It also includes error handling and informs the user that a pod restart is needed for changes to take effect.
6. Connect to Clickhouse
Once the pods are up and healthy, you can validate connectivity to ClickHouse via port forwarding. In a production environment you should use a load-balancer.
You can configure CA certs or other TLS settings via the file clickhouse-client.yml
Here’s a sample command to connect and test:
and run a simple query:
You should see output like this:

​
Appendix - Troubleshooting

Missing all-icr-io secret in clickhouse-operator-system namespace.
Missing local path storage CRDs
If Install local storage driver is skipped, when provisioning the Keepers you’ll get a message visible via describe like this:
Digging into the PVC you’ll see this:
Describing a keeper shows this:
Keeper images won’t install due to operator not populating needed STS configuration
If you see this:
Describing Keeper shows this:
Patch STS with 4_fix_keeper_sts.sh. The main thing is to populate the secret in the clickhouse-operator-system and ns-$CLUSTER_NAME namespaces. For example:
  • kubectl create secret docker-registry all-icr-io —docker-server=359774249275.dkr.ecr.us-east-2.amazonaws.com —docker-username=AWS —docker-password=$(aws ecr get-login-password) -n clickhouse-operator-system
  • kubectl create secret docker-registry all-icr-io —docker-server=359774249275.dkr.ecr.us-east-2.amazonaws.com —docker-username=AWS —docker-password=(awsecrget−login−password)−nns−(aws ecr get-login-password) -n ns-CLUSTER_NAME Note: If you have previously run this and time has passed then your token may have expired, so this needs to be re-run after deleting the secrets since they can’t be updated in place.
Keeper pods CrashLoopBackOffing
Check pod logs. Probably MinIO bucket and key wasn’t configured.
Keeper pods launch and are running/healthy but server pods haven’t spawned
Possibly the operator is not healthy:
Describing the pod shows this:
Attempt to delete operator pod to have it respawn:
Issue - Using operator 1.13538.1 causes this error. Rolling back to 1.13500.1 works fine.
If server pods are pending for a few minutes then check the operator logs, it may be a problem with your sc local-nvme-sc configuration.
kubectl logs -l app=clickhouse-operator -n clickhouse-operator-system |jq .
More evidence:
Debugging CrashLoopBackOff Errors
If a pod is in CrashLoopBackOff state, describe the pod to get error messages and events:
Example describe output indicating issues:
If you see errors, check the logs of the problematic pod:
S3 Access Denied Errors
If you encounter Access Denied errors related to S3 in the container logs (e.g., for keepers), you might need to shim the S3 credentials into the pod’s environment variables. Instead of kubectl edit, you can use kubectl patch with a YAML file. Sample error:
Create a file named sts_keeper_patch.yaml with the following content, replacing YOUR_KEY and YOUR_SECRET with your actual AWS credentials:
Then apply the patch:
After patching, watch the pods. You might need to delete the problematic pod to force it to respawn with the new configuration:
Copy kubectl Secret to Another Namespace (if needed) If your kubectl secret needs to be copied to another namespace (e.g., from default to clickhouse-operator-system), use the following command. This should be handled already be the scripts.
Remove cached container images on nodes to validate the imagePullSecrets are working, otherwise it only matters the first time it is run on a node.
Run script remove_cached_images.sh