Kafka Kubernetes deployment — Brokers with KRaft (Zookeeperless deployment) (3)

This is a series of articles that explains how to set up a Kafka Cluster in Kubernetes using the Strimzi Kafka operator.

Introduction
Strimzi Operator and Entity Operators
Kafka Deployment with Kraft
Kafka CRD components
Cruise Control
Kafka Bridge Deployment
Kafka Connect Deployment
Kafka Mirror Maker 2
Metrics And Monitoring

Deploying Kafka cluster

After deploying the Strimzi operator we can deploy the Kafka cluster. We are deploying a Kafka cluster with 3 broker nodes with node pools enabled in Kraft mode along with the entity operators, cruise control and Kafka exporter. We will discuss these components in more detail in the coming parts of this article.

Nodepools:

Node pools provide configuration for a set of Kafka nodes. By using node pools, nodes can have different configurations within the same Kafka cluster. For each Kafka node in the node pool, any configuration not defined in the node pool is inherited from the cluster configuration in the Kafka resource. The deployment uses a YAML file to provide the specification to create a KafkaNodePool resource.

To deploy a Kafka cluster in KRaft mode, you must use the KafkaNodePool resources.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: dev-cluster
spec:
  replicas: 3
  roles:
    - broker
    - controller
  storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false

To enable node pool and kraft we need to provide the below configuration to the Kafka resource.

kind: Kafka
metadata:
  name: dev-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:   
  kafka:
    version: 3.6.1
    metadataVersion: 3.6-IV2

KRAFT:

KRaft stands for Kafka Raft Metadata mode, which means that Kafka uses the Raft consensus protocol to manage its metadata instead of relying on ZooKeeper. KRaft mode is more scalable than ZooKeeper mode, allowing Kafka clusters to handle more traffic and data. KRaft mode is faster than ZooKeeper mode, resulting in lower latency and higher throughput.

In the KRaft mode, the Kafka cluster will consist only of Kafka nodes. But they have different roles — either controllers, or brokers, or a combination of both

We can apply the below config for deploying a basic cluster. We will see each component in detail in the next article.

kubectl apply -f kafka.yaml -b ns-kafka

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: dev-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:   
  kafka:
    version: 3.6.1
    metadataVersion: 3.6-IV2
    # The replicas field is required by the Kafka CRD schema while the KafkaNodePools feature gate is in alpha phase.
    # But it will be ignored when Kafka Node Pools are used
    replicas: 3
    resources:
      requests:
        memory: 25Gi
        cpu: "6"
      limits:
        memory: 25Gi
        cpu: "6"
    logging: 
      type: inline
      loggers:
        log4j.logger.io.strimzi: "DEBUG"
        log4j.logger.kafka: "DEBUG"
        log4j.logger.org.apache.kafka: "DEBUG"
    authorization:
      type: opa
      
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          brokers:
          - broker: 0
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          - broker: 1
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          - broker: 2
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
    # The storage field is required by the Kafka CRD schema while the KafkaNodePools feature gate is in alpha phase.
    # But it will be ignored when Kafka Node Pools are used
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-kraft-metrics-config.yml
  # The ZooKeeper section is required by the Kafka CRD schema while the UseKRaft feature gate is in alpha phase.
  # But it will be ignored when running in KRaft mode
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    resources:
      requests:
        memory: 10Gi
        cpu: "2"
      limits:
        memory: 10Gi
        cpu: "2"
  entityOperator:
    tlsSidecar: # (28)
      resources:
        requests:
          cpu: 200m
          memory: 64Mi
        limits:
          cpu: 500m
          memory: 128Mi
    topicOperator: 
      watchedNamespace: ns-kafka
      reconciliationIntervalSeconds: 60
      logging: # (29)
        type: inline
        loggers:
          rootLogger.level: INFO
      resources:
        requests:
          memory: 512Mi
          cpu: "1"
        limits:
          memory: 512Mi
          cpu: "1"
    userOperator: 
      watchedNamespace: ns-kafka
      reconciliationIntervalSeconds: 60
      logging: # (30)
        type: inline
        loggers:
          rootLogger.level: INFO
      resources:
        requests:
          memory: 512Mi
          cpu: "1"
        limits:
          memory: 512Mi
          cpu: "1"

  kafkaExporter: 
    template:
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: agentpool
                    operator: In
                    values:
                    - kafkaexport
  cruiseControl: 
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: cruise-control-metrics
          key: metrics-config.yml

The ZooKeeper section is required by the Kafka CRD schema while the UseKRaft feature gate is in the alpha phase in Strimzi, But it will be ignored when running in KRaft mode

Pod Scheduling:

It is important to set the configs for pod scheduling in the corresponding nodes we have in our cluster. For each of the above components, we can set the pod scheduling rules using the Node affinity and Pod anti-affinity features of Kubernetes. Below is an example configuration

template:
  pod:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: agentpool
              operator: In
              values:
              - zookeeper
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: strimzi.io/name
                  operator: In
                  values:
                  - dev-cluster-zookeeper
                - key: strimzi.io/cluster
                  operator: In
                  values:
                  - dev-cluster
            topologyKey: "kubernetes.io/hostname"

After deploying this we can see the pods using the below command and you can see the below output

kubectl get pods -n ns-kafka

Previous: Strimzi Operator and Entity Operators
Next: Kafka CRD components