Introduction

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant messaging. It is widely used for building real-time data pipelines and event-driven applications. Monitoring Kafka is essential to ensure optimal performance, detect potential bottlenecks, and maintain system reliability.

To monitor Kafka efficiently, we can use the OpenTelemetry Kafka Receiver, which collects key Kafka performance metrics. These include broker health, topic throughput, partition activity, consumer lag, and more, helping to gain deep visibility into Kafka’s performance and health.

Steps to Configure

Step 1: Check for Existing ConfigMap

Run the following command to see if the opsramp-workload-metric-user-config ConfigMap already exists:

kubectl get configmap opsramp-workload-metric-user-config -n <agent-installed-namespace>

Step 2: Create a New ConfigMap (if not available)

If this is your first setup, create a workload-master ConfigMap. Use the following template and update it as per your workload requirements:

apiVersion: v1
kind: ConfigMap
metadata:
  name: opsramp-workload-metric-user-config
  namespace: opsramp-agent
data:
  workloads: |
    kafka:
      - name: kafka
        collectionInterval: 60s
        protocolVersion: "2.0.0"
        scrapers:
          - brokers
          - topics
          - consumers
        auth: none
        port: 9092
        targetPodSelector:
          matchLabels:
            - key: app.kubernetes.io/instance
              operator: ==
              value:
                - my-kafka-plain
            - key: app.kubernetes.io/name
              operator: ==
              value:
                - kafka

      - name: kafka-sasl-plaintext
        collectionInterval: 60s
        protocolVersion: "2.0.0"
        scrapers:
          - brokers
          - topics
          - consumers
        auth: sasl_plaintext
        userName:
          value: "user1"
        password:
          value: "password1"
        port: 9092
        targetPodSelector:
          matchLabels:
            - key: app.kubernetes.io/instance
              operator: ==
              value:
                - my-kafka-plain
            - key: app.kubernetes.io/name
              operator: ==
              value:
                - kafka

      - name: kafka-tls
        collectionInterval: 60s
        protocolVersion: "2.0.0"
        scrapers:
          - brokers
          - topics
          - consumers
        auth: tls
        certFile:
          source: secret
          value: default/kafka-client-tls-secret/client.pem
        keyFile:
          source: secret
          value: default/kafka-client-tls-secret/client-key.pem
        insecureSkipVerify: true
        port: 9093
        targetPodSelector:
          matchLabels:
            - key: app.kubernetes.io/instance
              operator: ==
              value:
                - my-kafka-plain
            - key: app.kubernetes.io/name
              operator: ==
              value:
                - kafka

Step 3: Apply the ConfigMap

kubectl apply -f <path-to-your-manifest-file>

Step 4: Verify in OpsRamp Portal

After a few minutes, check metrics in the OpsRamp Portal > Dashboards > Dashboard.

Step 5: Set Up Alerts and Availability Rules

Once metrics are visible, you can configure alerts and availability rules in the OpsRamp portal. Edit the relevant ConfigMaps to adjust workload or application-specific monitoring.

Supported Metrics

Supported metrics for this workload as provided by the Kubernetes 2.0 Agent.