Overview

The Controller Manager runs background control loops that regulate cluster state (replica count, endpoint updates, node health checks, etc.).

Why it’s Useful

  • Detects synchronization failures.
  • Tracks how quickly cluster state converges to the desired state.
  • Identifies unhealthy controllers that may cause outages.

What Users Can Do

  • Monitor control loop activity.
  • Tune scrape frequency for detailed visibility.
  • Troubleshoot issues in workloads not scaling properly.

Steps to Configure

  1. Get the existing ConfigMap.
    kubectl get cm opsramp-k8s-infra-metric-user-config -n <agent-installed-namespace> -o yaml
  2. Edit the ConfigMap.
    kubectl edit cm opsramp-k8s-infra-metric-user-config -n <agent-installed-namespace>
  3. Locate the kube_controller_manager section in ConfigMap.
    k8s_cluster:
      enabled: true
      config:
        scrape_interval: "2m"
  4. Update the required parameters.
  5. Save and apply the changes.

Supported Metrics

Metric NameDisplay Name
workqueue_unfinished_work_secondsWorkqueue unfinished work seconds
cronjob_controller_job_creation_skew_duration_secondsCronjob controller job creation skew duration seconds
cronjob_controller_job_creation_skew_duration_secondsCronjob controller job creation skew duration seconds
cronjob_controller_job_creation_skew_duration_secondsCronjob controller job creation skew duration seconds
disabled_metricsDisabled metrics
endpoint_slice_controller_changesEndpoint slice controller changes
endpoint_slice_controller_desired_endpoint_slicesEndpoint slice controller desired endpoint slices
endpoint_slice_controller_endpoints_added_per_syncEndpoint slice controller endpoints added per sync
endpoint_slice_controller_endpoints_added_per_syncEndpoint slice controller endpoints added per sync
endpoint_slice_controller_endpoints_added_per_syncEndpoint slice controller endpoints added per sync
endpoint_slice_controller_endpoints_desiredEndpoint slice controller endpoints desired
endpoint_slice_controller_endpoints_removed_per_syncEndpoint slice controller endpoints removed per sync
endpoint_slice_controller_endpoints_removed_per_syncEndpoint slice controller endpoints removed per sync
endpoint_slice_controller_endpoints_removed_per_syncEndpoint slice controller endpoints removed per sync
endpoint_slice_controller_endpointslices_changed_per_syncEndpoint slice controller endpointslices changed per sync
endpoint_slice_controller_endpointslices_changed_per_syncEndpoint slice controller endpointslices changed per sync
endpoint_slice_controller_endpointslices_changed_per_syncEndpoint slice controller endpointslices changed per sync
endpoint_slice_controller_num_endpoint_slicesEndpoint slice controller num endpoint slices
endpoint_slice_controller_syncsEndpoint slice controller syncs
endpoint_slice_mirroring_controller_addresses_skipped_per_syncEndpoint slice mirroring controller addresses skipped per sync
endpoint_slice_mirroring_controller_addresses_skipped_per_syncEndpoint slice mirroring controller addresses skipped per sync
endpoint_slice_mirroring_controller_addresses_skipped_per_syncEndpoint slice mirroring controller addresses skipped per sync
endpoint_slice_mirroring_controller_desired_endpoint_slicesEndpoint slice mirroring controller desired endpoint slices
endpoint_slice_mirroring_controller_endpoints_added_per_syncEndpoint slice mirroring controller endpoints added per sync
endpoint_slice_mirroring_controller_endpoints_added_per_syncEndpoint slice mirroring controller endpoints added per sync
endpoint_slice_mirroring_controller_endpoints_added_per_syncEndpoint slice mirroring controller endpoints added per sync
endpoint_slice_mirroring_controller_endpoints_desiredEndpoint slice mirroring controller endpoints desired
endpoint_slice_mirroring_controller_endpoints_removed_per_syncEndpoint slice mirroring controller endpoints removed per sync
endpoint_slice_mirroring_controller_endpoints_removed_per_syncEndpoint slice mirroring controller endpoints removed per sync
endpoint_slice_mirroring_controller_endpoints_removed_per_syncEndpoint slice mirroring controller endpoints removed per sync
endpoint_slice_mirroring_controller_endpoints_sync_durationEndpoint slice mirroring controller endpoints sync duration
endpoint_slice_mirroring_controller_endpoints_sync_durationEndpoint slice mirroring controller endpoints sync duration
endpoint_slice_mirroring_controller_endpoints_sync_durationEndpoint slice mirroring controller endpoints sync duration
endpoint_slice_mirroring_controller_endpoints_updated_per_syncEndpoint slice mirroring controller endpoints updated per sync
endpoint_slice_mirroring_controller_endpoints_updated_per_syncEndpoint slice mirroring controller endpoints updated per sync
endpoint_slice_mirroring_controller_endpoints_updated_per_syncEndpoint slice mirroring controller endpoints updated per sync
endpoint_slice_mirroring_controller_num_endpoint_slicesEndpoint slice mirroring controller num endpoint slices
ephemeral_volume_controller_create_failuresEphemeral volume controller create failures
ephemeral_volume_controller_createEphemeral volume controller create
node_collector_unhealthy_nodes_in_zoneNode collector unhealthy nodes in zone
node_collector_zone_sizeNode collector zone size
workqueue_addsWorkqueue adds total
workqueue_depthWorkqueue depth
workqueue_longest_running_processor_secondsWorkqueue longest running processor seconds
workqueue_queue_duration_secondsWorkqueue queue duration seconds
workqueue_queue_duration_secondsWorkqueue queue duration seconds
workqueue_retriesWorkqueue retries
workqueue_work_duration_secondsWorkqueue work duration seconds
workqueue_work_duration_secondsWorkqueue work duration seconds