Use cAdvisor/Kubelet metrics

Autopilot Standard

You can configure a Google Kubernetes Engine (GKE) cluster to send a curated set of cAdvisor/Kubelet metrics to Cloud Monitoring using Google Cloud Managed Service for Prometheus. This document describes how these metrics are formatted when they are written to Cloud Monitoring and how to query them. This document also provides tables that list the metrics in each set and provides information about how you can use these metrics.

Before you can use cAdvisor/Kubelet metrics, you must enable their collection.

Metric format

All Kubernetes cAdvisor/Kubelet metrics written to Cloud Monitoring use the resource type prometheus_target. Each metric name is prefixed with prometheus.googleapis.com/ and has a suffix indicating the Prometheus metric type, such as /gauge, /histogram, or /counter. Otherwise, each metric name is identical to the metric name exposed by open source Kubernetes.

Exporting from Cloud Monitoring

The cAdvisor/Kubelet metrics can be exported from Cloud Monitoring by using the Cloud Monitoring API. Because all cAdvisor/Kubelet metrics are ingested by using Google Cloud Managed Service for Prometheus, cAdvisor/Kubelet metrics can be queried by using Prometheus Query Language (PromQL). They can also be queried by using by using Monitoring Query Language (MQL).

Querying metrics

When you query cAdvisor/Kubelet metrics, the name you use depends on whether you are using PromQL or Cloud Monitoring-based features like MQL or the Metrics Explorer menu-driven interface.

The following tables of cAdvisor/Kubelet metrics show two versions of each metric name:

PromQL metric name: When using PromQL in Cloud Monitoring pages of the Google Cloud console or in PromQL fields of the Cloud Monitoring API, use the PromQL metric name.
Cloud Monitoring metric name When using other Cloud Monitoring features, use the Cloud Monitoring metric name in the tables below. This name must be prefixed with prometheus.googleapis.com/, which has been omitted from the entries in the table.

cAdvisor metrics

The Cloud Monitoring metric names in this table must be prefixed with prometheus.googleapis.com/. That prefix has been omitted from the entries in the table.

PromQL metric name Cloud Monitoring metric name
Kind, Type, Unit Monitored resources Required GKE version	Description Labels
`container_cpu_cfs_periods_total` `container_cpu_cfs_periods_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of elapsed enforcement period intervals. Sampled every 30 seconds. `cpu`
`container_cpu_cfs_throttled_periods_total` `container_cpu_cfs_throttled_periods_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of throttled period intervals. Sampled every 30 seconds. `cpu`
`container_cpu_usage_seconds_total` `container_cpu_usage_seconds_total/counter`
`CUMULATIVE`, `DOUBLE`, `s` prometheus_target 1.29.3-gke.1093000	Cumulative cpu time consumed. Sampled every 30 seconds. `cpu`
`container_fs_limit_bytes` `container_fs_limit_bytes/gauge`
`GAUGE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Number of bytes that can be consumed by the container on this filesystem. Sampled every 30 seconds. `disk`
`container_fs_read_seconds_total` `container_fs_read_seconds_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of reads completed. Sampled every 30 seconds. `diskIO`
`container_fs_reads_bytes_total` `container_fs_reads_bytes_total/counter`
`CUMULATIVE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Cumulative count of bytes read. Sampled every 30 seconds. `diskIO`
`container_fs_reads_total` `container_fs_reads_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of reads completed. Sampled every 30 seconds. `diskIO`
`container_fs_usage_bytes` `container_fs_usage_bytes/gauge`
`GAUGE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Number of bytes that are consumed by the container on this filesystem. Sampled every 30 seconds. `disk`
`container_fs_write_seconds_total` `container_fs_write_seconds_total/counter`
`CUMULATIVE`, `DOUBLE`, `s` prometheus_target 1.29.3-gke.1093000	Cumulative count of seconds spent writing. Sampled every 30 seconds. `diskIO`
`container_fs_writes_bytes_total` `container_fs_writes_bytes_total/counter`
`CUMULATIVE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Cumulative count of bytes written. Sampled every 30 seconds. `diskIO`
`container_fs_writes_total` `container_fs_writes_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of writes completed. Sampled every 30 seconds. `diskIO`
`container_memory_rss` `container_memory_rss/gauge`
`GAUGE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Size of RSS. Sampled every 30 seconds. `memory`
`container_memory_working_set_bytes` `container_memory_working_set_bytes/gauge`
`GAUGE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Current working set. Sampled every 30 seconds. `memory`
`container_network_receive_bytes_total` `container_network_receive_bytes_total/counter`
`CUMULATIVE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Cumulative count of bytes received. Sampled every 30 seconds. `network`
`container_network_receive_packets_dropped_total` `container_network_receive_packets_dropped_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of packets dropped while receiving. Sampled every 30 seconds. `network`
`container_network_receive_packets_total` `container_network_receive_packets_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of packets received. Sampled every 30 seconds. `network`
`container_network_transmit_bytes_total` `container_network_transmit_bytes_total/counter`
`CUMULATIVE`, `DOUBLE`, `By` prometheus_target 1.29.3-gke.1093000	Cumulative count of bytes transmitted. Sampled every 30 seconds. `network`
`container_network_transmit_packets_dropped_total` `container_network_transmit_packets_dropped_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of packets dropped while transmitting. Sampled every 30 seconds. `network`
`container_network_transmit_packets_total` `container_network_transmit_packets_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative count of packets transmitted. Sampled every 30 seconds. `network`

Kubelet metrics

The Cloud Monitoring metric names in this table must be prefixed with prometheus.googleapis.com/. That prefix has been omitted from the entries in the table.

PromQL metric name Cloud Monitoring metric name
Kind, Type, Unit Monitored resources Required GKE version	Description Labels
`kubelet_certificate_manager_server_ttl_seconds` `kubelet_certificate_manager_server_ttl_seconds/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate. The value is in seconds until certificate expiry (negative if already expired). If serving certificate is invalid or unused, the value will be +INF. Sampled every 30 seconds.
`kubelet_node_name` `kubelet_node_name/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	The node's name. The count is always 1. Sampled every 30 seconds. `node`
`kubelet_pleg_relist_duration_seconds` `kubelet_pleg_relist_duration_seconds/histogram`
`CUMULATIVE`, `DISTRIBUTION`, `s` prometheus_target 1.29.3-gke.1093000	Duration in seconds for relisting pods in PLEG. Sampled every 30 seconds.
`kubelet_pod_worker_duration_seconds` `kubelet_pod_worker_duration_seconds/histogram`
`CUMULATIVE`, `DISTRIBUTION`, `s` prometheus_target 1.29.3-gke.1093000	Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync. Sampled every 30 seconds. `operation_type`
`kubelet_running_containers` `kubelet_running_containers/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of containers currently running. Sampled every 30 seconds. `container_state`
`kubelet_running_pods` `kubelet_running_pods/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of pods that have a running pod sandbox. Sampled every 30 seconds.
`kubelet_runtime_operations_total` `kubelet_runtime_operations_total/counter`
`CUMULATIVE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Cumulative number of runtime operations by operation type. Sampled every 30 seconds. `operation_type`
`kubelet_volume_stats_available_bytes` `kubelet_volume_stats_available_bytes/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of available bytes in the volume. Sampled every 30 seconds. `namespace` `persistentvolumeclaim`
`kubelet_volume_stats_capacity_bytes` `kubelet_volume_stats_capacity_bytes/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Capacity in bytes of the volume. Sampled every 30 seconds. `namespace` `persistentvolumeclaim`
`kubelet_volume_stats_inodes` `kubelet_volume_stats_inodes/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Maximum number of inodes in the volume. Sampled every 30 seconds. `namespace` `persistentvolumeclaim`
`kubelet_volume_stats_inodes_free` `kubelet_volume_stats_inodes_free/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of free inodes in the volume. Sampled every 30 seconds. `namespace` `persistentvolumeclaim`
`kubelet_volume_stats_inodes_used` `kubelet_volume_stats_inodes_used/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of used inodes in the volume. Sampled every 30 seconds. `namespace` `persistentvolumeclaim`
`kubelet_volume_stats_used_bytes` `kubelet_volume_stats_used_bytes/gauge`
`GAUGE`, `DOUBLE`, `1` prometheus_target 1.29.3-gke.1093000	Number of used bytes in the volume. Sampled every 30 seconds. `namespace` `persistentvolumeclaim`