This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Prometheus Configuration

Prometheus is an open-source systems monitoring and alerting toolkit. It collects and stores metrics as time series data.

Best Practice

Any package configuration options listed under Reference/Packages should be modified through package yaml files (with kind: Package) through command eksctl anywhere apply package -f packageFileName. Modifying objects outside of package yaml files may lead to unpredictable behaviors.

For automatic namespace (targetNamespace) creation, see createNamespace field: PackagebundleController.spec

Configuration options for Prometheus

1 - v2.39.1

Configuring Prometheus in EKS Anywhere package spec

Example

apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
  name: generated-prometheus
  namespace: eksa-packages-<cluster-name>
spec:
  packageName: prometheus
  targetNamespace: observability
  config: | 
    server:
      replicaCount: 2
      statefulSet:
        enabled: true

Configurable parameters and default values under spec.config

Parameter Description Default
General
rbac.create Specifies if clusterRole / role and clusterRoleBinding / roleBinding will be created for prometheus-server and node-exporter true
sourceRegistry Specifies image source registry for prometheus-server and node-exporter "783794618700.dkr.ecr.us-west-2.amazonaws.com"
Node-Exporter
nodeExporter.enabled Indicates if node-exporter is enabled true
nodeExporter.hostNetwork Indicates if node-exporter shares the host network namespace true
nodeExporter.hostPID Indicates if node-exporter shares the host process ID namespace true
nodeExporter.image.pullPolicy Specifies node-exporter image pull policy: IfNotPresent, Always, Never "IfNotPresent"
nodeExporter.image.repository Specifies node-exporter image repository "prometheus/node-exporter"
nodeExporter.resources Specifies resource requests and limits of the node-exporter container. Refer to the Kubernetes API documentation ResourceRequirements field for more details {}
nodeExporter.service Specifies how to expose node-exporter as a network service See footnote 1
nodeExporter.tolerations Specifies node tolerations for node-exporter scheduling to nodes with taints. Refer to the Kubernetes API documentation toleration field for more details. See footnote 2
serviceAccounts.nodeExporter.annotations Specifies node-exporter service account annotations {}
serviceAccounts.nodeExporter.create Indicates if node-exporter service account will be created true
serviceAccounts.nodeExporter.name Specifies node-exporter service account name ""
Prometheus-Server
server.enabled Indicates if prometheus-server is enabled true
server.global.evaluation_interval Specifies how frequently the prometheus-server rules are evaluated "1m"
server.global.scrape_interval Specifies how frequently prometheus-server will scrape targets "1m"
server.global.scrape_timeout Specifies how long until a prometheus-server scrape request times out "10s"
server.image.pullPolicy Specifies prometheus-server image pull policy: IfNotPresent, Always, Never "IfNotPresent"
server.image.repository Specifies prometheus-server image repository "prometheus/prometheus"
server.name Specifies prometheus-server container name "server"
server.persistentVolume.accessModes Specifies prometheus-server data Persistent Volume access modes "ReadWriteOnce"
server.persistentVolume.enabled Indicates if prometheus-server will create/use a Persistent Volume Claim true
server.persistentVolume.existingClaim Specifies prometheus-server data Persistent Volume existing claim name. It requires server.persistentVolume.enabled: true. If defined, PVC must be created manually before volume will be bound ""
server.persistentVolume.size Specifies prometheus-server data Persistent Volume size "8Gi"
server.remoteRead Specifies prometheus-server remote read configs. Refer to Prometheus docs remote_read for more details []
server.remoteWrite Specifies prometheus-server remote write configs. Refer to Prometheus docs remote_write for more details []
server.replicaCount Specifies the replicaCount for prometheus-server deployment / statefulSet. Note: server.statefulSet.enabled should be set to true if server.replicaCount is greater than 1 1
server.resources Specifies resource requests and limits of the prometheus-server container. Refer to the Kubernetes API documentation ResourceRequirements field for more details {}
server.retention Specifies prometheus-server data retention period "15d"
server.service Specifies how to expose prometheus-server as a network service See footnote 3
server.statefulSet.enabled Indicates if prometheus-server is deployed as a statefulSet. If set to false, prometheus-server will be deployed as a deployment false
serverFiles.“prometheus.yml”.scrape_configs Specifies a set of targets and parameters for prometheus-server describing how to scrape them. Refer to Prometheus docs scrape_config for more details See footnote 4
serviceAccounts.server.annotations Specifies prometheus-server service account annotations {}
serviceAccounts.server.create Indicates if prometheus-server service account will be created true
serviceAccounts.server.name Specifies prometheus-server service account name ""

  1. Node-exporter service is exposed as a clusterIP with port: 9100 (controlled by nodeExporter.service.servicePort below) and targetPort: 9100 (controlled by nodeExporter.service.hostPort below) by default. Note the annotation prometheus.io/scrape: "true" is mandatory in order for node-exporter to be discovered by prometheus-server as a scrape target. See below specification for details.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
...
spec:
  config: |
  spec:
  config: |
    nodeExporter:
      service:
        annotations:
          prometheus.io/scrape: "true"
        hostPort: 9100
        servicePort: 9100
        type: ClusterIP
     ↩︎

  2. Node-exporter pods have the following toleration by default, which allows daemonSet to be scheduled on control plane node.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
...
spec:
  config: |
    nodeExporter:
      tolerations:
        # For K8 version prior to 1.24
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
        # For K8 version 1.24+
        - key: "node-role.kubernetes.io/control-plane"
          operator: "Exists"
          effect: "NoSchedule"
     ↩︎

  3. Prometheus-server service is exposed as a clusterIP with port: 9090 (controlled by server.service.servicePort below) and targetPort: 9090 (not overridable) by default. See below specification for details.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
...
spec:
  config: |
    server:
      service:
        enabled: true
        servicePort: 9090
        type: ClusterIP
     ↩︎

  4. Prometheus-server by default has the following scrape configs.

    apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
...
spec:
  config: | 
    serverFiles:
      prometheus.yml:
        scrape_configs:
          - job_name: prometheus
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: http
            follow_redirects: true
            enable_http2: true
            static_configs:
            - targets:
              - localhost:9090
          - job_name: kubernetes-apiservers
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: https
            authorization:
              type: Bearer
              credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
              separator: ;
              regex: default;kubernetes;https
              replacement: $1
              action: keep
            kubernetes_sd_configs:
            - role: endpoints
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-nodes
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: https
            authorization:
              type: Bearer
              credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - separator: ;
              regex: __meta_kubernetes_node_label_(.+)
              replacement: $1
              action: labelmap
            - separator: ;
              regex: (.*)
              target_label: __address__
              replacement: kubernetes.default.svc:443
              action: replace
            - source_labels: [__meta_kubernetes_node_name]
              separator: ;
              regex: (.+)
              target_label: __metrics_path__
              replacement: /api/v1/nodes/$1/proxy/metrics
              action: replace
            kubernetes_sd_configs:
            - role: node
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-nodes-cadvisor
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: https
            authorization:
              type: Bearer
              credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - separator: ;
              regex: __meta_kubernetes_node_label_(.+)
              replacement: $1
              action: labelmap
            - separator: ;
              regex: (.*)
              target_label: __address__
              replacement: kubernetes.default.svc:443
              action: replace
            - source_labels: [__meta_kubernetes_node_name]
              separator: ;
              regex: (.+)
              target_label: __metrics_path__
              replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
              action: replace
            kubernetes_sd_configs:
            - role: node
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-service-endpoints
            honor_labels: true
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: http
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
              separator: ;
              regex: "true"
              replacement: $1
              action: keep
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
              separator: ;
              regex: "true"
              replacement: $1
              action: drop
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
              separator: ;
              regex: (https?)
              target_label: __scheme__
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
              separator: ;
              regex: (.+)
              target_label: __metrics_path__
              replacement: $1
              action: replace
            - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
              separator: ;
              regex: (.+?)(?::\d+)?;(\d+)
              target_label: __address__
              replacement: $1:$2
              action: replace
            - separator: ;
              regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
              replacement: __param_$1
              action: labelmap
            - separator: ;
              regex: __meta_kubernetes_service_label_(.+)
              replacement: $1
              action: labelmap
            - source_labels: [__meta_kubernetes_namespace]
              separator: ;
              regex: (.*)
              target_label: namespace
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_service_name]
              separator: ;
              regex: (.*)
              target_label: service
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_node_name]
              separator: ;
              regex: (.*)
              target_label: node
              replacement: $1
              action: replace
            kubernetes_sd_configs:
            - role: endpoints
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-service-endpoints-slow
            honor_labels: true
            honor_timestamps: true
            scrape_interval: 5m
            scrape_timeout: 30s
            metrics_path: /metrics
            scheme: http
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
              separator: ;
              regex: "true"
              replacement: $1
              action: keep
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
              separator: ;
              regex: (https?)
              target_label: __scheme__
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
              separator: ;
              regex: (.+)
              target_label: __metrics_path__
              replacement: $1
              action: replace
            - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
              separator: ;
              regex: (.+?)(?::\d+)?;(\d+)
              target_label: __address__
              replacement: $1:$2
              action: replace
            - separator: ;
              regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
              replacement: __param_$1
              action: labelmap
            - separator: ;
              regex: __meta_kubernetes_service_label_(.+)
              replacement: $1
              action: labelmap
            - source_labels: [__meta_kubernetes_namespace]
              separator: ;
              regex: (.*)
              target_label: namespace
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_service_name]
              separator: ;
              regex: (.*)
              target_label: service
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_node_name]
              separator: ;
              regex: (.*)
              target_label: node
              replacement: $1
              action: replace
            kubernetes_sd_configs:
            - role: endpoints
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: prometheus-pushgateway
            honor_labels: true
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: http
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
              separator: ;
              regex: pushgateway
              replacement: $1
              action: keep
            kubernetes_sd_configs:
            - role: service
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-services
            honor_labels: true
            honor_timestamps: true
            params:
              module:
              - http_2xx
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /probe
            scheme: http
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
              separator: ;
              regex: "true"
              replacement: $1
              action: keep
            - source_labels: [__address__]
              separator: ;
              regex: (.*)
              target_label: __param_target
              replacement: $1
              action: replace
            - separator: ;
              regex: (.*)
              target_label: __address__
              replacement: blackbox
              action: replace
            - source_labels: [__param_target]
              separator: ;
              regex: (.*)
              target_label: instance
              replacement: $1
              action: replace
            - separator: ;
              regex: __meta_kubernetes_service_label_(.+)
              replacement: $1
              action: labelmap
            - source_labels: [__meta_kubernetes_namespace]
              separator: ;
              regex: (.*)
              target_label: namespace
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_service_name]
              separator: ;
              regex: (.*)
              target_label: service
              replacement: $1
              action: replace
            kubernetes_sd_configs:
            - role: service
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-pods
            honor_labels: true
            honor_timestamps: true
            scrape_interval: 1m
            scrape_timeout: 10s
            metrics_path: /metrics
            scheme: http
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              separator: ;
              regex: "true"
              replacement: $1
              action: keep
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
              separator: ;
              regex: "true"
              replacement: $1
              action: drop
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
              separator: ;
              regex: (https?)
              target_label: __scheme__
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              separator: ;
              regex: (.+)
              target_label: __metrics_path__
              replacement: $1
              action: replace
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              separator: ;
              regex: (.+?)(?::\d+)?;(\d+)
              target_label: __address__
              replacement: $1:$2
              action: replace
            - separator: ;
              regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
              replacement: __param_$1
              action: labelmap
            - separator: ;
              regex: __meta_kubernetes_pod_label_(.+)
              replacement: $1
              action: labelmap
            - source_labels: [__meta_kubernetes_namespace]
              separator: ;
              regex: (.*)
              target_label: namespace
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_name]
              separator: ;
              regex: (.*)
              target_label: pod
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_phase]
              separator: ;
              regex: Pending|Succeeded|Failed|Completed
              replacement: $1
              action: drop
            kubernetes_sd_configs:
            - role: pod
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
          - job_name: kubernetes-pods-slow
            honor_labels: true
            honor_timestamps: true
            scrape_interval: 5m
            scrape_timeout: 30s
            metrics_path: /metrics
            scheme: http
            follow_redirects: true
            enable_http2: true
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow]
              separator: ;
              regex: "true"
              replacement: $1
              action: keep
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
              separator: ;
              regex: (https?)
              target_label: __scheme__
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              separator: ;
              regex: (.+)
              target_label: __metrics_path__
              replacement: $1
              action: replace
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              separator: ;
              regex: (.+?)(?::\d+)?;(\d+)
              target_label: __address__
              replacement: $1:$2
              action: replace
            - separator: ;
              regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
              replacement: __param_$1
              action: labelmap
            - separator: ;
              regex: __meta_kubernetes_pod_label_(.+)
              replacement: $1
              action: labelmap
            - source_labels: [__meta_kubernetes_namespace]
              separator: ;
              regex: (.*)
              target_label: namespace
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_name]
              separator: ;
              regex: (.*)
              target_label: pod
              replacement: $1
              action: replace
            - source_labels: [__meta_kubernetes_pod_phase]
              separator: ;
              regex: Pending|Succeeded|Failed|Completed
              replacement: $1
              action: drop
            kubernetes_sd_configs:
            - role: pod
              kubeconfig_file: ""
              follow_redirects: true
              enable_http2: true
     ↩︎