Kubernetes: Prometheus service discovery

Service discovery using “prometheus.io/*” annotations does not work out of the box in Kubernetes. It can be done with a custom “relabel_config” (https://gist.github.com/reachlin/a98b90afcbff4604c90c183a0169474f).

In Kubernetes we need a custom ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: service-monitor
  namespace: monitoring
  labels:
    k8s-app: service-monitor
spec:
  jobLabel: k8s-app
  endpoints:
    - port: http
      interval: 15s
      relabelings:
        # Keep targets with label 
        # __meta_kubernetes_service_annotation_prometheus_io_scrape equals 'true',
        # which means the user added prometheus.io/scrape: true in the service's
        # annotation.
        - action: keep
          sourceLabels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          regex: 'true'
        # If __meta_kubernetes_service_annotation_prometheus_io_scheme is http or
        # https, replace __scheme__ with its value that is the default 
        # replacement: $1. This means the user added 
        # prometheus.io/scheme in the service's annotation.
        - action: replace
          sourceLabels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          targetLabel: __scheme__
          regex: '(https?)'
        # This is also for service annotation of prometheus, if the user overrides
        # the scrape path, its value will be put in __metrics_path__
        - action: replace
          sourceLabels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          targetLabel: __metrics_path__
          regex: '(.+)'
        # If the user added prometheus.io/port in the service annotation, use this
        # port to replace the port in __address__
        - sourceLabels: 
            - __address__
            - __meta_kubernetes_service_annotation_prometheus_io_port
          action: replace
          targetLabel: __address__
          regex: (.+)(?::\d+);(\d+)
          replacement: $1:$2

        # all above rules are for handling prometheus annotation

        # All __meta_kubernetes_service_label_(.+) will be changed to the (.+), e.g.,
        # __meta_kubernetes_service_label_app='armada-api' to app='armada-api'
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        # All __meta_kubernetes_pod_label_(.+) will be changed to the (.+), e.g.,
        # __meta_kubernetes_pod_label_app='armada-api' to app='armada-api'
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        # If __meta_kubernetes_namespace matches .*, put its value in label
        # kubernetes_namespace
        - sourceLabels: [__meta_kubernetes_namespace]
          action: replace
          targetLabel: kubernetes_namespace
        # If __meta_kubernetes_service_name matches .*, put its
        # value in label service_name
        - sourceLabels: [__meta_kubernetes_service_name]
          action: replace
          targetLabel: service_name
        # If __meta_kubernetes_pod_node_name matches .*, put its
        # value in label hostname
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          action: replace
          targetLabel: hostname
  selector:
    matchLabels: {}  # Empty to match all labels
  namespaceSelector:
    matchNames:
      - [list of namespaces we want to monitor]