Observability Platform — Metrics (Prometheus & Grafana)

This post is part of a series of blogs about setting up an Observability platform for an organization. This series includes the details of observability platform components, architecture, and the tool stack used to build the platform.

1. Observability Platform — Introduction
2. Observability Platform — Components and tools
3. Observability Platform — Metrics (Prometheus & Grafana)
4. Observability Platform — Logs
5. Observability Platform — Traces

To deploy Prometheus and Grafana we can either use the Kube Prometheus Stack helm chart or the dedicated helm charts for Prometheus and Grafana

Kube Prometheus Stack helm chart has options to deploy the below components:

• Prometheus Operator
• Prometheus and its components (kube-state-matrics, Prometheus-node-exporter etc)
• Grafana

We need to deploy Prometheus in all Kubernetes clusters from which you want to collect the metrics. We need to deploy Grafana in a single cluster only which can be used as the central point where we can monitor all of our clusters.

Deployment using Kube Prometheus Stack Chart

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -f prometheus.yml -n app-monitoring

You can use values.yaml file from the git repo to create your custom values.yaml file and add configurations for Grafana and Prometheus in the same values.yaml file.

To enable or disable Grafana deployment you can use the below attribute in the chart.

grafana:
  enabled: false
  namespaceOverride: ""

Deployment using dedicated Charts

## Prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus -f prometheus-values.yaml 

## Grafana

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana -f graphana-values.yam

Ingress Deployment:

The values file has options to enable ingress. We need to set the attribute ‘enabled’ to ‘true’ and need to provide the tls secret name and the hostname. This will enable us to access Grafana from the browser.

ingress:
  enabled: true
  ingressClassName: nginx
  annotations: {}
  labels: {}
  path: /
  pathType: Prefix

  hosts:
    - DOMAIN_NAME_VALUE
  extraPaths: []
  tls: 
   - secretName: <tls-secretname>
     hosts:
       - <grafana_host_name>

The charts have the option to set Prometheus ingress also in the values.yaml file. In the dedicated chart is it under the section ‘server’ and in the common chart it is under the section ‘prometheus’. Here also we need to set the attribute ‘enabled’ to ‘true’ and provide the tls secret name and the hostname.

ingress:
    enabled: true
    ingressClassName: nginx
    annotations: {}
    extraLabels: {}
    hosts: 
      - <promethues.domain.com>
    path: /
    pathType: Prefix
    extraPaths: []
    tls: 
      - secretName: <tls secretname>
        hosts:
          - <promethues.domain.com>

We will be using the URL from the ingress config (promethues.domain.com) to create a Data source in Grafana.

Setting Prometheus Datasource in Grafana:

Provide a name for your datasource and put the prometheus ingress url in the url field. Then save the configuration.

Now from the ‘Explore’ menu, you can select the data source you have created and query the Prometheus data source to create dashboards.

We will discuss logs in the next part of this series, Observability Platform — Logs