Observability Platform — Logs

This post is part of a series of blogs about setting up an Observability platform for an organization. This series includes the details of observability platform components, architecture, and the tool stack used to build the platform.

1. Observability Platform — Introduction
2. Observability Platform — Components and tools
3. Observability Platform — Metrics (Prometheus & Grafana)
4. Observability Platform — Logs
5. Observability Platform — Traces

We have two methods to write application logs.

1. Write logs to the console
2. Write logs to files.

Collecting and analysing logs from Kubernetes pods is easy compared to the legacy file-based logs collection methods. For collecting console logs from Kubernetes pods we can use Grafana Promtail and to visualize these logs we can use Grafana Loki. Still some legacy systems like the ones written in the .Net framework rely on file-based logs. To collect logs from these systems deployed in the Kubernetes we can use Filebeat, Elastic Search and Grafana.

Collecting console logs:

1. Deploy Loki along with Grafana to the cluster where we are planning to visualize the logs.
2. Deploy promtail to the cluster from which we need to collect the logs

Loki Deployment:

Grafana Loki is an open-source log aggregation system developed by Grafana Labs. It is designed to be cost-effective and easy to operate, especially when compared to traditional log aggregation systems like the ELK (Elasticsearch, Logstash, Kibana) stack.

Loki logging stack

A typical Loki-based logging stack consists of 3 components:

• Agent — An agent or client, which is distributed with Loki. The agent scrapes logs, turns the logs into streams by adding labels, and pushes the streams to Loki through an HTTP API. We are using Promtail as the agent
• Loki — The main server, responsible for ingesting and storing logs and processing queries.
• Grafana for querying and displaying log data. You can also query logs from the command line, using LogCLI or using the Loki API directly.

We can deploy Loki in three ways:

• monolithic Loki
• microservice Loki
• scalable Loki

Here we are deploying in scalable mode. First we need to create the values.yaml file to set the deployment configurations. Below is an example ‘yaml’ file with important configurations. In this we are setting the promtail tenant details, Gateway service and ingress, then the Azure storage details to store the logs.

loki:
  tenants:
    - username: <promtail_tenant_user>
      password: <promtail_tenant_password>
      orgid: <promtail_tenant_id>
    - username: <promtail_tenant_user1>
      password: <promtail_tenant_password1>
      orgid: <promtail_tenant_id1>
gateway:
  enabled: true
  replicas: 1
  image:
    registry: docker.io
    repository: nginxinc/nginx-unprivileged
    tag: 1.19-alpine
    digest: null
    pullPolicy: IfNotPresent
  service:
    port: 80
    type: ClusterIP
  ingress:
    enabled: true
    ingressClassName: "nginx"
    # -- Annotations for the gateway ingress
    annotations: 
     nginx.ingress.kubernetes.io/configuration-snippet: |
       proxy_set_header X-Scope-OrgID $remote_user;
    hosts:
      - host: logs.app.domain.com
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: logs-domain-tls
        hosts:
          - logs.app.domain.com
storage:
    bucketNames:
      chunks: <LOKI_CONTAINER_NAME>
      ruler: <LOKI_CONTAINER_NAME>
      admin: <LOKI_CONTAINER_NAME>
    type: azure
    azure:
      accountName: <AZURE_STORAGE_ACCOUNT_NAME>
      accountKey: <AZURE_STORAGE_ACCOUNT_KEY>
deploymentMode: SimpleScalable

backend:
  replicas: 3
read:
  replicas: 3
write:
  replicas: 3

# Enable minio for storage
minio:
  enabled: true

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0

ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install --values values.yaml loki grafana/loki

#Upgrade
helm upgrade --values values.yaml loki grafana/loki

Now setup the Loki DataSource in Grafana

Promtail Agent:

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that runs applications which need to be monitored.

To install promtail first we need to create the values.yaml file with the configurations. Then use the Helm chart to deploy it. In the values file maily we need to set the Loki config values.

config:
# publish data to loki
  clients:
    - url: https://<PROMTAIL_TENANT_USER>:<PROMTAIL_TENANT_PASSWORD>@logs.app.domain.com/loki/api/v1/push
      tenant_id: PROMTAIL_TENANT_ID

helm repo add grafana https://grafana.github.io/helm-charts
# The default helm configuration deploys promtail as a daemonSet (recommended)
helm upgrade --values values.yaml --install promtail grafana/promtail

Now promtail will collect the logs from each pods and ship them to the Loki. In Grafana by exploring the Loki datasource we can get the logs.

We will discuss traces in the next part of this series, Observability Platform — Traces.