MinIO as Storage

1. What is MinIO?

2. How Loki Uses MinIO

3. Loki-MinIO Configuration

The Loki setup with MinIO typically works as follows:

The configuration might look like this in loki-config.yaml:

storage_config:
  aws:
    s3: http://loki-minio:9000
    bucketnames: loki
    access_key_id: <your-minio-access-key>
    secret_access_key: <your-minio-secret-key>

4. How Data Flows in This Setup

  1. Log Collection:
    • Logs from Kubernetes pods are collected by agents like Promtail or Fluent Bit.
  2. Log Ingestion:
    • Logs are ingested into Loki, which splits them into chunks and indexes.
  3. Storage in MinIO:
    • Chunks (compressed log data) are stored as objects in MinIO.
    • Indexes are stored either in MinIO or another indexing backend like BoltDB or DynamoDB.
  4. Querying via Grafana:
    • When you query logs in Grafana, it asks Loki.
    • Loki retrieves the relevant log chunks and metadata from MinIO to fulfill the query.

5. Verifying MinIO Storage

You can check the logs and storage directly in MinIO:

  1. Access MinIO UI (if enabled):
    • MinIO typically runs a web interface on port 9000.
    • Access it via http://:9000 or through a Kubernetes service.
  2. List Stored Chunks:
    • Once logged in, you’ll see the loki bucket.
    • Inside, you’ll find folders corresponding to different log streams.

6. Useful Commands

kubectl logs <loki-pod> -n <namespace>
kubectl logs loki-minio-0 -n <namespace>

Revision #1
Created 15 November 2024 05:49:51 by Ahmad
Updated 15 November 2024 05:58:51 by Ahmad