MinIO as Storage
1. What is MinIO?
- MinIO is a high-performance, distributed object storage system that is compatible with the Amazon S3 API.
- It's often used in self-hosted environments to provide object storage for applications like Loki.
2. How Loki Uses MinIO
- Loki stores its log data (chunks) in MinIO as objects.
- Metadata (indexes) may also be stored in MinIO or another supported backend, depending on your configuration.
3. Loki-MinIO Configuration
The Loki setup with MinIO typically works as follows:
- MinIO Pod: The pod you see (loki-minio-0) runs a MinIO instance, which serves as the object storage.
- Loki Configuration: Loki is configured to use this MinIO instance as its storage backend.
The configuration might look like this in loki-config.yaml:
storage_config:
aws:
s3: http://loki-minio:9000
bucketnames: loki
access_key_id: <your-minio-access-key>
secret_access_key: <your-minio-secret-key>
4. How Data Flows in This Setup
- Log Collection:
- Logs from Kubernetes pods are collected by agents like Promtail or Fluent Bit.
- Log Ingestion:
- Logs are ingested into Loki, which splits them into chunks and indexes.
- Storage in MinIO:
- Chunks (compressed log data) are stored as objects in MinIO.
- Indexes are stored either in MinIO or another indexing backend like BoltDB or DynamoDB.
- Querying via Grafana:
- When you query logs in Grafana, it asks Loki.
- Loki retrieves the relevant log chunks and metadata from MinIO to fulfill the query.
5. Verifying MinIO Storage
You can check the logs and storage directly in MinIO:
- Access MinIO UI (if enabled):
- MinIO typically runs a web interface on port 9000.
- Access it via http://
:9000 or through a Kubernetes service.
- List Stored Chunks:
- Once logged in, you’ll see the loki bucket.
- Inside, you’ll find folders corresponding to different log streams.
6. Useful Commands
- Check Loki-MinIO Connection: Inspect Loki’s logs to see if it’s writing to MinIO successfully:
kubectl logs <loki-pod> -n <namespace>
- Check MinIO Logs:
kubectl logs loki-minio-0 -n <namespace>
No Comments