Sample of Pod's Log
1. loki-minio-0
MinIO Object Storage Server
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-09-17T00-09-45Z (go1.18.6 linux/amd64)
Status: 2 Online, 0 Offline.
API: http://10.42.10.212:9000 http://127.0.0.1:9000
Console: http://10.42.10.212:9001 http://127.0.0.1:9001
Documentation: https://docs.min.io
You are running an older version of MinIO released 2 years ago
Update: Run `mc admin update`
2. loki-write-0
Uploading
level=info ts=2024-11-09T22:17:38.46012347Z caller=index_set.go:86 msg="uploading table loki_index_20023"
level=info ts=2024-11-09T22:17:38.46012908Z caller=index_set.go:107 msg="finished uploading table loki_index_20023"
level=info ts=2024-11-09T22:17:38.460135521Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_20023"
level=info ts=2024-11-09T22:17:38.460142591Z caller=index_set.go:86 msg="uploading table loki_index_20034"
level=info ts=2024-11-09T22:17:38.460148231Z caller=index_set.go:107 msg="finished uploading table loki_index_20034"
level=info ts=2024-11-09T22:17:38.460155091Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_20034"
level=info ts=2024-11-09T22:17:38.473078798Z caller=table_manager.go:171 index-store=boltdb-shipper-2022-01-11 msg="handing over indexes to shipper"
Flushing Stream
level=info ts=2024-11-09T22:17:47.39610939Z caller=flush.go:167 msg="flushing stream" user=fake fp=b766631cb1175681 immediate=false num_chunks=1 labels="{app=\"town-server73\", container=\"town-server\", filename=\"/var/log/pods/prd-ns_town-server73-64f86fc75b-m9xdz_cd0855bf-4e12-4c92-9e1b-2efdfa2da1f5/town-server/0.log\", job=\"prd-ns/town-server73\", namespace=\"prd-ns\", node_name=\"gamep-server17-new\", pod=\"town-server73-64f86fc75b-m9xdz\", stream=\"stdout\"}"
level=info ts=2024-11-09T22:17:47.396172471Z caller=flush.go:167 msg="flushing stream" user=fake fp=6212867a2fd2a163 immediate=false num_chunks=1 labels="{app=\"cattle-cluster-agent\", container=\"cluster-register\", filename=\"/var/log/pods/cattle-system_cattle-cluster-agent-5459b78d66-tsnqr_17c665be-8d3e-43e0-8091-9995ee89e88f/cluster-register/5528.log\", job=\"cattle-system/cattle-cluster-agent\", namespace=\"cattle-system\", node_name=\"gamelog-log02\", pod=\"cattle-cluster-agent-5459b78d66-tsnqr\", stream=\"stderr\"}"
level=info ts=2024-11-09T22:17:47.396315393Z caller=flush.go:167 msg="flushing stream" user=fake fp=91943891befc33f6 immediate=false num_chunks=1 labels="{app=\"town-server75\", container=\"town-server\", filename=\"/var/log/pods/prd-ns_town-server75-7fd49dcf6b-nq472_d5a894ed-1a5d-4816-8cc7-5e3eacc0f865/town-server/0.log\", job=\"prd-ns/town-server75\", namespace=\"prd-ns\", node_name=\"game-server18-new\", pod=\"town-server75-7fd49dcf6b-nq472\", stream=\"stdout\"}"
3. loki-backend-0
Defaulted container "loki-sc-rules" out of: loki-sc-rules, loki
{"time": "2024-09-05T03:44:49.989787+00:00", "msg": "Starting collector", "level": "INFO"}
{"time": "2024-09-05T03:44:49.990064+00:00", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory", "level": "WARNING"}
{"time": "2024-09-05T03:44:49.990367+00:00", "msg": "Loading incluster config ...", "level": "INFO"}
{"time": "2024-09-05T03:44:49.991621+00:00", "msg": "Config for cluster api at 'https://10.43.0.1:443' loaded...", "level": "INFO"}
{"time": "2024-09-05T03:44:49.991801+00:00", "msg": "Unique filenames will not be enforced.", "level": "INFO"}
{"time": "2024-09-05T03:44:49.991954+00:00", "msg": "5xx response content will not be enabled.", "level": "INFO"}
4. loki-canary-xxxxxx
Confirmation Entry
confirmation query result: 1730337782384497148
confirmation query result: 1730337781386565219
confirmation query result: 1730337780384644663
missing websocket entry 1730337790384749414 was found 119.99978837 seconds after it was originally sent
missing websocket entry 1730337791385351900 was found 118.999185884 seconds after it was originally sent
missing websocket entry 1730337792384428548 was found 118.000109216 seconds after it was originally sent
Queriying
Querying loki for logs with query: http://loki-gateway.log-ns.svc.cluster.local.:80/loki/api/v1/query_range?start=1730325543384460783&end=1730325563384460783&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-2cn9h%22%7D+&limit=1000
Querying loki for logs with query: http://loki-gateway.log-ns.svc.cluster.local.:80/loki/api/v1/query_range?start=1730326444384455826&end=1730326464384455826&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-2cn9h%22%7D+&limit=1000
Querying loki for logs with query: http://loki-gateway.log-ns.svc.cluster.local.:80/loki/api/v1/query_range?start=1730327344384760541&end=1730327364384760541&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-2cn9h%22%7D+&limit=1000
Connecting to loki at ws://loki-gateway.log-ns.svc.cluster.local.:80/loki/api/v1/tail?query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-2cn9h%22%7D+, querying for label 'pod' with value 'loki-canary-2cn9h'
error reading websocket, will retry in 10 seconds: websocket: close 1011 (internal server error): rpc error: code = Code(400) desc = max concurrent tail requests limit exceeded, count > limit (11 > 10)
1730337921385246358 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
1730337922384964178 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
1730337923385028676 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
5. loki-gateway
10.42.107.139 - - [13/Nov/2024:18:40:21 +0000] 200 "GET /loki/api/v1/query_range?start=1731520755239250219&end=1731520775239250219&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-7krp4%22%7D+&limit=1000 HTTP/1.1" 4562 "-" "loki-canary/2.9.4" "-"
10.42.107.139 - - [13/Nov/2024:18:40:21 +0000] 200 "GET /loki/api/v1/query_range?start=1731521656239239310&end=1731521676239239310&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-7krp4%22%7D+&limit=1000 HTTP/1.1" 4530 "-" "loki-canary/2.9.4" "-"
10.42.107.139 - - [13/Nov/2024:18:40:21 +0000] 200 "GET /loki/api/v1/query_range?start=1731522556239246429&end=1731522576239246429&query=%7Bstream%3D%22stdout%22%2Cpod%3D%22loki-canary-7krp4%22%7D+&limit=1000 HTTP/1.1" 4402 "-" "loki-canary/2.9.4" "-"
10.42.20.200 - - [13/Nov/2024:18:40:21 +0000] 204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.9.3" "-"
10.42.1.157 - - [13/Nov/2024:18:40:21 +0000] 204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.9.3" "-"
10.42.0.43 - - [13/Nov/2024:18:40:21 +0000] 204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.9.3" "-"
6. loki-logs-xxxxx
ts=2024-11-15T06:26:54.325615779Z caller=filetarget.go:342 level=error component=logs logs_config=log-ns/loki msg="failed to tail file, stat failed" error="stat /var/log/pods/log-ns_loki-canary-q2xv5_f36d6fff-d494-42c5-bb27-e0d715fc6a19/loki-canary/0.log: no such file or directory" filename=/var/log/pods/log-ns_loki-canary-q2xv5_f36d6fff-d494-42c5-bb27-e0d715fc6a19/loki-canary/0.log
ts=2024-11-15T06:26:54.325695409Z caller=filetarget.go:342 level=error component=logs logs_config=log-ns/loki msg="failed to tail file, stat failed" error="stat /var/log/pods/log-ns_loki-canary-q2xv5_f36d6fff-d494-42c5-bb27-e0d715fc6a19/loki-canary/1.log: no such file or directory" filename=/var/log/pods/log-ns_loki-canary-q2xv5_f36d6fff-d494-42c5-bb27-e0d715fc6a19/loki-canary/1.log
7. loki-read
ts_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s
level=info ts=2024-11-15T03:13:17.060839201Z caller=metrics.go:159 component=querier org_id=fake latency=fast query="count_over_time({stream=\"stdout\", pod=\"loki-canary-hbdnn\"}[15m] offset 8h15m0s)" query_hash=2449038259 query_type=metric range_type=instant length=0s start_delta=651.118217ms end_delta=651.118467ms step=0s duration=3.186376ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=16.164835ms splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s
level=info ts=2024-11-15T03:13:17.060892781Z caller=metrics.go:159 component=querier org_id=fake latency=fast query="count_over_time({stream=\"stdout\", pod=\"loki-canary-hbdnn\"}[15m] offset 8h15m0s)" query_hash=2449038259 query_type=metric range_type=instant length=0s start_delta=651.175718ms end_delta=651.175908ms step=0s duration=3.038683ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=15.818169ms splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s
level=info ts=2024-11-15T03:13:17.063239803Z caller=engine.go:232 component=querier org_id=fake msg="executing query" type=instant query="count_over_time({stream=\"stdout\", pod=\"loki-canary-hbdnn\"}[15m] offset 8h15m0s)" query_hash=2449038259
level=info ts=2024-11-15T03:13:17.063264453Z caller=engine.go:232 component=querier org_id=fake msg="executing query" type=instant query="count_over_time({stream=\"stdout\", pod=\"loki-canary-hbdnn\"}[15m] offset 8h15m0s)" query_hash=2449038259
level=info ts=2024-11-15T03:13:17.063264073Z caller=engine.go:232 component=querier org_id=fake msg="executing query" type=instant query="count_over_time({stream=\"stdout\", pod=\"loki-canary-hbdnn\"}[15m] offset 8h15m0s)" query_hash=2449038259
8. grafana
logger=ngalert.sender.router rule_uid=fdsom9xvzc7i8f org_id=1 t=2024-09-15T07:30:12.692705191Z level=info msg="Sending alerts to local notifier" count=1
logger=cleanup t=2024-09-15T07:35:22.30089577Z level=info msg="Completed cleanup jobs" duration=42.170638ms
logger=plugins.update.checker t=2024-09-15T07:35:22.64731653Z level=info msg="Update check succeeded" duration=178.444195ms
logger=context userId=0 orgId=0 uname= t=2024-09-15T08:23:53.918612841Z level=info msg="Request Completed" method=GET path=/dana-na/nc/nc_gina_ver.txt status=302 remote_addr=34.171.200.78 time_ms=0 duration=76.821µs size=29 referer= handler=notfound status_source=server
logger=context userId=0 orgId=0 uname= t=2024-09-15T08:23:53.922143144Z level=info msg="Request Completed" method=GET path=/tmui/login.jsp status=302 remote_addr=34.171.200.78 time_ms=0 duration=90.711µs size=29 referer= handler=notfound status_source=server
logger=context userId=0 orgId=0 uname= t=2024-09-15T08:23:53.92303671Z level=info msg="Request Completed" method=GET path=/application.wadl status=302 remote_addr=34.171.200.78 time_ms=0 duration=84.821µs size=29 referer= handler=notfound status_source=server
9. loki-grafana-agent-operator-xxxxx
level=info ts=2024-07-15T09:26:16.665620737Z controller=node controllerGroup= controllerKind=Node Node=/game-server13-new namespace= name=game-server13-new reconcileID=cd04fbd3-e1b3-40b2-bd12-3151ef3bb842 msg=Reconciling
level=info ts=2024-07-15T09:26:16.665650397Z controller=node controllerGroup= controllerKind=Node Node=/game-server13-new namespace= name=game-server13-new reconcileID=cd04fbd3-e1b3-40b2-bd12-3151ef3bb842 msg="reconciling node"
level=info ts=2024-07-15T09:26:16.676399643Z controller=node controllerGroup= controllerKind=Node Node=/game-server13-new namespace= name=game-server13-new reconcileID=cd04fbd3-e1b3-40b2-bd12-3151ef3bb842 msg="Reconcile successful"
10. promtail-xxxxx
level=info ts=2024-11-14T05:16:49.938657804Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-vsdp6_2160bff1-98ad-4a61-8fad-ce6a786cffb4/tokenize-server
level=info ts=2024-11-14T05:16:49.938674284Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-nkzdj_7dc38fae-1145-445e-b63e-802cc6fe7c79/tokenize-server
level=info ts=2024-11-14T05:16:49.938766216Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-vsdp6_2160bff1-98ad-4a61-8fad-ce6a786cffb4/tokenize-server/0.log
ts=2024-11-14T05:16:49.938800927Z caller=log.go:168 level=info msg="Seeked /var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-vsdp6_2160bff1-98ad-4a61-8fad-ce6a786cffb4/tokenize-server/0.log - &{Offset:0 Whence:0}"
level=info ts=2024-11-14T05:16:49.938803937Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-nkzdj_7dc38fae-1145-445e-b63e-802cc6fe7c79/tokenize-server/0.log
ts=2024-11-14T05:16:49.938815617Z caller=log.go:168 level=info msg="Seeked /var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-nkzdj_7dc38fae-1145-445e-b63e-802cc6fe7c79/tokenize-server/0.log - &{Offset:0 Whence:0}"
ts=2024-11-15T03:57:24.36579026Z caller=log.go:168 level=info msg="Re-opening moved/deleted file /var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-vsdp6_2160bff1-98ad-4a61-8fad-ce6a786cffb4/tokenize-server/0.log ..."
ts=2024-11-15T03:57:24.365882122Z caller=log.go:168 level=info msg="Successfully reopened /var/log/pods/prd-ns_tokenize-server02-66c8dd64b6-vsdp6_2160bff1-98ad-4a61-8fad-ce6a786cffb4/tokenize-server/0.log"
