Monitoring

Metrics and logs from every host and pod end up on one monitoring VM running Prometheus, Grafana, Loki and Alertmanager.

Metrics

Inside k3s, the Prometheus Helm chart runs a small Prometheus with kube-state-metrics and a node-exporter DaemonSet. Alertmanager and the pushgateway are disabled there, because alerting lives on the central VM.

The central Prometheus pulls from the cluster through federation rather than having the cluster push. That keeps everything pull-based like the rest of Prometheus:

- job_name: federate
  honor_labels: true
  metrics_path: /federate
  params:
    'match[]':
      - '{job="kubernetes-service-endpoints"}'
      - '{app_kubernetes_io_name="kube-state-metrics"}'
      - '{__name__=~"kube.*"}'

Dropping what I don’t use

By default you scrape everything. Dropping unused metrics with relabel rules cut the series count by a third and the chunk count by almost two thirds:

Logs

Grafana Alloy ships everything to Loki. It replaced fluent-bit and the last promtail installs in 2026.

Labels are kept low-cardinality on purpose. session-N.scope units are not labelled, because one value per SSH login would create a new stream every time. Bot traffic and requests for assets, robots.txt and favicons are dropped before they reach Loki.

{job="alloy", unit="gitea.service"}
{job="alloy_k3s", namespace="prod", app="n8n"}

Alerting

Alertmanager on the monitoring VM sends alerts to Telegram. Uptime Kuma checks the user-facing endpoints and ntfy pushes notifications to my phone.