AI SRE

I built the on-call log review I used to do by hand into a service. It reads Loki logs across five hosts, writes a prioritized incident report through Claude, and — the part that makes it more than a summarizer — remembers open issues across runs so every report tells me what's new, still ongoing, or newly resolved.

By the numbers

What it is

A single-file FastAPI service (ai-sre-api) sitting between Grafana Loki and the Anthropic Messages API. Logs from five sources ship into one Loki instance; the service pulls the relevant lines with LogQL and asks a Claude model to interpret them as a Site Reliability Engineer would. The whole stack — Loki, Promtail, and the API — is three containers defined by one docker-compose.yml, running on a 4 GB Proxmox VM.

• POST /report — a balanced multi-host infrastructure report: high-priority issues first, per-host health (🟢/🟡/🔴/⚪), security observations, recommendations.
• POST /ask — a free-form natural-language question against any log source (or raw LogQL) for ad-hoc investigation.

Log ingestion is deliberately heterogeneous, because real infrastructure is: Promtail on the Linux hosts, OPNsense via remote syslog to a receiver, Proxmox via journald→Promtail, and the nginx box shipping parsed access logs. It all lands in Loki under a coarse, queryable label scheme.

The standout: cross-run memory

An LLM summarizing a time window is stateless — run it twice and it re-reports the same disk warning as if it were new every time. That's noise, and noise gets ignored. So each scheduled run carries a persistent issue-ledger:

• The API reads the prior ledger and injects it into the prompt, so the model diffs current findings against known state and tags each one NEW, ONGOING since <date>, or RESOLVED.
• It writes back an updated ledger — an HTML-comment-delimited table of open issues only, regex-extracted from the model's free-form output.
• Writes are atomic (temp file + os.replace) and are skipped entirely when a response is an error, empty, or a parse-miss — so a bad run can never clobber a good ledger.
• The ledger key is slug-sanitized against path traversal before it touches the filesystem.

<!-- SRE-LEDGER-BEGIN -->
| id                           | host | first_seen | last_seen  | status  | priority | summary
| pve-smart-drive-degradation  | pve  | 2026-07-01 | 2026-07-05 | ONGOING | P2       | /dev/sdc SMART prefailure escalating
<!-- SRE-LEDGER-END -->

This is context engineering applied to observability: state persisted between otherwise-stateless calls so the reports build on each other instead of analyzing every window cold.

Engineering against LLM failure modes

Most of the hard work was making a language model behave like a trustworthy monitor instead of an anxious one. Each of these fixes a specific failure I watched it make:

• Silence ≠ outage. Every host gets two queries — error/warning lines and a small non-error activity sample. A quiet host reads 🟢 Healthy; a genuinely silent one reads ⚪ No Data. Without this the model invents outages from the absence of logs.
• Domain guidance on disk health. A prompt block teaches the model that Seagate's normalized Raw_Read_Error_Rate / Seek_Error_Rate churn is benign — only reallocated/pending/uncorrectable sectors or an explicit FAILED verdict count. This killed a recurring false disk-failure P1.
• The log stack shouldn't log about itself. A LogQL negative-regex noise filter strips Loki/Promtail's own Go logfmt chatter from every query, so the model sees application signal, not the observability tooling narrating.
• Resolution is real. Prompt-engineered the model out of re-filing already-closed issues as ONGOING — "absence of evidence for a fixed problem is evidence it's fixed."

Operational rigor

• Scheduled, not on-demand. A cron-driven script runs 6-hourly, daily, and weekly full-infra reports plus an hourly nginx security watch (4xx/5xx, scanner probes, auth failures).
• Push alerting with escalation. The model is required to emit a one-line SUMMARY: first line; the script parses it and raises ntfy notification priority to high whenever a P1 is present.
• Cost and latency bounded. Per-host line budgets so a noisy host can't crowd out the others, input caps (~1000 lines / 50k chars, most-recent tail), and explicit Loki (90s) and Anthropic (240s) timeouts.
• Timezone-correct. Internal Loki math stays UTC; human-facing report windows render in Pacific via zoneinfo, with a UTC fallback.
• Deliberate backend choice. Started on a local Ollama model, migrated to the Claude API when the local model couldn't hold the structured-report and ledger-diff contract reliably — a cost-vs-quality call, made and documented.

Why it matters (the wedge)

Everyone in 2026 can point an LLM at a log file. The leverage is in the parts that make it operable: memory so alerts aren't repeated, guards so it doesn't cry wolf, budgets so it doesn't cost a fortune, and schedules so it runs whether or not I'm watching. That's twenty years of build/release discipline aimed at a new tool — and it's the same shape as the AI pipelines I run at work, just on infrastructure I can show you.