> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archil.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agentic observability

> Write application logs straight to an Archil disk, then have an AI agent triage errors on a schedule and build dashboards on request.

Dashboards and alerts only cover the questions someone thought to ask in advance. An agent can ask new ones, as long as it
can read the logs. In this guide, your services write their logs as plain JSON files straight to an Archil disk, and an
agent gets read-only access to them. You'll build two things on top:

* **A triage job** that runs every few minutes. Deterministic SQL finds anomalies, a decision model that costs a fraction
  of a cent per run picks the ones that matter, and a frontier model investigates only those. Quiet runs don't call a
  model at all.
* **Dashboards on request.** Ask a question in plain English and get back a link to a page of charts that answers it,
  plus a script that rebuilds the page without the model.

There's no log pipeline to run: no collector, no ingest service, no index. Apart from Archil and the model, the tools are
open source:

| Job | Project | What it does here |
| - | - | - |
| Query | [DuckDB](https://duckdb.org) | Runs SQL directly over the log files, with no import step |
| Chart | [Vega-Lite](https://vega.github.io/vega-lite/) | Renders dashboard charts to static SVG |
| Agent loop | [AI SDK](https://ai-sdk.dev) | Runs the model with Archil's file, grep, and bash tools |

```mermaid theme={null}
flowchart LR
  S[Your services] -->|one file per service, hour, and host| L[(app-logs disk)]
  L -->|read-only| D[Detect: SQL]
  D -->|anomalies| C[Classify: decision model]
  C -->|real problems| I[Investigate: agent]
  L -->|read-only| B[Dashboard agent]
  I -->|reports, notes| W[(obs-agent disk)]
  B -->|dashboards| W
  W -->|share link| Y[You or Slack]
```

This works well on Archil for a few reasons:

* **Logs are ordinary files,** so the agent uses `grep` and SQL, which it already knows well, instead of a vendor query
  language.
* **Queries run next to the data.** The agent's commands run in [serverless microVMs](https://docs.archil.com/compute/serverless-execution.md) that
  mount the disk, so logs are never downloaded to where the agent runs. In our runs, a DuckDB query across every log file
  returned in 237 ms.
* **The agent can't change the logs.** It mounts the log disk read-only and keeps its notes, reports, and dashboards on a
  second disk.
* **Every instance writes to the same disk.** In [conditional mode](https://docs.archil.com/concepts/sharing-disks.md), any number of hosts can
  write to one disk at the same time, and each line is readable by other clients as soon as it's written.

## Prerequisites

Export your Archil API key, a region that offers sandboxes (`aws-us-east-1`, `aws-us-west-2`, or `aws-eu-west-1`), and an
[AI Gateway](https://vercel.com/ai-gateway) key for the model. Then install the dependencies:

```bash theme={null}
export ARCHIL_API_KEY=key-...
export ARCHIL_REGION=aws-us-west-2
export AI_GATEWAY_API_KEY=...

npm init -y && npm pkg set type=module
npm install disk @archildata/ai-sdk ai zod tsx
```

Save each file in this guide in your project directory, and run scripts with `npx tsx <name>.ts`.

## Write logs to a disk

You'll run a small demo shop in a [persistent sandbox](https://docs.archil.com/compute/sandboxes/introduction.md) that mounts the log disk and
writes to it directly. In production, you mount the disk wherever your services run; see
[Run it in production](#run-it-in-production).

### The log handler

`archil_logging.py` is a standard Python `logging` handler. It writes each record as one JSON line to
`<service>/<YYYY-MM-DD>/<HH>/<host>.ndjson`, with hours in UTC:

* **One directory per hour,** so a query for a time window only reads the hours it needs.
* **One file per host.** When many hosts write to one disk in conditional mode and two of them write the same part of a
  file, the last write wins. Giving every writer its own file avoids that. The host name comes from the `INSTANCE`
  environment variable, or the machine's hostname.
* **Every line has `ts`, `level`, `service`, `host`, and `msg`,** plus whatever fields you pass, so the agent can rely on
  them.
* **Writes happen on a background thread.** `archil_logger` puts records on a queue with Python's `QueueHandler`, and a
  `QueueListener` thread writes them out. In our runs, an append took microseconds, but closing the previous hour's file
  waits for its writes to reach the disk in conditional mode. The queue keeps waits like that off your request path.

```python archil_logging.py theme={null}
import atexit
import json
import logging
import os
import queue
from datetime import datetime, timezone
from logging.handlers import QueueHandler, QueueListener


class HourlyJsonHandler(logging.Handler):
    """Appends each record as one JSON line to <root>/<service>/<YYYY-MM-DD>/<HH>/<host>.ndjson, in UTC."""

    def __init__(self, root, service, host):
        super().__init__()
        self.root, self.service, self.host = root, service, host
        self.path, self.file = None, None

    def emit(self, record):
        ts = datetime.fromtimestamp(record.created, timezone.utc)
        path = f"{self.root}/{self.service}/{ts:%Y-%m-%d/%H}/{self.host}.ndjson"
        if path != self.path:
            if self.file:
                self.file.close()
            os.makedirs(os.path.dirname(path), exist_ok=True)
            self.file, self.path = open(path, "a"), path
        entry = {
            "ts": ts.isoformat(timespec="milliseconds"),
            "level": record.levelname,
            "service": self.service,
            "host": self.host,
            "msg": record.getMessage(),
            **getattr(record, "fields", {}),
        }
        self.file.write(json.dumps(entry) + "\n")
        self.file.flush()


def archil_logger(service, root="/mnt/logs", host=None):
    """A logger whose records are written to the disk by a background thread, so logging never blocks on storage."""
    records = queue.Queue(-1)
    handler = HourlyJsonHandler(root, service, host or os.environ.get("INSTANCE") or os.uname().nodename)
    listener = QueueListener(records, handler)
    listener.start()
    atexit.register(listener.stop)  # write out queued records on exit
    logger = logging.getLogger(service)
    logger.addHandler(QueueHandler(records))
    logger.setLevel(logging.INFO)
    return logger
```

### The demo services

`shop.py` stands in for three services: `api`, `checkout`, and `worker`. Every api request has a `request_id` that also
appears in the checkout service's log line, so the agent can follow a request across services. The worker fails about 8%
of `send_receipt` jobs with an SMTP error. This is the kind of steady background noise that a useful triage job should not
page you for. Two flag files let you cause trouble when you're ready: `/tmp/break-checkout` makes checkout's payment calls
time out, and `/tmp/heic-uploads` makes the worker fail on HEIC images, a new error that doesn't affect anyone's order.

```python shop.py theme={null}
import os
import random
import time
import uuid
from logging import ERROR, INFO

from archil_logging import archil_logger

BREAK_FLAG = "/tmp/break-checkout"
HEIC_FLAG = "/tmp/heic-uploads"
api, checkout, worker = (archil_logger(name) for name in ["api", "checkout", "worker"])


def log(logger, level, msg, **fields):
    logger.log(level, msg, extra={"fields": fields})


def pay(request_id):
    order_id = f"ord_{uuid.uuid4().hex[:10]}"
    amount = random.choice([1299, 2499, 4999, 8900])
    if os.path.exists(BREAK_FLAG) and random.random() < 0.35:
        ms = random.randint(5000, 5100)
        log(checkout, ERROR, "payment authorization failed", request_id=request_id, order_id=order_id,
            amount_cents=amount, provider="payments-gateway", duration_ms=ms,
            error="upstream timeout after 5000ms (pool=payments-gateway, in_use=64/64)")
        return 502, ms
    ms = round(random.lognormvariate(5.0, 0.3))
    log(checkout, INFO, "payment authorized", request_id=request_id, order_id=order_id,
        amount_cents=amount, provider="payments-gateway", duration_ms=ms)
    return 200, ms


while True:
    path = random.choices(["/", "/search", "/cart", "/checkout"], weights=[4, 3, 2, 1])[0]
    request_id = f"req_{uuid.uuid4().hex[:12]}"
    if path == "/checkout":
        status, ms = pay(request_id)
        ms += random.randint(5, 20)
    else:
        status, ms = 200, round(random.lognormvariate(3, 0.5))
    log(api, ERROR if status >= 500 else INFO, "request", method="POST" if path == "/checkout" else "GET",
        path=path, status=status, duration_ms=ms, request_id=request_id, user_id=f"u_{random.randint(1, 5000)}")

    if random.random() < 0.05:
        job = random.choice(["send_receipt", "sync_inventory", "resize_image"])
        if job == "send_receipt" and random.random() < 0.08:
            log(worker, ERROR, "job failed", job=job, attempt=1, error="smtp 550: mailbox unavailable")
        elif job == "resize_image" and os.path.exists(HEIC_FLAG) and random.random() < 0.5:
            log(worker, ERROR, "job failed", job=job, attempt=1, error="unsupported image format: heic (thumbnail skipped)")
        else:
            log(worker, INFO, "job done", job=job, duration_ms=round(random.lognormvariate(4, 0.7)))

    time.sleep(random.uniform(0.01, 0.05))
```

### Start everything

The setup script creates two disks: `app-logs` for the logs and `obs-agent` for the agent's tools and output. It then
starts the sandbox and runs the shop as a supervised [service](https://docs.archil.com/compute/sandboxes/introduction.md#hosting-network-services).
The sandbox mounts `app-logs` in conditional mode, the mode you'd use for a fleet of hosts. `idleTtlSeconds: 0` keeps it
running when no one is connected.

Finally, the script installs the agent's tools on `obs-agent`: the DuckDB command-line binary and Vega. With them, this
small script renders a Vega-Lite chart to SVG:

```javascript vl2svg.mjs theme={null}
// Usage: node vl2svg.mjs < chart.vl.json > chart.svg
import { readFileSync } from "node:fs";
import * as vega from "vega";
import { compile } from "vega-lite";

const spec = JSON.parse(readFileSync(0, "utf8"));
const view = new vega.View(vega.parse(compile(spec).spec), { renderer: "none" });
process.stdout.write(await view.toSVG());
```

```typescript setup.ts theme={null}
import * as archil from "disk";
import { readFile } from "node:fs/promises";

const { disk: logs } = await archil.createDisk({ name: "app-logs" });
const { disk: agent } = await archil.createDisk({ name: "obs-agent" });

// The application host. Its services write their logs straight to the disk.
const sandbox = await archil.createSandbox({
  name: "shop",
  baseImage: "python:3.13",
  vcpuCount: 1,
  memSizeMiB: 1024,
  idleTtlSeconds: 0,
  mounts: [{ disk: logs, path: "/mnt/logs", conditional: true }],
});
for (const file of ["shop.py", "archil_logging.py"]) {
  await sandbox.files.uploadFile(await readFile(file), `/root/${file}`);
}
await sandbox.exec("archil-sandbox services create shop --env INSTANCE=shop-1 -- python3 /root/shop.py");

// The agent's workspace: DuckDB for SQL, and Vega to render charts as SVG.
await agent.putObject("bin/vl2svg.mjs", await readFile("vl2svg.mjs"));
const res = await archil.exec({
  disks: { agent: { disk: agent, conditional: true } },
  command: [
    "cd agent",
    "curl -sfL -o /tmp/duckdb.zip https://github.com/duckdb/duckdb/releases/download/v1.5.6/duckdb_cli-linux-amd64.zip",
    "unzip -oq /tmp/duckdb.zip -d bin",
    "npm install --no-audit --no-fund --silent vega@6.4.0 vega-lite@6.5.0",
    "bin/duckdb --version",
  ].join(" && "),
});
if (res.exitCode !== 0) throw new Error(res.stderr);
console.log(res.stdout);
```

```
v1.5.6 (Variegata) 069cc9f9b5
```

The shop starts logging immediately. After a minute, check that the lines are arriving. This command runs DuckDB in a
serverless microVM with both disks mounted read-only:

```typescript query.ts theme={null}
import * as archil from "disk";

const [logs] = await archil.listDisks({ name: "app-logs" });
const [agent] = await archil.listDisks({ name: "obs-agent" });

const { stdout, timing } = await archil.exec({
  disks: { logs: { disk: logs, readOnly: true }, agent: { disk: agent, readOnly: true } },
  command: `agent/bin/duckdb -c "
    SELECT service, level, count(*) AS lines, max(ts) AS latest
    FROM read_json('logs/*/*/*/*.ndjson', union_by_name=true)
    GROUP BY ALL ORDER BY ALL"`,
});
console.log(stdout, `${timing.totalMs} ms`);
```

```
┌──────────┬─────────┬───────┬─────────────────────────┐
│ service  │  level  │ lines │         latest          │
│ varchar  │ varchar │ int64 │        timestamp        │
├──────────┼─────────┼───────┼─────────────────────────┤
│ api      │ INFO    │   648 │ 2026-11-06 15:37:31.867 │
│ checkout │ INFO    │    61 │ 2026-11-06 15:37:31.774 │
│ worker   │ INFO    │    29 │ 2026-11-06 15:37:31.619 │
└──────────┴─────────┴───────┴─────────────────────────┘
 237 ms
```

## Give the agent the logs

`createDiskTools` turns a [workspace](https://docs.archil.com/integrations/ai-sdk.md) of Archil disks into AI SDK tools: `read_file`, `write_file`,
`delete_file`, `list_files`, `glob`, `grep`, and `run_bash`. This workspace mounts `app-logs` read-only at `/logs` and
`obs-agent` at `/agent`. `run_bash` runs each command in a fresh serverless microVM with both disks mounted, and `grep`
searches files in parallel next to the data with [`disk.grep`](https://docs.archil.com/compute/search-files).

The instructions describe the log layout and how to query it. Both agents in this guide share this module:

```typescript agent.ts theme={null}
import * as archil from "disk";
import { createDiskTools } from "@archildata/ai-sdk";

export const [logs] = await archil.listDisks({ name: "app-logs" });
export const [agentDisk] = await archil.listDisks({ name: "obs-agent" });

// Logs are mounted read-only: the agent can read every line but can't change one.
export const tools = createDiskTools(
  archil.workspace({
    logs: { disk: logs, readOnly: true },
    agent: { disk: agentDisk, conditional: true },
  }),
);

export const model = "anthropic/claude-sonnet-5.5";

// Each agent step resends the conversation so far. Caching bills the repeated part at a fraction of the input price.
export const providerOptions = { gateway: { caching: "auto" as const } };

export const instructions = `
You are the on-call engineer for a web shop with three services: api, checkout, and worker.

Logs are under /logs/<service>/<YYYY-MM-DD>/<HH>/<host>.ndjson, partitioned by UTC hour. Each line is one JSON object
with ts (UTC), level, service, msg, host, and service-specific fields. request_id links an api request to the
checkout call it made. /logs is read-only. Your own files live under /agent.

Query logs with SQL in run_bash, from the workspace root:
  agent/bin/duckdb -c "SELECT ... FROM read_json('logs/*/2026-11-06/*/*.ndjson', union_by_name=true)"
Glob only the days and hours you need. Use the grep tool to find specific lines.
Base every claim on a query or a log line you have read, and quote request_ids or timestamps as evidence.
`.trim();
```

## Triage errors on a schedule

Most triage runs find nothing, and a frontier model is an expensive way to find nothing. So triage runs in three steps,
and each step runs only if the one before it found something:

1. **Detect** with deterministic SQL. A DuckDB query compares the last 15 minutes with the hour before. It flags error
   types that are new or at least twice as common as before, and paths whose p95 latency at least doubled. It runs in one
   serverless call, next to the data, with no model.
2. **Classify** with a decision model. [`experimental_decide`](https://ai-sdk.dev/docs/reference/ai-sdk-core/decide) asks a small decision model whether
   each anomaly needs an investigation, and gets back a typed answer with probabilities instead of prose. This guide uses
   `typesafe-ai/jev` through AI Gateway. Any decision model in the gateway works.
3. **Investigate** with a frontier model, only for what the classifier flagged. The agent from the previous section
   confirms the problem, measures its impact, traces requests across services, writes a report, and records the issue in
   `/agent/triage/known-issues.md`. The classifier reads those notes on later runs, so an issue that's already been
   reported isn't investigated again unless it gets worse.

The detection query groups errors by fingerprint: the message and error text with numbers masked, so
`smtp 550` and `smtp 551` count as one error type:

```sql detect.sql theme={null}
-- Compare the last 15 minutes with the hour before it. Returns one row per anomaly, or nothing.
-- triage.ts fills in the list of log files for the current and previous two hours.
SET VARIABLE t1 = (now() AT TIME ZONE 'UTC')::TIMESTAMP;
SET VARIABLE t0 = getvariable('t1') - INTERVAL 15 MINUTE;
SET VARIABLE b0 = getvariable('t0') - INTERVAL 60 MINUTE;

CREATE TEMP TABLE events AS
SELECT ts, service, path, duration_ms, ts >= getvariable('t0') AS recent,
       level = 'ERROR' OR coalesce(status, 0) >= 500 AS is_error,
       -- An error's fingerprint is its message with numbers masked, so repeats group together.
       msg || coalesce(': ' || regexp_replace(error, '[0-9]+', 'N', 'g'), '') AS fingerprint,
       coalesce(error, msg) AS example
FROM read_json([FILES], union_by_name=true)
WHERE ts >= getvariable('b0') AND ts < getvariable('t1');

WITH volume AS (
  SELECT service, count(*) FILTER (recent) AS recent_n, count(*) FILTER (NOT recent) AS baseline_n
  FROM events GROUP BY service
),
errors AS (
  SELECT service, fingerprint, any_value(example) AS example, min(ts) FILTER (recent) AS first_seen,
         count(*) FILTER (recent) AS recent_errors, count(*) FILTER (NOT recent) AS baseline_errors
  FROM events WHERE is_error GROUP BY ALL
),
latency AS (
  SELECT service, path,
         quantile_cont(duration_ms, 0.95) FILTER (recent) AS recent_p95,
         quantile_cont(duration_ms, 0.95) FILTER (NOT recent) AS baseline_p95,
         count(*) FILTER (recent) AS recent_n, count(*) FILTER (NOT recent) AS baseline_n
  FROM events WHERE duration_ms IS NOT NULL GROUP BY ALL
)
-- Errors that are new, or whose share of the service's lines at least doubled
SELECT 'errors' AS kind, service, fingerprint AS signal, example, first_seen,
       round(100 * recent_errors / recent_n, 2) AS recent_pct,
       round(100 * baseline_errors / nullif(baseline_n, 0), 2) AS baseline_pct,
       recent_errors AS recent_count, baseline_errors AS baseline_count
FROM errors JOIN volume USING (service)
WHERE recent_errors >= 5
  AND (baseline_errors = 0 OR recent_errors / recent_n >= 2 * baseline_errors / baseline_n)
UNION ALL
-- p95 latency that at least doubled on a path with enough traffic to compare
SELECT 'latency', service, coalesce(path, '(all)'), NULL, NULL,
       round(recent_p95), round(baseline_p95), recent_n, baseline_n
FROM latency
WHERE recent_n >= 20 AND baseline_n >= 20 AND recent_p95 >= 2 * baseline_p95
ORDER BY kind, recent_count DESC;
```

`triage.ts` runs the three steps and prints the cost of each model call, which AI Gateway reports with every response.
The investigation returns structured output, validated against a [Zod](https://zod.dev) schema, so you can route issues
by severity or open tickets from them without parsing prose. If `SLACK_WEBHOOK_URL` is set, the script also posts each
issue to that [incoming webhook](https://api.slack.com/messaging/webhooks).

```typescript triage.ts theme={null}
import { experimental_decide as decide, Output, ToolLoopAgent, isStepCount } from "ai";
import * as archil from "disk";
import { readFile } from "node:fs/promises";
import { z } from "zod";
import { agentDisk, instructions, logs, model, providerOptions, tools } from "./agent";

const cost = (metadata: any) => Number(metadata?.gateway?.cost ?? 0);

// 1. Detect: deterministic SQL next to the data. No model.
const sql = await readFile("detect.sql", "utf8");
const gate = await archil.exec({
  disks: { logs: { disk: logs, readOnly: true }, agent: { disk: agentDisk, readOnly: true } },
  command: `files=$(for h in 0 1 2; do ls logs/*/$(date -u -d "-$h hour" +%F/%H)/*.ndjson 2>/dev/null; done | sed "s/.*/'&'/" | paste -sd, -)
[ -n "$files" ] || exit 0
sed "s|\\[FILES\\]|[$files]|" <<'SQL' | agent/bin/duckdb -json
${sql}
SQL`,
});
if (gate.exitCode !== 0) throw new Error(gate.stderr);
const anomalies: Record<string, string | number | null>[] = gate.stdout.trim() ? JSON.parse(gate.stdout) : [];
console.log(`detect: ${anomalies.length} anomalies in ${gate.timing.totalMs} ms`);
if (anomalies.length === 0) process.exit(0);

// 2. Classify: a decision model decides which anomalies deserve an investigation.
const known = await agentDisk.getObject("triage/known-issues.md").then((b) => new TextDecoder().decode(b), () => "");
const decision = await decide({
  model: "typesafe-ai/jev",
  state: [
    { type: "text", text: `Issues already reported, with their rates:\n${known || "(none)"}` },
    { type: "json", value: anomalies },
  ],
  questions: Object.fromEntries(
    anomalies.map((anomaly, i) => [
      `anomaly_${i}`,
      {
        type: "choice",
        instructions: `Does this anomaly from a web shop's logs need an engineer to investigate? ${JSON.stringify(anomaly)}`,
        criteria: {
          investigate: "Customers are likely affected (failed requests, payments, or orders, or much slower responses), and it is new or clearly worse than an issue already reported.",
          ignore: "Little or no customer impact, such as a skipped background task or a rejected email, or an issue already reported at a similar rate.",
        },
      } as const,
    ]),
  ),
});
const flagged = anomalies.filter((anomaly, i) => {
  const answer = decision.answers[`anomaly_${i}`];
  const p = answer.probabilities?.[answer.choice];
  console.log(`classify: ${answer.choice} (${p}) ${anomaly.kind} ${anomaly.service} ${anomaly.signal}`);
  return answer.choice === "investigate";
});
console.log(`classify: $${cost(decision.providerMetadata).toFixed(6)}`);
if (flagged.length === 0) process.exit(0);

// 3. Investigate: a frontier model digs into what the first two steps flagged.
const Issue = z.object({
  severity: z.enum(["critical", "warning"]),
  service: z.string(),
  title: z.string().describe("One line a human can act on, e.g. 'Checkout payments timing out'"),
  started: z.string().describe("UTC timestamp of the first occurrence"),
  impact: z.string().describe("Quantified impact, e.g. '31% of checkouts failing (412 of 1,320)'"),
  evidence: z.array(z.string()).describe("Two or three log lines or query results that prove it"),
  nextStep: z.string(),
});

const agent = new ToolLoopAgent({
  model,
  instructions,
  tools,
  providerOptions,
  stopWhen: isStepCount(40),
  output: Output.object({ schema: z.object({ issues: z.array(Issue), summary: z.string() }) }),
});

const now = new Date().toISOString();
const result = await agent.generate({
  prompt: `
It is ${now}. An anomaly detector compared the last 15 minutes of logs with the hour before and flagged:
${JSON.stringify(flagged, null, 2)}

Investigate each one: confirm it, find when it started, quantify how many requests, jobs, or users it affects, and check
whether other services are involved. Trace a few request_ids across services. Anomalies with the same cause are one issue.
Report only these anomalies. A classifier has already judged the detector's other findings to be low-impact.
Write a short Markdown report to /agent/triage/reports/${now.slice(0, 16).replace(":", "")}.md, and add each issue to
/agent/triage/known-issues.md with its current rate.`,
});

const { issues, summary } = result.output;
const usage = result.totalUsage;
const spent = result.steps.reduce((sum, step) => sum + cost(step.providerMetadata), 0);
console.log(
  `investigate: ${result.steps.length} steps, ${usage.inputTokens} input tokens ` +
    `(${usage.inputTokenDetails.cacheReadTokens ?? 0} cached), $${spent.toFixed(4)}\n`,
);
console.log(summary);
for (const issue of issues) {
  console.log(`\n[${issue.severity}] ${issue.service}: ${issue.title}`);
  console.log(`  started ${issue.started}; ${issue.impact}`);
  for (const line of issue.evidence) console.log(`  - ${line}`);
  console.log(`  next: ${issue.nextStep}`);
}

if (issues.length > 0 && process.env.SLACK_WEBHOOK_URL) {
  const text = issues.map((i) => `*[${i.severity}] ${i.service}: ${i.title}*\n${i.impact}\n${i.nextStep}`);
  await fetch(process.env.SLACK_WEBHOOK_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text: text.join("\n\n") }),
  });
}
```

Let the shop run for an hour so the detector has a baseline, then run triage:

```
detect: 0 anomalies in 553 ms
```

The worker's SMTP failures are steady, so the detector doesn't flag them, and no model is called.

Now cause two problems at once, a real incident and a harmless new error:

```typescript break.ts theme={null}
import * as archil from "disk";

const sandbox = (await archil.listSandboxes()).find((s) => s.name === "shop")!;
// Break checkout, and at the same time start a new, harmless error: users uploading HEIC images.
await sandbox.exec("touch /tmp/break-checkout /tmp/heic-uploads");
console.log(`checkout broken at ${new Date().toISOString()}`);
```

Run `triage.ts` again a couple of minutes later:

```
detect: 5 anomalies in 545 ms
classify: investigate (0.81) errors api request
classify: investigate (0.92) errors checkout payment authorization failed: upstream timeout after Nms (pool=payments-gateway, in_use=N/N)
classify: ignore (0.9) errors worker job failed: unsupported image format: heic (thumbnail skipped)
classify: investigate (0.79) latency api /checkout
classify: investigate (0.85) latency checkout (all)
classify: $0.000068
investigate: 9 steps, 948513 input tokens (824558 cached), $0.4653

All four detector anomalies (api errors, checkout errors, api /checkout latency, checkout latency) share one cause. Checkout calls to payments-gateway time out after 5000ms because the pool is full (in_use=64/64). The first failure was at 16:41:32.783, and about a third of checkouts have failed since, with no sign of recovery. The report is at /agent/triage/reports/2026-11-06T1648.md and the issue is added to /agent/triage/known-issues.md.

[critical] checkout: Checkout payments timing out: payments-gateway pool exhausted (api 502s and latency are the same issue)
  started 2026-11-06 16:41:32.783; 410 of 1,241 POST /checkout failed (33.0%) from 16:41:30 to 16:48:00. 395 of 1,096 checkout users were affected, with about $17,847 in failed orders. The last 3 minutes were 201 of 585 failed (34.4%), so it is not recovering. Failed requests take about 5.06s, while successful ones are unchanged (p50 162ms). Other api endpoints are unaffected.
  - checkout 2026-11-06 16:41:32.783 ERROR payment authorization failed req_ece8f904342e ord_fad33359b9 amount=1299 error='upstream timeout after 5000ms (pool=payments-gateway, in_use=64/64)'.
  - 410 of 410 api 502s have a checkout ERROR with the same request_id, and no api 200 has one. Before 16:41:30 there were 0 api 5xx in 128,046 requests and 0 checkout errors since 15:37. Checkout errors per minute from 16:41 to 16:47: 30, 47, 64, 68, 72, 74, 55.
  next: Check payments-gateway status and latency with the provider. Inspect checkout's gateway connection pool for stuck or leaked connections, and look at what changed around 16:41. If nothing clear turns up, restart checkout to reset the pool. As a stopgap, raise the pool size or lower the 5s timeout so requests fail fast.
```

The detector flagged five anomalies in about half a second. The classifier sent the four related to checkout to
investigation and dropped the HEIC errors, for less than a hundredth of a cent. The investigation found that the four
share one cause, found the first failing request to the millisecond, and matched every api error to its checkout error
by `request_id`. It took 9 steps and \$0.47, with most of its input served from the gateway's cache. The model writes its
own summary, so your wording and numbers will differ.

Run triage again while checkout is still broken. The classifier finds the issue in the notes and stops there:

```
detect: 5 anomalies in 499 ms
classify: ignore (0.61) errors api request
classify: ignore (0.7) errors checkout payment authorization failed: upstream timeout after Nms (pool=payments-gateway, in_use=N/N)
classify: ignore (0.93) errors worker job failed: unsupported image format: heic (thumbnail skipped)
classify: ignore (0.67) latency api /checkout
classify: ignore (0.68) latency checkout (all)
classify: $0.000078
```

To run triage continuously, schedule `triage.ts` every few minutes with any scheduler, such as cron or a scheduled GitHub
Actions workflow. Quiet runs cost one serverless call:

```bash theme={null}
*/5 * * * * cd /path/to/project && npx tsx triage.ts >> triage.log 2>&1
```

## Build dashboards on request

The dashboard agent explores the logs, and then writes `build.sh`, a script that queries the logs with DuckDB, renders each
chart to SVG, and writes a static `index.html`. Then it runs the script and checks the page. The script uses time windows
relative to when it runs, so the page can be rebuilt later without the model.

The script shares the page with a [share URL](https://docs.archil.com/sdks/typescript.md#share-urls). Anyone with the link can open it, with no
Archil account, until it expires (24 hours by default, up to 7 days). The link serves the page with scripts disabled, so
the charts are rendered to SVG ahead of time rather than in the browser.

```typescript dashboard.ts theme={null}
import { Output, ToolLoopAgent, isStepCount } from "ai";
import { z } from "zod";
import { agentDisk, instructions, model, providerOptions, tools } from "./agent";

const question = process.argv.slice(2).join(" ");

const agent = new ToolLoopAgent({
  model,
  instructions,
  tools,
  providerOptions,
  stopWhen: isStepCount(40),
  output: Output.object({
    schema: z.object({
      slug: z.string().describe("Short kebab-case name of the dashboard directory"),
      findings: z.string().describe("Two or three sentences answering the question"),
    }),
  }),
});

const { output, steps } = await agent.generate({
  prompt: `
Build a dashboard that answers: "${question}"

Explore the logs first, then write /agent/dashboards/<slug>/build.sh. When it runs from the workspace root, it must
query the logs with agent/bin/duckdb -json, render 2 to 4 Vega-Lite charts to SVG with
node agent/bin/vl2svg.mjs < spec.json > chart.svg, and write /agent/dashboards/<slug>/index.html: one static page with
a title, a one-line takeaway above each chart, and the SVGs inlined. The page is served with scripts disabled, so don't
use JavaScript. Give charts a fixed width of about 800 pixels and keep each one under 500 data points.
Use time windows relative to now, so the script still works when it runs again later.
Run build.sh, then read index.html back to check it.`,
});

const share = await agentDisk.share(`dashboards/${output.slug}/index.html`, { expiresIn: 7 * 24 * 3600 });
const spent = steps.reduce((sum, step) => sum + Number((step.providerMetadata?.gateway as any)?.cost ?? 0), 0);
console.log(`${steps.length} steps, $${spent.toFixed(4)}\n`);
console.log(output.findings);
console.log(`\n${share.url}`);
```

```bash theme={null}
npx tsx dashboard.ts "How is checkout doing? Show the error rate and latency over the last 30 minutes, and which users are affected."
```

```
13 steps, $0.6318

Checkout is failing right now. In the 30 minutes to 16:53Z (16:23–16:53), 743 of 5,935 checkouts failed (12.5%). There
were no errors before 16:41:32Z. Since then about a third fail, and the last 5 minutes are at 34.1%. [...] Successful
checkouts are unchanged (p50 149ms, p95 244ms), but failures wait about 5,048ms, which pushes overall p95 to 5,059ms.
681 of 3,465 users who tried to check out (19.7%) hit at least one failure, about $31.5k in orders, and 225 of them failed
on every attempt. [...]

https://control.green.us-west-2.aws.prod.archil.com/api/shared/eyJkIjoiZnMt...
```

![Checkout health dashboard. The share of failed checkouts per minute jumps from 0 to about 35% at 16:41 UTC and drops back to 0 after 16:54. Payment authorization p95 jumps from about 250 ms to 5 seconds over the same minutes while p50 stays near 150 ms. A third chart shows 30 to 70 users a minute hitting a failed checkout during the incident.](https://mintcdn.com/regattastorage/POnfsiHp-CqvMnhg/images/guides/agentic-observability-dashboard.png?fit=max\&auto=format\&n=POnfsiHp-CqvMnhg\&q=85\&s=38e8834a87171edd069b43ad92749444)

The top of the dashboard the agent built, rebuilt without the model after checkout recovered.

The agent decided what to chart on its own. The run took about two minutes. A dashboard is a question a person asked,
so it goes straight to the frontier model, unlike triage.

### Rebuild a dashboard without the model

`build.sh` is an ordinary script on the disk, so rebuilding a dashboard is one serverless call, with no model:

```typescript refresh.ts theme={null}
import * as archil from "disk";

const slug = process.argv[2];
const [logs] = await archil.listDisks({ name: "app-logs" });
const [agent] = await archil.listDisks({ name: "obs-agent" });

const res = await archil.exec({
  disks: {
    logs: { disk: logs, readOnly: true },
    agent: { disk: agent, conditional: true },
  },
  command: `bash agent/dashboards/${slug}/build.sh`,
});
if (res.exitCode !== 0) throw new Error(res.stderr);
console.log(`rebuilt in ${res.timing.totalMs} ms`);
console.log((await agent.share(`dashboards/${slug}/index.html`)).url);
```

```bash theme={null}
npx tsx refresh.ts checkout-health
```

```
rebuilt in 3885 ms
https://control.green.us-west-2.aws.prod.archil.com/api/shared/eyJkIjoiZnMt...
```

The script recomputes the takeaways as well as the charts. After checkout recovered, the rebuilt page's first takeaway
read "the last 5 minutes are at 1.6%, so it has mostly recovered." Rebuild a dashboard on a schedule to keep it current,
or ask the agent for a new one when the question changes.

## Run it in production

Mount the `app-logs` disk in conditional mode wherever your services run, and log through `archil_logger`. The agent
scripts stay the same.

> [tab] Kubernetes
>
> Mount the disk with the [CSI driver](https://docs.archil.com/reference/csi-driver.md) as a `ReadWriteMany` volume, which mounts it in
> conditional mode. Create the PersistentVolume and a PersistentVolumeClaim named `app-logs` as shown in
> [Static provisioning](https://docs.archil.com/reference/csi-driver.md#static-provisioning), then mount the claim into each service's pods:
>
> ```yaml theme={null}
> spec:
>   template:
>     spec:
>       containers:
>         - name: api
>           volumeMounts:
>             - name: app-logs
>               mountPath: /mnt/logs
>       volumes:
>         - name: app-logs
>           persistentVolumeClaim:
>             claimName: app-logs
> ```
>
> A pod's hostname is its name, so each replica writes its own files without setting `INSTANCE`.

> [tab] VMs
>
> [Mount the disk](https://docs.archil.com/mounting/linux.md) in conditional mode on each host. Each host writes its own files, named after its
> hostname.
>
> ```bash theme={null}
> sudo archil mount <disk-id> /mnt/logs --region aws-us-west-2 --conditional
> ```

A few things to plan for:

* **Several processes per host.** If a host runs several processes that log, such as Gunicorn workers, give each one its
  own file: pass `host=f"{socket.gethostname()}-{os.getpid()}"` to `archil_logger`.

* **Other languages.** Any logger can write the same layout: one JSON object per line with `ts`, `level`, `service`,
  `host`, and `msg`, appended to `<service>/<YYYY-MM-DD>/<HH>/<host>.ndjson`, one file per process, written off the
  request path.

* **Retention.** Each day is its own directory, so retention is a daily delete. This serverless command removes the
  logs from 30 days ago. It mounts the disk in conditional mode, so it can delete while your services keep writing:

  ```typescript theme={null}
  await archil.exec({
    disks: { logs: { disk: logs, conditional: true } },
    command: "rm -rf logs/*/$(date -u -d '30 days ago' +%F)",
  });
  ```

* **Big log volumes.** Tell the agent to glob only the hours it needs (the instructions in `agent.ts` already do) so its
  queries read only those files. For long-range questions, have a scheduled job compact older hours into Parquet with
  DuckDB's `COPY ... TO`.

* **Sensitive data.** The agent and anyone with a share link can see what's in the logs. Remove secrets and personal data
  before you log them.

## Clean up

Delete the sandbox and both disks:

```typescript cleanup.ts theme={null}
import * as archil from "disk";

const sandbox = (await archil.listSandboxes()).find((s) => s.name === "shop")!;
await sandbox.stop();
await sandbox.delete();
for (const name of ["app-logs", "obs-agent"]) {
  const [disk] = await archil.listDisks({ name });
  await disk.delete();
}
```

## Next steps

* [AI SDK integration](https://docs.archil.com/integrations/ai-sdk.md): mount several disks as one workspace and choose what the agent can reach
* [Search files](https://docs.archil.com/compute/search-files.md): limits and concurrency for the parallel search behind the `grep` tool
* [Search logs across sandboxes](https://docs.archil.com/guides/analytics/log-search.md): search the same kind of logs with `disk.grep` and `jq`, without an agent
* [Analytics with DuckDB](https://docs.archil.com/guides/analytics/duckdb.md): query Parquet in an S3 bucket and serve results over HTTPS
* [Sharing disks](https://docs.archil.com/concepts/sharing-disks.md): how exclusive, shared, and conditional mounts behave


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.