> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archil.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Search logs across sandboxes

> Write structured logs from several sandboxes to one Archil disk, then search and aggregate them from your laptop.

This guide runs three instances of a small Python service in [persistent sandboxes](/compute/sandboxes/introduction). Each
instance writes JSON logs with Python's standard `logging` module to its own directory on a shared Archil disk. From your
laptop, you search every instance's logs at once with [`disk.grep`](/compute/search-files) and compute per-instance error
counts with [serverless execution](/compute/serverless-execution) and `jq`.

There's no log shipper, agent, or index to run. The application writes ordinary files, and the search runs next to the data.

## Prerequisites

Export your Archil API key and a region that offers sandboxes (`aws-us-east-1`, `aws-us-west-2`, or `aws-eu-west-1`), then
install the SDK for the language you'll orchestrate from.

```bash theme={null}
export ARCHIL_API_KEY=key-...
export ARCHIL_REGION=aws-us-west-2
```

<CodeGroup>
  ```bash TypeScript theme={null}
  npm init -y && npm pkg set type=module
  npm install disk tsx
  ```

  ```bash Python theme={null}
  pip install archil
  ```
</CodeGroup>

Save each script in this guide in your project directory. Run TypeScript scripts with `npx tsx <name>.ts` and Python
scripts with `python <name>.py`.

## Write the service

The service below stands in for your application: it handles a stream of fake requests and logs one JSON line per request,
with timeouts on `/checkout` logged as errors. Logging is plain `logging` with two additions:

* A formatter that turns each record into one line of JSON, so the logs are easy to grep and to parse.
* A `RotatingFileHandler`, which renames `app.log` to `app.log.1` when it reaches 1 MB and keeps five old files.

Each instance writes to `/mnt/archil/logs/<instance>/`. Pass the instance name in explicitly rather than using the hostname,
because every sandbox reports the same hostname.

```python app.py theme={null}
import json
import logging
import os
import random
import time
from datetime import datetime, timezone
from logging.handlers import RotatingFileHandler

INSTANCE = os.environ["INSTANCE"]
ERROR_RATE = float(os.environ.get("ERROR_RATE", "0.02"))
LOG_DIR = f"/mnt/archil/logs/{INSTANCE}"


class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "ts": datetime.fromtimestamp(record.created, timezone.utc).isoformat(timespec="milliseconds"),
            "level": record.levelname,
            "instance": INSTANCE,
            "msg": record.getMessage(),
            **getattr(record, "fields", {}),
        }
        return json.dumps(entry)


os.makedirs(LOG_DIR, exist_ok=True)
handler = RotatingFileHandler(f"{LOG_DIR}/app.log", maxBytes=1_000_000, backupCount=5)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("app")
log.addHandler(handler)
log.setLevel(logging.INFO)

log.info("started")
while True:
    path = random.choice(["/", "/search", "/cart", "/checkout"])
    ms = round(random.lognormvariate(3, 0.6))
    if path == "/checkout" and random.random() < ERROR_RATE * 4:
        log.error("upstream timeout", extra={"fields": {"path": path, "status": 504, "ms": ms + 3000}})
    else:
        log.info("request", extra={"fields": {"path": path, "status": 200, "ms": ms}})
    time.sleep(random.uniform(0.005, 0.02))
```

The 1 MB limit is small so that you see a rotation while following this guide; use a larger limit in production.

## Start three instances

Create the disk and three sandboxes that mount it in [conditional mode](/concepts/sharing-disks). Conditional mode sends
writes straight to the server instead of taking an exclusive lock on the disk, so all three instances can write at the same
time, and every write is visible to other clients as soon as it lands. Writes in conditional mode are slower than on an
exclusive mount, which doesn't matter for appending log lines. Each sandbox runs the service as a supervised
[service](/compute/sandboxes/introduction#hosting-network-services), which restarts it if it crashes and starts it again when
the sandbox reboots. `app-3` gets a higher error rate, so there's something to find.

<CodeGroup>
  ```typescript start.ts theme={null}
  import * as archil from "disk";
  import { readFile } from "node:fs/promises";

  const { disk } = await archil.createDisk({ name: "app-logs" });

  for (const name of ["app-1", "app-2", "app-3"]) {
    const sandbox = await archil.createSandbox({
      name,
      baseImage: "python:3.13",
      vcpuCount: 1,
      memSizeMiB: 1024,
      mounts: [{ disk, path: "/mnt/archil", conditional: true }],
    });
    await sandbox.files.uploadFile(await readFile("app.py"), "/root/app.py");
    const errorRate = name === "app-3" ? "0.1" : "0.01";
    await sandbox.exec(
      `archil-sandbox services create app --env INSTANCE=${name} --env ERROR_RATE=${errorRate} -- python3 /root/app.py`,
    );
    console.log(`${name} is running`);
  }
  ```

  ```python start.py theme={null}
  import archil

  disk = archil.create_disk(name="app-logs").disk

  for name in ["app-1", "app-2", "app-3"]:
      sandbox = archil.create_sandbox(
          name=name,
          base_image="python:3.13",
          vcpu_count=1,
          mem_size_mib=1024,
          mounts=[archil.SandboxMountSpec(disk=disk, path="/mnt/archil", conditional=True)],
      )
      sandbox.files.upload_file("app.py", "/root/app.py")
      error_rate = "0.1" if name == "app-3" else "0.01"
      sandbox.exec(
          f"archil-sandbox services create app --env INSTANCE={name} --env ERROR_RATE={error_rate} -- python3 /root/app.py"
      )
      print(f"{name} is running")
  ```
</CodeGroup>

Each instance logs roughly 80 lines per second. Give them a couple of minutes before you search, so the logs rotate at
least once.

## Search every instance at once

`disk.grep` takes an extended regular expression and a directory on the disk. Archil lists the files under that directory and
fans the search out across parallel workers that run next to the data, so you don't download anything. This search finds
every error line, in current and rotated files, across all three instances:

<CodeGroup>
  ```typescript search.ts theme={null}
  import * as archil from "disk";

  const [disk] = await archil.listDisks({ name: "app-logs" });
  const res = await disk.grep({ directory: "logs", pattern: '"level": "ERROR"', recursive: true, maxResults: 10000 });

  console.log(`${res.matches.length} matches in ${res.filesScanned} files (${res.stoppedReason}, ${res.durationMs} ms)`);
  for (const m of res.matches.slice(0, 3)) console.log(`${m.file}:${m.line} ${m.text}`);
  ```

  ```python search.py theme={null}
  import archil

  disk = archil.list_disks(name="app-logs")[0]
  res = disk.grep(directory="logs", pattern='"level": "ERROR"', recursive=True, max_results=10000)

  print(f"{len(res.matches)} matches in {res.files_scanned} files ({res.stopped_reason}, {res.duration_ms} ms)")
  for m in res.matches[:3]:
      print(f"{m.file}:{m.line} {m.text}")
  ```
</CodeGroup>

```
1113 matches in 6 files (completed, 98 ms)
logs/app-3/app.log:8 {"ts": "2026-10-17T13:35:40.941+00:00", "level": "ERROR", "instance": "app-3", "msg": "upstream timeout", "path": "/checkout", "status": 504, "ms": 3026}
logs/app-3/app.log:15 {"ts": "2026-10-17T13:35:41.038+00:00", "level": "ERROR", "instance": "app-3", "msg": "upstream timeout", "path": "/checkout", "status": 504, "ms": 3022}
logs/app-3/app.log:20 {"ts": "2026-10-17T13:35:41.114+00:00", "level": "ERROR", "instance": "app-3", "msg": "upstream timeout", "path": "/checkout", "status": 504, "ms": 3037}
```

`stoppedReason` is `completed` when every file was scanned. A search stops early, and returns a sample of the matches,
when it reaches `maxResults` (1,000 by default) or `maxDurationSeconds` (30 by default). There are more than 1,000 errors
here, so the search raises `maxResults` to 10,000.

Search results are current. Python's file handler flushes after every record, and conditional mode sends each write to the
server, so a line is searchable as soon as the service logs it, without closing or rotating the file. If you use a logger that buffers output in memory, lines become searchable when it flushes.

## Count errors per instance

For questions that need more than matching lines, run a command next to the data with `disk.exec`. The serverless runtime
includes `jq`, and the disk is mounted at `/mnt/archil`, which is also the working directory. This command lists `app-3`'s
log files, then parses every log line from every instance and counts lines and errors per instance:

<CodeGroup>
  ```typescript stats.ts theme={null}
  import * as archil from "disk";

  const [disk] = await archil.listDisks({ name: "app-logs" });
  const { stdout } = await disk.exec(
    `ls -l logs/app-3 && jq -nr '(["instance", "lines", "errors"] | @tsv), ([inputs] | group_by(.instance)[] | [.[0].instance, length, (map(select(.level == "ERROR")) | length)] | @tsv)' logs/*/app.log*`,
  );
  console.log(stdout);
  ```

  ```python stats.py theme={null}
  import archil

  disk = archil.list_disks(name="app-logs")[0]
  res = disk.exec(
      """ls -l logs/app-3 && jq -nr '(["instance", "lines", "errors"] | @tsv), ([inputs] | group_by(.instance)[] | [.[0].instance, length, (map(select(.level == "ERROR")) | length)] | @tsv)' logs/*/app.log*"""
  )
  print(res.stdout)
  ```
</CodeGroup>

```
total 153
-rw-r--r-- 1 root root 246059 Oct 17 13:36 app.log
-rw-r--r-- 1 root root 999989 Oct 17 13:35 app.log.1
instance	lines	errors
app-1	9143	91
app-2	9077	101
app-3	8920	943
```

The `app.log.1` file is the rotated log. Because the glob `logs/*/app.log*` matches rotated files too, the counts cover
everything the instances have logged since they started (up to the five rotated files each one keeps). Serverless commands
mount the disk in shared mode, so they read the logs while the instances keep writing.

## Clean up

Stop and delete the three sandboxes, then delete the disk and the logs on it.

<CodeGroup>
  ```typescript cleanup.ts theme={null}
  import * as archil from "disk";

  for (const sandbox of await archil.listSandboxes()) {
    if (["app-1", "app-2", "app-3"].includes(sandbox.name)) {
      await sandbox.stop();
      await sandbox.delete();
    }
  }
  const [disk] = await archil.listDisks({ name: "app-logs" });
  await disk.delete();
  ```

  ```python cleanup.py theme={null}
  import archil

  for sandbox in archil.list_sandboxes():
      if sandbox.name in ["app-1", "app-2", "app-3"]:
          sandbox.stop()
          sandbox.delete()
  archil.list_disks(name="app-logs")[0].delete()
  ```
</CodeGroup>

## Next steps

* [Search files](/compute/search-files) — concurrency and limits for `disk.grep`
* [Serverless execution](/compute/serverless-execution) — run commands across several disks, read-only or pinned to a subdirectory
* [Sharing disks](/concepts/sharing-disks) — how exclusive, shared, and conditional mounts behave
* [Analytics with DuckDB](/guides/analytics/duckdb) — query Parquet data on an Archil disk with SQL


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.