Search logs across sandboxes

This guide runs three instances of a small Python service in persistent sandboxes. Each instance writes JSON logs with Python’s standard logging module to its own directory on a shared Archil disk. From your laptop, you search every instance’s logs at once with disk.grep and compute per-instance error counts with serverless execution and jq. There’s no log shipper, agent, or index to run. The application writes ordinary files, and the search runs next to the data.

Prerequisites

Export your Archil API key and a region that offers sandboxes (aws-us-east-1, aws-us-west-2, or aws-eu-west-1), then install the SDK for the language you’ll orchestrate from.

export ARCHIL_API_KEY=key-...
export ARCHIL_REGION=aws-us-west-2

npm init -y && npm pkg set type=module
npm install disk tsx

pip install archil

Save each script in this guide in your project directory. Run TypeScript scripts with npx tsx <name>.ts and Python scripts with python <name>.py.

Write the service

The service below stands in for your application: it handles a stream of fake requests and logs one JSON line per request, with timeouts on /checkout logged as errors. Logging is plain logging with two additions:

A formatter that turns each record into one line of JSON, so the logs are easy to grep and to parse.
A RotatingFileHandler, which renames app.log to app.log.1 when it reaches 1 MB and keeps five old files.

Each instance writes to /mnt/archil/logs/<instance>/. Pass the instance name in explicitly rather than using the hostname, because every sandbox reports the same hostname.

app.py

import json
import logging
import os
import random
import time
from datetime import datetime, timezone
from logging.handlers import RotatingFileHandler

INSTANCE = os.environ["INSTANCE"]
ERROR_RATE = float(os.environ.get("ERROR_RATE", "0.02"))
LOG_DIR = f"/mnt/archil/logs/{INSTANCE}"


class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "ts": datetime.fromtimestamp(record.created, timezone.utc).isoformat(timespec="milliseconds"),
            "level": record.levelname,
            "instance": INSTANCE,
            "msg": record.getMessage(),
            **getattr(record, "fields", {}),
        }
        return json.dumps(entry)


os.makedirs(LOG_DIR, exist_ok=True)
handler = RotatingFileHandler(f"{LOG_DIR}/app.log", maxBytes=1_000_000, backupCount=5)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("app")
log.addHandler(handler)
log.setLevel(logging.INFO)

log.info("started")
while True:
    path = random.choice(["/", "/search", "/cart", "/checkout"])
    ms = round(random.lognormvariate(3, 0.6))
    if path == "/checkout" and random.random() < ERROR_RATE * 4:
        log.error("upstream timeout", extra={"fields": {"path": path, "status": 504, "ms": ms + 3000}})
    else:
        log.info("request", extra={"fields": {"path": path, "status": 200, "ms": ms}})
    time.sleep(random.uniform(0.005, 0.02))

The 1 MB limit is small so that you see a rotation while following this guide; use a larger limit in production.

Start three instances

Create the disk and three sandboxes that mount it in conditional mode. Conditional mode sends writes straight to the server instead of taking an exclusive lock on the disk, so all three instances can write at the same time, and every write is visible to other clients as soon as it lands. Writes in conditional mode are slower than on an exclusive mount, which doesn’t matter for appending log lines. Each sandbox runs the service as a supervised service, which restarts it if it crashes and starts it again when the sandbox reboots. app-3 gets a higher error rate, so there’s something to find.

import * as archil from "disk";
import { readFile } from "node:fs/promises";

const { disk } = await archil.createDisk({ name: "app-logs" });

for (const name of ["app-1", "app-2", "app-3"]) {
  const sandbox = await archil.createSandbox({
    name,
    baseImage: "python:3.13",
    vcpuCount: 1,
    memSizeMiB: 1024,
    mounts: [{ disk, path: "/mnt/archil", conditional: true }],
  });
  await sandbox.files.uploadFile(await readFile("app.py"), "/root/app.py");
  const errorRate = name === "app-3" ? "0.1" : "0.01";
  await sandbox.exec(
    `archil-sandbox services create app --env INSTANCE=${name} --env ERROR_RATE=${errorRate} -- python3 /root/app.py`,
  );
  console.log(`${name} is running`);
}

import archil

disk = archil.create_disk(name="app-logs").disk

for name in ["app-1", "app-2", "app-3"]:
    sandbox = archil.create_sandbox(
        name=name,
        base_image="python:3.13",
        vcpu_count=1,
        mem_size_mib=1024,
        mounts=[archil.SandboxMountSpec(disk=disk, path="/mnt/archil", conditional=True)],
    )
    sandbox.files.upload_file("app.py", "/root/app.py")
    error_rate = "0.1" if name == "app-3" else "0.01"
    sandbox.exec(
        f"archil-sandbox services create app --env INSTANCE={name} --env ERROR_RATE={error_rate} -- python3 /root/app.py"
    )
    print(f"{name} is running")

Each instance logs roughly 80 lines per second. Give them a couple of minutes before you search, so the logs rotate at least once.

Search every instance at once

disk.grep takes an extended regular expression and a directory on the disk. Archil lists the files under that directory and fans the search out across parallel workers that run next to the data, so you don’t download anything. This search finds every error line, in current and rotated files, across all three instances:

import * as archil from "disk";

const [disk] = await archil.listDisks({ name: "app-logs" });
const res = await disk.grep({ directory: "logs", pattern: '"level": "ERROR"', recursive: true, maxResults: 10000 });

console.log(`${res.matches.length} matches in ${res.filesScanned} files (${res.stoppedReason}, ${res.durationMs} ms)`);
for (const m of res.matches.slice(0, 3)) console.log(`${m.file}:${m.line} ${m.text}`);

import archil

disk = archil.list_disks(name="app-logs")[0]
res = disk.grep(directory="logs", pattern='"level": "ERROR"', recursive=True, max_results=10000)

print(f"{len(res.matches)} matches in {res.files_scanned} files ({res.stopped_reason}, {res.duration_ms} ms)")
for m in res.matches[:3]:
    print(f"{m.file}:{m.line} {m.text}")

1113 matches in 6 files (completed, 98 ms)
logs/app-3/app.log:8 {"ts": "2026-10-17T13:35:40.941+00:00", "level": "ERROR", "instance": "app-3", "msg": "upstream timeout", "path": "/checkout", "status": 504, "ms": 3026}
logs/app-3/app.log:15 {"ts": "2026-10-17T13:35:41.038+00:00", "level": "ERROR", "instance": "app-3", "msg": "upstream timeout", "path": "/checkout", "status": 504, "ms": 3022}
logs/app-3/app.log:20 {"ts": "2026-10-17T13:35:41.114+00:00", "level": "ERROR", "instance": "app-3", "msg": "upstream timeout", "path": "/checkout", "status": 504, "ms": 3037}

stoppedReason is completed when every file was scanned. A search stops early, and returns a sample of the matches, when it reaches maxResults (1,000 by default) or maxDurationSeconds (30 by default). There are more than 1,000 errors here, so the search raises maxResults to 10,000. Search results are current. Python’s file handler flushes after every record, and conditional mode sends each write to the server, so a line is searchable as soon as the service logs it, without closing or rotating the file. If you use a logger that buffers output in memory, lines become searchable when it flushes.

Count errors per instance

For questions that need more than matching lines, run a command next to the data with disk.exec. The serverless runtime includes jq, and the disk is mounted at /mnt/archil, which is also the working directory. This command lists app-3’s log files, then parses every log line from every instance and counts lines and errors per instance:

import * as archil from "disk";

const [disk] = await archil.listDisks({ name: "app-logs" });
const { stdout } = await disk.exec(
  `ls -l logs/app-3 && jq -nr '(["instance", "lines", "errors"] | @tsv), ([inputs] | group_by(.instance)[] | [.[0].instance, length, (map(select(.level == "ERROR")) | length)] | @tsv)' logs/*/app.log*`,
);
console.log(stdout);

import archil

disk = archil.list_disks(name="app-logs")[0]
res = disk.exec(
    """ls -l logs/app-3 && jq -nr '(["instance", "lines", "errors"] | @tsv), ([inputs] | group_by(.instance)[] | [.[0].instance, length, (map(select(.level == "ERROR")) | length)] | @tsv)' logs/*/app.log*"""
)
print(res.stdout)

total 153
-rw-r--r-- 1 root root 246059 Oct 17 13:36 app.log
-rw-r--r-- 1 root root 999989 Oct 17 13:35 app.log.1
instance	lines	errors
app-1	9143	91
app-2	9077	101
app-3	8920	943

The app.log.1 file is the rotated log. Because the glob logs/*/app.log* matches rotated files too, the counts cover everything the instances have logged since they started (up to the five rotated files each one keeps). Serverless commands mount the disk in shared mode, so they read the logs while the instances keep writing.

Clean up

Stop and delete the three sandboxes, then delete the disk and the logs on it.

import * as archil from "disk";

for (const sandbox of await archil.listSandboxes()) {
  if (["app-1", "app-2", "app-3"].includes(sandbox.name)) {
    await sandbox.stop();
    await sandbox.delete();
  }
}
const [disk] = await archil.listDisks({ name: "app-logs" });
await disk.delete();

import archil

for sandbox in archil.list_sandboxes():
    if sandbox.name in ["app-1", "app-2", "app-3"]:
        sandbox.stop()
        sandbox.delete()
archil.list_disks(name="app-logs")[0].delete()

Next steps

Search files — concurrency and limits for disk.grep
Serverless execution — run commands across several disks, read-only or pinned to a subdirectory
Sharing disks — how exclusive, shared, and conditional mounts behave
Analytics with DuckDB — query Parquet data on an Archil disk with SQL

Agents

Analytics

Training

Apps & databases

Use Archil on other platforms

Search logs across sandboxes

Prerequisites

Write the service

Start three instances

Search every instance at once

Count errors per instance

Clean up

Next steps

​Prerequisites

​Write the service

​Start three instances

​Search every instance at once

​Count errors per instance

​Clean up

​Next steps

Prerequisites

Write the service

Start three instances

Search every instance at once

Count errors per instance

Clean up

Next steps