Search files

Archil disks in AWS regions provide a built-in, high-performance search tool that’s effective for searching across millions of files on your disk, up to 10x faster than tools like ripgrep.

import * as archil from "disk";

const d = await archil.getDisk("dsk-abc123");
const { matches } = await d.grep({
  directory: "logs",
  pattern: "ERROR|FATAL",
  recursive: true,
});

import archil

d = archil.get_disk("dsk-abc123")

res = d.grep(directory="logs", pattern="ERROR|FATAL", recursive=True)

Teams building the highest-performance AI agents have discovered that agent performance degrades as you give them more tools to select between at each turn. As a result, teams are increasingly moving away from specialized products like vector databases for context and towards lowest-common denominator infrastructure like file storage. However, existing file system tools like grep are too slow for searching across large document sets. With Archil’s built-in grep APIs, you can effectively search across millions of documents on your disk using map-reduce inside of Archil. Archil’s grep API is priced the same as if you’d manually run the search on serverless sandboxes, billed as the total compute time needed to perform the search.

Managing concurrency

Archil’s grep API works by performing a massively parallel search across the files stored in the folder that you’re searching. Archil automatically manages the concurrency of each search to optimize for search latency. If you’d prefer to control this concurrency manually, you can specify the exact number of workers to use via the concurrency option.

TypeScript

const { matches } = await d.grep({
  directory: "logs",
  pattern: "ERROR|FATAL",
  concurrency: 10,
});

Limiting work

For some kinds of search, you do not need an exhaustive set of results. For example, if you’re trying to find the first occurrence of an error, you only need one result. You can manage the amount of work that Archil’s search performs by specifying either the maxResults or maxDurationSeconds options. When these options are specified, the search will stop as soon as the result or time limit is reached, regardless of whether the search was exhaustive.

Next steps

TypeScript SDK: searching files — the disk.grep API surface
Python SDK: searching files — the Disk.grep API surface
API reference: Search Files — the raw HTTP endpoint
Serverless sandboxes — run arbitrary commands when you need more than search

Getting started

Compute

Concepts

Data Sources

Details

Administration

Protocols

Reference

Legal

Managing concurrency

Limiting work

Next steps

​Managing concurrency

​Limiting work

​Next steps

Managing concurrency

Limiting work

Next steps