ls, grep, cat, Python, git, and other file tools on the bucket’s data without downloading it first.
This works with Amazon S3, Google Cloud Storage, Cloudflare R2, and S3-compatible stores such as Tigris, MinIO, Wasabi,
Backblaze B2, and DigitalOcean Spaces. Bringing your own bucket is available on the Developer, Team, and Enterprise
plans. See Plans and limits.
1. Grant Archil access to the bucket
For Amazon S3, attach a bucket resource policy that lets Archil’s IAM role reach the bucket, with anarchilSessionId
condition that matches the disk’s session ID. No long-lived credentials change hands. You can use static IAM credentials instead.
Google Cloud Storage uses HMAC keys, and Cloudflare R2 and other S3-compatible stores use an access key and endpoint.
The S3 Object Storage page has the policy, the principal ARN for each region, and the
setup steps for every provider. The Archil console walks you through the same setup when
you create a disk there: it generates the session ID and a ready-made policy. In Terraform, the session ID is
mount[0].session_id. The SDK examples below use static IAM credentials.
2. Create a disk that points at the bucket
Create the disk in the console, or with the SDK, CLI, or Terraform. SetbucketPrefix to
expose only part of the bucket as the disk’s root.
3. Mount the disk where your agents run
Pick the place your agents run:-
A Linux server, VM, or container: install the client and run
archil mount. See Mount on Linux and Mount in a container.Name the disk by its owner and name, or by its ID (dsk-...). On a personal account, the owner is your email address, for exampleyou@example.com/research-data. -
An Archil sandbox: pass the disk in
mountswhen you create the sandbox. Archil handles mount authentication. See Attaching an Archil disk. -
One command at a time, with no server: run
disk.exec()ornpx disk execagainst the disk. See Serverless execution. - Kubernetes: use the CSI driver.
- Another sandbox provider: see the guides for Daytona, E2B, Modal, Vercel Sandbox, and Cloudflare Containers.
What to expect
- Your bucket stays the system of record. Each file is an ordinary object at its path. Archil also writes
zero-byte directory-marker objects, such as
datasets/out/, and an.archil/entry. POSIX metadata such as mode bits and ownership is tracked by Archil and isn’t visible in the bucket. See POSIX metadata. - Writes are durable on Archil once
fsyncreturns, and copied to the bucket in the background. The disk is strongly consistent, so other Archil clients and the disk’s S3 API read it, not the bucket; only tools that read the bucket directly wait for the copy. Archil copies a file once it has gone 15 seconds without a write, so it usually reaches the bucket about 20 seconds after its last write. A file that keeps changing is copied no later than 10 minutes after its first write that hasn’t reached the bucket. See From the disk to the bucket. - Objects written straight to the bucket appear after Archil lists that directory again. When a directory in use is read and its listing is more than five minutes old, Archil relists it in the background, so new objects usually appear within 5 to 10 minutes. A directory the disk hasn’t listed yet shows them on its first read. See From the bucket to the disk.
- Don’t write the same file from both sides at once. Writing different paths through the bucket and the disk is fine. See Concurrent writes.
- Deleting the disk leaves your data in the bucket. Archil stops synchronizing and discards its cached copy.
- Limits follow the bucket: files up to 5 TiB and paths up to 1,024 bytes on Amazon S3, one data source per disk. See Limits.
- Checkpoints and branches on disks that use your own bucket are available on Enterprise. See Checkpoints on BYOB disks.
Compared with Amazon S3 Files and Mountpoint for Amazon S3
All three let you reach data in an existing bucket through file paths, without moving it. They differ in which file operations work, where you can mount, and when writes reach the bucket.
Choose Amazon S3 Files when your compute is on AWS, you want IAM-governed file access to existing S3 data with S3
Versioning as history, and your compute already isolates the code that runs there.
Choose Mountpoint for Amazon S3 when reads of large objects dominate, such as training data or data lakes, one
client at a time writes new files sequentially, and you want a free client with no service in the data path.
Choose Archil when agents edit files in place, rename directories, or run git, npm, or SQLite on the data; when
several machines write the same tree; when the data is in GCS or R2 or your compute runs outside AWS; or when each
agent needs an isolated sandbox or its own branch next to the data.
For benchmarks and more detail, see Solution Comparison, and on archil.com
Archil vs Amazon S3 Files,
Archil vs Mountpoint for Amazon S3, and
Mounting a POSIX file system over an existing S3 bucket without migrating data.
Related
- Data Sources: how synchronization, consistency, and atomicity work with your bucket
- S3 Object Storage: access setup for each provider
- Isolated per-tenant storage: one disk per tenant, including tenants with their own buckets
- Data ownership and portability