# Best practices
Source: https://docs.archil.com/administration/best-practices
Recommended ways to structure your Archil accounts, disks, and credentials for production and multi-tenant workloads.
This page collects recommendations for running Archil in production — how to structure accounts, disks, and
credentials so that workloads and tenants stay isolated, and billing stays manageable as you scale.
## Use a disk as your unit of isolation
A disk is Archil's unit of access isolation. Credentials — whether [disk tokens or IAM
authorization](/concepts/disk-users) — are scoped to an entire disk, including all of its
[branches and checkpoints](/concepts/branches-and-checkpoints). Anyone who can mount a disk can read everything
on it, and POSIX permissions are not a security boundary between untrusted users.
So when you need to keep workloads or customers from accessing each other's data:
* **Give each tenant its own disk** (or set of disks) rather than sharing one disk across tenants, and issue
per-tenant, disk-scoped [disk tokens](/concepts/disk-users).
* **Don't use branches as a security boundary.** A branch is an alternate history of the same disk, not an
isolated disk — it shares the parent disk's credentials.
* **Don't rely on POSIX permissions** to separate untrusted tenants on a shared disk.
## Give each agent or job its own fork
For AI-agent and CI fan-out workloads, run each agent or job against its own writable
[branch](/concepts/branches-and-checkpoints) — or its own disk when the work is untrusted. Branches are
copy-on-write and fully independent: each can be mounted on a different VM, writes are isolated from the parent
and siblings, and you're only billed for the unique data each fork adds (see [Metering](/concepts/metering)).
This gives every agent a clean, isolated working copy without duplicating the base data set.
When forks must be isolated from each other for security — for example, one end customer per fork — give each
its own disk instead, since a branch shares the parent disk's credentials.
## Structure accounts and billing
Use [organizations](/administration/organizations) to group disks by team or environment and to keep billing
separate from personal usage. A common setup is a separate **Dev** and **Production** organization, so experimental
disks and credentials stay isolated from production data and you can manage access and billing for each
independently. If you operate Archil on behalf of your own customers, you can likewise map multiple Archil
organizations to your internal boundaries: multiple organizations can be billed under a single contract, so
splitting work across organizations — whether by environment or by tenant — doesn't fragment your billing.
[Contact us](mailto:support@archil.com) to set up a single contract across multiple organizations.
# Pricing & Billing
Source: https://docs.archil.com/administration/billing
Plans, invoicing, usage monitoring, and cost management for Archil
Archil charges based on active data — the data currently cached in our high-speed storage layer.
There are no charges for API calls, egress, or data transfer. For details on how active data is
measured, see [How Metering Works](/concepts/metering).
## Plans
**\$0.20 / GiB / month**
* Usage-based, pay as you go
* Community + email support
* Credit card via Stripe
* No credit card required to start
[Get started →](https://console.archil.com)
**Custom pricing**
* Volume discounts and committed spend
* Dedicated support
* Invoice-based payment (net 30)
* SLAs, DPAs, and BAAs included
[Contact us →](mailto:support@archil.com)
Use the [pricing calculator](https://archil.com/pricing) to estimate your costs.
## What you're charged for
You pay for the time-weighted average of active data across all your disks over each billing
period. That's it — no hidden fees.
* **No egress fees** — reads from your disk are free
* **No API call charges** — control plane operations are free
* **No per-request fees** — mount, unmount, and metadata operations are free
## Monitoring usage
### Per-disk usage
Each disk in the [Archil console](https://console.archil.com) has a **Usage** tab showing a
real-time chart of active data over time. You can view usage across multiple time ranges (6 hours,
24 hours, 7 days, 30 days, or all time).
### Account-wide billing
The [Billing](https://console.archil.com/billing) page provides a unified view across all disks
and regions:
* **Current active data** per disk
* **Month-to-date average** and **projected monthly cost** per disk and in total
* **Past billing cycles** with per-disk breakdowns and links to Stripe invoices
## Managing your subscription
You can manage your subscription, update payment methods, and view invoices through the
**Billing** page in the console. Clicking **Manage Billing** opens the Stripe billing portal.
## Enterprise billing
For organizations that need custom pricing, committed spend agreements, volume discounts, or
invoice-based payment (net 30), [contact us](mailto:support@archil.com) to discuss Enterprise
plans.
Enterprise plans also include:
* Dedicated support
* Custom SLA terms
* Cost allocation by team or project upon request
# Organizations
Source: https://docs.archil.com/administration/organizations
Use Organizations to effectively share disks between members of your team
Organizations let you share disks across your team, manage billing separately from personal usage, and
delegate disk administration to team members. If you're working solo, you can skip this — personal accounts
work for everything. Organizations become useful when multiple people need to create, configure, or mount
the same disks.
## Creating an organization
You can create an organization by clicking the "Personal account" dropdown in the top right corner of the screen and
selecting "Create Organization". When you create an organization, the "slug" for that organization will become the prefix
when mounting an organization-owned disk.
For example, if you create an organization with the slug "my-org", you can mount an organization-owned disk with the name "my-org/my-disk".
## Managing disks in an organization
When you create a disk in the Archil console, it will automatically be owned by the organization that is selected in the
top-right corner of the screen.
When you view the list of disks in your Archil account, the list will only show disks that are owned by the organization
that is selected in the top-right corner of the screen.
You can use the dropdown in the top-right corner of the screen to switch between your personal account and any organizations
you belong to.
There is **no limit on the number of disks** you can create in an organization — create as many as you need. Archil is
[fully elastic](/concepts/metering): there's no per-disk fee, and billing is based on usage rather than disk count, so
patterns like a disk per tenant, per workspace, or per agent are expected.
## Managing members in an organization
If you are the administrator of an organization, you can manage the members of the organization by clicking the "Manage" button
next to the organization name in dropdown in the top-right corner of the screen, and navigating to the "Members" tab.
Members can be assigned either a "Member" or an "Administrator" role.
Members with the "Member" role have the ability to view and manage the disks in the organization, including deleting disks
and managing the authorization of disk users.
Members with the "Administrator" role have the same permissions as the "Member" role, but also have the ability to manage
the organization itself, including adding and removing members.
# Add Disk User
Source: https://docs.archil.com/api-reference/disk-users/add-user
POST /api/disks/{id}/users
Adds an authorized user to a disk. Users can authenticate via:
- **token**: A shared token with a nickname and 4-character suffix
- **awssts**: AWS STS role assumption with an IAM principal ARN
# Remove Disk User
Source: https://docs.archil.com/api-reference/disk-users/remove-user
DELETE /api/disks/{id}/users/{userType}
Removes an authorized user from a disk.
# Create Disk
Source: https://docs.archil.com/api-reference/disks/create-disk
POST /api/disks
Creates a new disk with the specified configuration. A default disk
token user is automatically generated and returned in the response,
so the disk is immediately mountable. The one-time disk token appears
in `authorizedUsers[].token` and cannot be retrieved again. (This
disk token is scoped to the new disk and is separate from the API
key you used to make this request.)
To provide your own users instead, pass the deprecated `authMethods`
field or call AddDiskUser after creation.
# Create Share URL
Source: https://docs.archil.com/api-reference/disks/create-share-url
POST /api/disks/{id}/share
Creates a signed, time-limited URL for downloading a file without
authentication. The URL expires after 24 hours by default; pass
`expiresIn` to set a custom lifetime up to 7 days.
The returned URL points to a public endpoint (`GET /api/shared/{token}`)
that verifies the signature, checks expiry, and streams the file with a
sensible `Content-Type` and `Content-Disposition`. Range requests are
supported. No credentials are needed to open the URL.
# Delete Disk
Source: https://docs.archil.com/api-reference/disks/delete-disk
DELETE /api/disks/{id}
Deletes a disk and all associated resources.
# Get Disk
Source: https://docs.archil.com/api-reference/disks/get-disk
GET /api/disks/{id}
Returns detailed information about a specific disk.
# List Disks
Source: https://docs.archil.com/api-reference/disks/list-disks
GET /api/disks
Returns all disks owned by the authenticated account.
# Build Image
Source: https://docs.archil.com/api-reference/images/build-image
POST /api/images
Builds an OCI image for use by sandboxes. Poll [Get Image](/api-reference/images/get-image)
until it is `ready`, then pass its `image_id` when creating a sandbox.
Repeated requests join an active build. Completed tags rebuild; completed digest references
are reused. During a rebuild, sandboxes can use the last successful digest. Submit the request
again to retry a failed or unresponsive build.
# Get Image
Source: https://docs.archil.com/api-reference/images/get-image
GET /api/images/{image_id}
Returns the image's build status and latest successful digest, if available. Failed builds include a `failure_reason`. Use [Build Image](/api-reference/images/build-image) (`POST /api/images`) to retry interrupted builds.
# API Reference
Source: https://docs.archil.com/api-reference/introduction
Programmatic access to the Archil Control Plane
The Archil Control Plane API provides programmatic access to manage disks, mounts, and API keys in the Archil distributed filesystem platform.
**API keys vs. disk tokens.** API keys authenticate requests to this Control Plane API and are scoped to your account. [Disk tokens](/concepts/disk-users#disk-token-authorization) are a separate kind of credential, scoped to a single disk, used by clients when mounting. The two are not interchangeable.
## Base URL
Each region has its own control plane endpoint:
| Region | Region ID | Endpoint |
| - | - | - |
| AWS US East (N. Virginia) | `aws-us-east-1` | `control.green.us-east-1.aws.prod.archil.com` |
| AWS EU West (Ireland) | `aws-eu-west-1` | `control.green.eu-west-1.aws.prod.archil.com` |
| AWS US West (Oregon) | `aws-us-west-2` | `control.green.us-west-2.aws.prod.archil.com` |
| GCP US Central (Iowa) | `gcp-us-central1` | `control.blue.us-central1.gcp.prod.archil.com` |
## Authentication
All endpoints require an API key:
```
Authorization: {API_KEY}
```
Create API keys in the [Archil Console](https://console.archil.com) or via the API.
## Response format
All responses use a consistent envelope:
```json Success Response theme={null}
{
"success": true,
"data": { ... }
}
```
```json Error Response theme={null}
{
"success": false,
"error": "Error message"
}
```
## HTTP status codes
| Code | Description |
| - | - |
| `200` | Success |
| `202` | Accepted; an asynchronous operation started |
| `400` | Bad request (validation error) |
| `403` | Forbidden (auth failure or access denied) |
| `404` | Not found |
| `409` | Resource state conflict |
| `429` | Capacity or concurrency limit reached |
| `500` | Internal server error |
| `503` | Feature unavailable or lifecycle transition temporarily blocked |
## Disk ID format
Disks are identified by a unique ID in the format:
```
dsk-{16 hex characters}
```
For example: `dsk-0123456789abcdef`
## Getting started
Signing up for Archil is free with no credit card required. [Create your free account](https://console.archil.com) to get an API key and start building.
1. [Create an API key](/api-reference/tokens/create-token) in the console or via API
2. Use the key in the `Authorization` header for all requests
3. [Create your first disk](/api-reference/disks/create-disk) or [persistent sandbox](/api-reference/sandboxes/create-sandbox)
# Create Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/create-sandbox
POST /api/sandboxes
Provisions a sandbox VM with the requested shape and an internal,
persistent Archil disk backing its root filesystem. This backing disk
cannot be disabled or replaced with an existing disk. Use `mounts`
to attach additional existing disks at guest paths; omit it or pass
an empty list to create a sandbox without additional disks.
By default the response reports `pending` after
the runtime accepts the start. Set `wait=true` to hold for `running`; if
the wait budget expires, the response remains `pending` and startup
continues.
# Create Sandbox Connection
Source: https://docs.archil.com/api-reference/sandboxes/create-sandbox-connection
POST /api/sandboxes/{sid}/connections
Returns a short-lived WebSocket URL used by the sandbox process API.
The URL contains a signed token that expires at `expires_at`; an open
WebSocket can remain connected after the token expires.
# Create Sandbox Port Token
Source: https://docs.archil.com/api-reference/sandboxes/create-sandbox-port-token
POST /api/sandboxes/{sid}/port-tokens
Creates token-authorized HTTP access to one port. Send the raw token returned
by this request in the `X-Archil-Token` request header at the stable
`-.` hostname. Public ports accept connections
without a token; otherwise a valid token for that sandbox and port is required.
The token is not returned again.
# Delete Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/delete-sandbox
DELETE /api/sandboxes/{sid}
Permanently deletes a stopped, exited, or failed sandbox and its backing disk. A sandbox with child forks cannot be deleted until those children are deleted, starting with the deepest ones.
# Delete Sandbox Port Token
Source: https://docs.archil.com/api-reference/sandboxes/delete-sandbox-port-token
DELETE /api/sandboxes/{sid}/port-tokens/{token_id}
New private connections using the token fail after this request
completes; established connections remain open. Expired tokens return not found.
# Expose Sandbox Port
Source: https://docs.archil.com/api-reference/sandboxes/expose-sandbox-port
PUT /api/sandboxes/{sid}/ports/{port}
Creates or returns public access to a TCP port in the sandbox.
# Fork Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/fork-sandbox
POST /api/sandboxes/{sid}/fork
Checkpoints a running source sandbox and creates an isolated writable
disk branch. Forks from running or paused sources resume from the
source's CPU, memory, and disk state. Forks from stopped sources
cold-boot from the saved disk checkpoint. A running source is
resumed after its checkpoint; paused and stopped sources remain
inactive. Forked sandboxes may themselves be forked.
Passing `checkpoint` forks that saved state without touching the
source, so clients can pause once, wait for the checkpoint, and fork
it from any number of places. Without `checkpoint` the server pauses a
running source inside the request and resumes it afterwards; that path
is kept for older clients and is bounded by the request timeout.
Sandboxes with additional disk mounts must be stopped before forking.
The fork inherits mounts of the same external disks; only the internal
root disk is branched, so external disk contents are not isolated.
# Get Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/get-sandbox
GET /api/sandboxes/{sid}
# Get Sandbox Network
Source: https://docs.archil.com/api-reference/sandboxes/get-sandbox-network
GET /api/sandboxes/{sid}/network
Returns the last committed network policy for a running, paused, or stopped sandbox.
# Get Sandbox Port Token
Source: https://docs.archil.com/api-reference/sandboxes/get-sandbox-port-token
GET /api/sandboxes/{sid}/port-tokens/{token_id}
Returns token metadata without its raw token. Expired tokens return not found.
# List Sandbox Port Tokens
Source: https://docs.archil.com/api-reference/sandboxes/list-sandbox-port-tokens
GET /api/sandboxes/{sid}/port-tokens
Lists unexpired private port token metadata without returning raw tokens.
# List Sandbox Ports
Source: https://docs.archil.com/api-reference/sandboxes/list-sandbox-ports
GET /api/sandboxes/{sid}/ports
Lists the explicitly exposed public ports for the sandbox.
# List Sandboxes
Source: https://docs.archil.com/api-reference/sandboxes/list-sandboxes
GET /api/sandboxes
All the account's sandboxes, oldest first.
# Pause Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/pause-sandbox
POST /api/sandboxes/{sid}/pause
Snapshot the CPU & memory state, then stop the VM. The sandbox reports
`pausing` until the snapshot commit marker is durable and the VM exits,
then reports `paused`. Attached process connections close, while
agent-owned processes remain available for reattachment after resume.
A sandbox that is still `pending` must reach `running` before it can be paused.
To restore the snapshot, call `/resume`.
# Resume Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/resume-sandbox
POST /api/sandboxes/{sid}/resume
Restores a paused sandbox's committed CPU and memory snapshot.
Resuming a running sandbox or a pending resume is idempotent. Only a
`paused` sandbox can begin a new resume; `stopped`, `exited`, `failed`,
`stopping`, `pausing`, or a pending cold start returns 409.
# Set Sandbox Timeouts
Source: https://docs.archil.com/api-reference/sandboxes/set-sandbox-timeout
POST /api/sandboxes/{sid}/timeout
Replaces one or both sandbox timeout settings. Updating `timeout` while
the sandbox is running resets its hard expiration deadline to that many
seconds from now, capped at 24 hours after the current session started.
Updating `idle_ttl_seconds` resets the idle
countdown when no direct process connection is active; zero disables
idle expiry. Omitted settings remain unchanged. Updates are retained
across later stops and starts. For an inactive sandbox, they apply to
the next powered-on session. Sandboxes that are starting or shutting
down, or whose timeout changes concurrently, return 409.
# Start Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/start-sandbox
POST /api/sandboxes/{sid}/start
Cold-boots from the persisted configuration and disks. If the sandbox
was paused, its memory snapshot is consumed and discarded before the
new VM runs; disk state remains. Without mount changes, starting a running
sandbox or a pending cold start is idempotent. A sandbox that is stopping,
pausing, or pending a resume returns 409. Providing `mounts` replaces the
additional disk mounts for this and later sessions and requires an inactive
sandbox; omit it to preserve saved mounts.
# Stop Sandbox
Source: https://docs.archil.com/api-reference/sandboxes/stop-sandbox
POST /api/sandboxes/{sid}/stop
Gracefully shuts down the guest, then checkpoints its disk before
entering `stopped`. The sandbox and any forks from it cold-boot from
disk state. Stopping a paused sandbox reuses its existing disk
checkpoint. In-flight processes receive the guest shutdown signal.
# Unexpose Sandbox Port
Source: https://docs.archil.com/api-reference/sandboxes/unexpose-sandbox-port
DELETE /api/sandboxes/{sid}/ports/{port}
Removes an explicitly exposed public port. An enabled sandbox service publishing the same TCP port remains publicly reachable.
# Update Sandbox Network
Source: https://docs.archil.com/api-reference/sandboxes/update-sandbox-network
PUT /api/sandboxes/{sid}/network
Applies the complete replacement policy to a running sandbox, then
records it as the current policy. A rejected update leaves the previous
policy unchanged. Free-plan accounts cannot update network policies and
receive 403 with code sandbox_network_plan_required. For paid accounts,
an empty object restores unrestricted egress. Existing network sessions
are not terminated; the replacement governs new connections and is
retained across later stops and resumes. Sandboxes
that are not running, or whose policy changes concurrently, return 409.
# Execute Command on Disk
Source: https://docs.archil.com/api-reference/serverless-execution/exec-disk
POST /api/disks/{id}/exec
Launches a container with the specified disk mounted, runs the given
command to completion, shuts down the container, and returns the command
result.
# Execute Command with Multiple Disks
Source: https://docs.archil.com/api-reference/serverless-execution/exec-multi
POST /api/exec
Launches a container with the supplied set of disks each mounted at its
own relative path under `/mnt/archil`, runs the command to completion,
and shuts down the container. Activation is atomic: every disk mounts
or none of them do.
Relative paths must be non-empty, non-absolute, and contain no `.` /
`..` segments. Mounting two disks at the same relative path is an
error.
# Search Files (grep)
Source: https://docs.archil.com/api-reference/serverless-execution/grep-disk
POST /api/disks/{id}/grep
Searches files under a directory on the disk for lines matching a
regular expression. Listing and matching are fanned out across
ephemeral exec containers, so the request finishes within the user's
time budget regardless of the size of the directory.
The user controls cost and latency with three knobs:
- `maxDurationSeconds` is the wall-clock deadline (capped at 30s).
- `concurrency` is the maximum number of parallel grep workers.
Higher concurrency finishes larger datasets within the deadline,
at proportionally more compute.
- `maxResults` causes the search to short-circuit after the
aggregator has collected this many matches.
With `recursive: false` only files directly under `directory` are
searched. With `recursive: true` subdirectories are walked
breadth-first and grep workers are dispatched as soon as each level
finishes listing — listing and matching overlap.
The matches returned when stopping early on `maxResults` are a sample
of whichever workers reported first, not the lexicographically first
N. The response surfaces `stoppedReason` so callers can distinguish
completion from early termination.
# Create API Token
Source: https://docs.archil.com/api-reference/tokens/create-token
POST /api/tokens
Creates a new API token for programmatic access. The full token value
is only returned once at creation time.
# Delete API Token
Source: https://docs.archil.com/api-reference/tokens/delete-token
DELETE /api/tokens/{id}
Revokes and deletes an API token.
# List API Tokens
Source: https://docs.archil.com/api-reference/tokens/list-tokens
GET /api/tokens
Returns all API tokens for the authenticated account.
# Sandbox Firewall
Source: https://docs.archil.com/compute/sandboxes/firewall
Network egress filters for sandboxes.
Archil's persistent sandboxes support defining a network policy to filter network egress from the sandbox. This is useful when running untrusted workloads, including agents.
To prevent platform abuse, network egress is limited to the Team plan and above. On the Developer plan, all network egress is denied. Sandboxes created on the Developer plan keep denying egress after an upgrade until you [replace their policy](#dynamic-updates).
The sandbox firewall operates on both the IP and transport layer, so you can filter both CIDR ranges and domains.
To deny all network egress you can set the `default` to `deny`. On top of this, you can apply additional `deny` and `allow` rules to whitelist certain targets. For example:
```typescript theme={null}
import * as archil from "disk";
const sandbox = await archil.createSandbox({
network: {
egress: {
default: "deny",
allow: ["192.168.14.0/24", "*.github.com"],
}
}
});
```
## Domain filtering
Domain filtering applies to all HTTP traffic (ports 80/443). HTTP/3 is blocked when domain filters or [`drain_on_pause`](#draining-requests-before-pause) selectors are in place. It validates SNI using the TLS ClientHello, the `Host` header, and the URI authority for every request. If any are denied, then the request will be rejected.
To support this, our egress proxy must terminate TLS, so we install an "archil egress" proxy CA into every sandbox. Every process gets `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, and `AWS_CA_BUNDLE` pointing at the system bundle (`/etc/ssl/certs/ca-certificates.crt`) and `NODE_EXTRA_CA_CERTS` pointing at the Archil CA (`/usr/local/share/ca-certificates/archil-egress.crt`). Values you set yourself take precedence, so a custom bundle must include the Archil CA.
## Credential brokering
For HTTP egress, we also support defining rules for transforming requests. This means that secret credentials can live outside the sandbox, so untrusted workloads cannot access them directly. For example, you can add API keys to outgoing HTTP requests so your sandbox never sees them:
```typescript theme={null}
import * as archil from "disk";
const sandbox = await archil.createSandbox({
network: {
egress: {
default: "deny",
allow: [{
target: "api.github.com",
transform: {
headers: {
Authorization: `Bearer ${process.env.GITHUB_TOKEN}`
}
}
}],
}
}
});
```
## Egress forwarding
Use `forward_url` on an allow rule to send matching requests to your own gateway. Applications inside the sandbox keep using the original API hostname, while your gateway can apply logging, usage limits, or request filtering before contacting the upstream service.
```typescript theme={null}
import * as archil from "disk";
const gatewayToken = process.env.EGRESS_GATEWAY_TOKEN;
if (!gatewayToken) throw new Error("Set EGRESS_GATEWAY_TOKEN before creating the sandbox");
const sandbox = await archil.createSandbox({
network: {
egress: {
default: "deny",
allow: [{
target: "api.openai.com",
forward_url: "https://gateway.example.com/archil",
transform: {
headers: {
Authorization: `Bearer ${gatewayToken}`,
},
},
}],
},
},
});
```
With this policy, a request to `https://api.openai.com/v1/responses?trace=1` is sent to `https://gateway.example.com/archil/v1/responses?trace=1`. Archil appends the original path and query to the gateway URL's path prefix and streams the request body and response. Your gateway decides whether to call the original upstream or return its own response.
The `transform` adds your gateway token before forwarding, so the gateway can authenticate the request without exposing the token to code inside the sandbox.
Archil also overwrites these headers on forwarded requests:
| Header | Value |
| - | - |
| `archil-forwarded-host` | Original request hostname, such as `api.openai.com`. |
| `archil-forwarded-scheme` | Original scheme: `http` or `https`. |
| `archil-forwarded-port` | Original port: `80` or `443`. |
| `archil-forwarded-path` | Original path and query, such as `/v1/responses?trace=1`. |
| `archil-sandbox-id` | ID of the sandbox making the request. |
Use these headers to identify the original destination and sandbox. Authenticate with the gateway token;
the metadata headers alone do not prove that a request came from Archil.
Forwarding works with domain allow rules for HTTP on port 80 and HTTPS on port 443. HTTPS clients must trust the
[Archil egress CA](#domain-filtering). Deny rules still take precedence. See
[Create Sandbox](/api-reference/sandboxes/create-sandbox) for URL restrictions and
[Dynamic Updates](#dynamic-updates) to change an existing sandbox's policy.
## Dynamic Updates
A running sandbox's complete network policy can be replaced with `updateNetwork`, and `getNetwork` returns the current policy. A new policy applies to new connections; existing connections stay open. For example you can:
1. initially pull data from an S3 bucket
2. deny all egress while an agent is running
3. run a verifier and upload evaluation results
```typescript theme={null}
import * as archil from "disk";
const sandbox = await archil.createSandbox({
network: {
egress: {
default: "deny",
allow: ["data-bucket.s3.us-east-1.amazonaws.com"],
}
}
});
// download data...
await sandbox.updateNetwork({
egress: {
default: "deny",
}
});
// run agent...
await sandbox.updateNetwork({
egress: {
default: "deny",
allow: ["result-bucket.s3.us-east-1.amazonaws.com"]
}
});
// run verifier...
```
## Draining requests before pause
To let in-flight requests to selected hosts finish before a sandbox pauses, list them in `drain_on_pause` when creating
the sandbox:
```typescript theme={null}
import * as archil from "disk";
const sandbox = await archil.createSandbox({
network: {
egress: {
default: "allow",
drain_on_pause: ["*.anthropic.com", "api.openai.com"],
}
}
});
await sandbox.pause();
```
Each entry is a lowercase hostname pattern without a scheme, port, or path. `*` matches anywhere, `"*"` selects every
host, and a leading `*.` excludes the apex domain. Selectors do not grant network access. When the sandbox pauses, new
requests to selected hosts are held and active HTTP(S) responses, including streams, get up to ten minutes to finish
before the snapshot is taken. Draining covers ports 80 and 443; clients should retry stale connections after resume.
Set selectors at creation, since connections opened before a policy update are not intercepted.
# Sandbox images
Source: https://docs.archil.com/compute/sandboxes/images
Start sandboxes from public or private OCI images and reuse prepared images.
Choose an OCI image with the tools and dependencies your workload needs. Archil supports public and private Linux
images built for `amd64`.
## Public images
For a public image, pass `baseImage` in TypeScript, or `base_image` in Python and REST, when creating a sandbox:
```typescript theme={null}
const sandbox = await client.sandboxes.create({ baseImage: "python:3.13" });
```
You can use Docker Hub shorthand, as above, or a full reference such as `ghcr.io/acme/agent-runtime:v2`.
If you omit the image, the sandbox starts from `ubuntu:26.04`.
## Private images
For a private image, build it with your registry credentials first. The SDK waits until the image is ready,
then you can pass its ID when creating a sandbox:
```typescript theme={null}
import { Archil } from "disk";
const client = new Archil({
apiKey: process.env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
const image = await client.images.build({
source: "ghcr.io/acme/agent-runtime:v2",
registryAuth: {
username: process.env.REGISTRY_USERNAME!,
password: process.env.REGISTRY_TOKEN!,
},
});
const sandbox = await client.sandboxes.create({ imageId: image.imageId });
```
```python theme={null}
import os
from archil import Archil, RegistryAuth
with Archil(region="aws-us-east-1") as client:
image = client.images.build(
source="ghcr.io/acme/agent-runtime:v2",
registry_auth=RegistryAuth(
username=os.environ["REGISTRY_USERNAME"],
password=os.environ["REGISTRY_TOKEN"],
),
)
sandbox = client.sandboxes.create(image_id=image.image_id)
```
```bash theme={null}
curl --request POST \
'https://control.green.us-east-1.aws.prod.archil.com/api/images' \
--header "Authorization: $ARCHIL_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"source": "ghcr.io/acme/agent-runtime:v2",
"registry_auth": {
"username": "YOUR_REGISTRY_USERNAME",
"password": "YOUR_REGISTRY_TOKEN"
}
}'
```
The response contains `data.image_id`. Set `IMAGE_ID` to that value and poll until `data.status` is `ready` or `failed`:
```bash theme={null}
curl "https://control.green.us-east-1.aws.prod.archil.com/api/images/$IMAGE_ID" \
--header "Authorization: $ARCHIL_API_KEY"
```
If the build fails, inspect `failure_reason`. Once it is `ready`, create the sandbox:
```bash theme={null}
curl --request POST \
'https://control.green.us-east-1.aws.prod.archil.com/api/sandboxes?wait=true' \
--header "Authorization: $ARCHIL_API_KEY" \
--header 'Content-Type: application/json' \
--data "{\"image_id\": \"$IMAGE_ID\"}"
```
If the sandbox response is still `pending`, poll [Get Sandbox](/api-reference/sandboxes/get-sandbox) for `running`.
If it becomes `failed`, inspect `exit_reason`.
Use a registry username and a password or token with pull access. Archil does not store these credentials, so supply
them each time you build. Once built, the image is available to sandboxes in your account without registry credentials.
See [Build Image](/api-reference/images/build-image) and [Get Image](/api-reference/images/get-image) for request
options and build failures.
## Reusing and updating images
Archil prepares each image for use as a sandbox filesystem. To finish this work before creating sandboxes, build
once and reuse the returned image ID. You can also prebuild public images by omitting the registry credentials.
If you publish a new version under the same tag, build it again to pick up the change. New sandboxes using that
image ID get the latest successful build. While a rebuild is in progress, they can still use the previous successful
build. Existing sandboxes keep their original image, including after a restart.
To pin an image to a specific version, use a digest reference such as `repository@sha256:...` instead of a tag.
# Introduction
Source: https://docs.archil.com/compute/sandboxes/introduction
Create long-running Linux VMs with persistent disks, interactive shells, services, and forks.
Archil's persistent sandboxes make it simple to create long-running machines that simplify access to your Archil disk.
They are ideal for starting network services (such as web hosting, git hosting, or databases), interactive development
environments, or running untrusted AI-generated code for longer than an individual bash tool.
Persistent sandboxes can keep running by resetting their [timeout](#setting-a-time-to-live), and can be forked so you can run multiple copies of
the same sandbox in parallel. Persistent sandboxes are built on full-featured Linux microVMs that are colocated with your
Archil disk data for maximum security and performance.
If you instead need the ability to run one-off bash, Python, or Node code against your Archil disk, consider using
[serverless execution](/compute/serverless-execution) instead.
Persistent sandboxes are available in preview in Archil's AWS regions. During the preview period,
sandboxes may be pre-empted. Planned maintenance pauses them so they can be resumed. If a host fails
unexpectedly, its sandboxes are marked `failed` and lose in-memory state; start them again to boot from disk.
You can easily create a persistent sandbox using the TypeScript SDK (or the [Python SDK](/sdks/python#persistent-sandboxes)):
```typescript theme={null}
import * as archil from "disk";
const sandbox = await archil.createSandbox({
// Sandbox name, for finding later
name: "agent-workspace",
// Supports public OCI images
baseImage: "python:3.13",
// Select the amount of vCPU and memory
vcpuCount: 2,
memSizeMiB: 4096,
});
const execution = await sandbox.run("python --version && pwd");
const result = await execution.wait();
console.log(result.stdout);
```
Archil sandboxes run on x86-64 (`amd64`) and support between 1 and 32 vCPU cores and between 0.25 and 64 GiB of memory. Sandboxes run as the root
Linux user (uid: 0, gid: 0).
## Network egress
To prevent platform abuse, network egress is limited to the Team plan and above. On the Developer plan, all network egress is denied. Once upgraded, network egress can also be configured for the [sandbox firewall](/compute/sandboxes/firewall). Sandboxes created on the Developer plan keep denying egress after an upgrade until you [replace their network policy](/compute/sandboxes/firewall#dynamic-updates).
## Attaching an Archil disk
Pass `mounts` when creating a sandbox to give it access to files on an existing Archil disk. The disk must belong
to your account and be in the same region as the sandbox:
```typescript theme={null}
import { Archil } from "disk";
const client = new Archil({
apiKey: process.env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
const sandbox = await client.sandboxes.create({
baseImage: "ubuntu:24.04",
mounts: [{ disk: "dsk-0123456789abcdef", path: "/mnt/data" }],
});
console.log((await sandbox.exec("ls /mnt/data")).stdout);
```
```python theme={null}
from archil import Archil, SandboxMountSpec
with Archil(region="aws-us-east-1") as client:
sandbox = client.sandboxes.create(
base_image="ubuntu:24.04",
mounts=[SandboxMountSpec(disk="dsk-0123456789abcdef", path="/mnt/data")],
)
print(sandbox.exec("ls /mnt/data").stdout)
```
```bash theme={null}
curl --request POST \
'https://control.green.us-east-1.aws.prod.archil.com/api/sandboxes?wait=true' \
--header "Authorization: $ARCHIL_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
"base_image": "ubuntu:24.04",
"mounts": [{ "disk_id": "dsk-0123456789abcdef", "path": "/mnt/data" }]
}'
```
If the response is still `pending`, poll [Get Sandbox](/api-reference/sandboxes/get-sandbox) until it is `running`
before executing commands. If it becomes `failed`, inspect `exit_reason`.
Archil handles mount authentication automatically. Deleting a sandbox removes its root filesystem.
Mounted disks and their data are preserved.
With a single mount, `path` is optional and defaults to `/mnt/archil`. To mount more disks, add an entry for each
one with a separate path, such as `/mnt/data` and `/mnt/models`.
Use `subdirectory` to expose only part of a disk. See [Create Sandbox](/api-reference/sandboxes/create-sandbox)
for the full mount options and path restrictions.
By default, a mount takes exclusive access to the disk. To share it across sandboxes, use `readOnly` (`read_only`
in Python and REST) for readers or `conditional` for writers. Coordinate concurrent writes in your application
to avoid overwriting each other's changes. See [Sharing Disks](/concepts/sharing-disks) for how these modes work.
Sandboxes keep their mounts across stop/start and pause/resume. To change them, stop the sandbox and pass a new
`mounts` list to [Start Sandbox](/api-reference/sandboxes/start-sandbox). Omit `mounts` to keep the saved list,
or pass `[]` to remove all mounts without deleting their disks. For a paused sandbox, `resume()` keeps its mounts;
use `start()` to change them, which boots from disk and discards the paused memory state.
## Passing environment variables
Environment variables can be set for the whole sandbox, for a single command, or both. Variables passed when the
sandbox is created apply to every command that runs inside it:
```typescript theme={null}
const sandbox = await archil.createSandbox({
baseImage: "python:3.13",
env: {
DATABASE_URL: "postgres://localhost/app",
LOG_LEVEL: "info",
},
});
```
Individual processes can add or override their own variables by passing `env` to `run`:
```typescript theme={null}
// Runs with DATABASE_URL from the sandbox, and LOG_LEVEL=debug
const migration = await sandbox.run("python manage.py migrate", {
env: { LOG_LEVEL: "debug" },
});
await migration.wait();
```
## Starting and stopping sandboxes
A sandbox runs until you shut it down or its [TTL](#setting-a-time-to-live) expires. There are two ways to suspend one,
and they differ in how much state they keep.
Stopping a sandbox sends `SIGTERM` to every running process and shuts down the VM, leaving the disk intact:
```typescript theme={null}
await sandbox.stop();
```
Starting it again boots a fresh VM from that same disk:
```typescript theme={null}
await sandbox.start();
```
Everything written to disk survives, but memory does not, so your application handles shutdown and startup the same way
it would on any machine. You can instead pause a VM to snapshot memory alongside the disk, so a resumed sandbox
picks up exactly where it left off, with running processes intact:
```typescript theme={null}
// Preserve running application state for resume
await sandbox.pause();
// Resume the sandbox from the paused state
await sandbox.resume();
```
Deleting a sandbox releases all of its resources and cannot be undone. A sandbox has to be stopped before it can be
deleted, unless it has already exited or failed:
```typescript theme={null}
await sandbox.stop();
await sandbox.delete();
```
## Forking a sandbox
In some situations, you may want to fork a sandbox into multiple branches from a single point in time. This makes it simple to,
for example, start sandboxes from a template without needing to incur startup latency for a slow-running program or run parallel
experiments on sandbox state from some point in the past.
A sandbox without additional disk mounts can be forked while running, paused, or stopped:
```typescript theme={null}
const newSandbox = await sandbox.fork();
```
After forking, `newSandbox` will be able to run commands independently of the original sandbox, starting from the exact state
of the original sandbox. Forking a running sandbox pauses it while the snapshot is taken and resumes it once the fork is
accepted, so attached process connections close; reconnect with `attach`. A paused or stopped sandbox is left
as it is, and a fork of a stopped sandbox boots from its disk.
A sandbox with mounted disks must be stopped before forking. The fork gets its own root filesystem but mounts
the same additional disks as its parent, so changes to those disks are shared. To run both sandboxes at once,
use a shared mount mode or give each sandbox separate disks. Separate disks also keep their data isolated.
During the preview period, all child sandboxes forked from a parent must be deleted before the parent sandbox is deleted.
## Hosting network services
Archil sandboxes support the ability to host network services, including creating per-sandbox preview URLs for CI-like
workflows.
There are two ways to expose a port for a network service, depending on whether you are inside or outside the sandbox.
Outside the sandbox, you can specify a list of ports to expose when creating the sandbox or dynamically expose a port later.
```typescript theme={null}
const sandbox = await archil.createSandbox({
ports: [3000]
});
const ports = await sandbox.listPorts();
const hostnames = ports.map((port) => port.hostname);
console.log(hostnames);
const hostname = await sandbox.exposePort(3001);
await sandbox.unexposePort(3001);
```
Exposed ports are public: they accept any TCP protocol over TLS with SNI, without authentication.
For private access, leave the port unexposed and create a port token. Send the token in the `X-Archil-Token` header;
requests without a valid token get `401`. Tokens are checked when a connection opens, so revoking or expiring a token
blocks new connections while open ones stay open. Token access supports HTTP/1.1 over HTTPS only, and publicly exposed
ports ignore tokens.
```typescript theme={null}
const access = await sandbox.createPortToken(8080, {
// optionally specify a TTL, up to "8760h"; omit for no expiry
ttl: "5m",
});
// pass X-Archil-Token when making requests to the hostname
const response = await fetch(`https://${access.hostname}`, {
headers: {
"X-Archil-Token": access.token,
}
});
await sandbox.deletePortToken(access);
```
Inside the sandbox, `archil services create` runs a command as a supervised service: it starts immediately and restarts if
it exits. Service ports are private by default, so reach them by exposing the port or with a port token. To let services
publish their own hostnames, create the sandbox with `enable_service_ingress: true` through the
[Create Sandbox API](/api-reference/sandboxes/create-sandbox).
```typescript theme={null}
const serviceProcess = await sandbox.run(
"archil services create web --tcp-port -- ",
);
const { stdout } = await serviceProcess.wait();
const service = JSON.parse(stdout);
console.log("Server available at: ", service.hostname);
```
For example, in a sandbox with service ingress enabled, the following runs `python3 -m http.server 8080` and returns a
stable HTTPS URL that proxies traffic to local port 8080. Without service ingress, the service still runs, but its
`hostname` is not routed until you expose the port.
```bash theme={null}
$ archil services create web --tcp-port 8080 -- python3 -m http.server 8080
{
"after": [],
"command": ["python3", "-m", "http.server", "8080"],
"enabled": true,
"env": {},
"hostname": "8080-.prod-aws-us-west-2-green.archil.app",
"name": "web",
"state": {
"exit_status": null,
"pid": null,
"restart_count": 0,
"result": null,
"status": "activating"
},
"tcp_port": 8080,
"working_dir": null
}
```
Traffic to an exposed port or service hostname, or a request with a valid port token, automatically starts a stopped
sandbox or resumes a paused one, including one paused by its TTL. All sandbox ports are served over TLS.
Exposed ports and service hostnames are publicly accessible. Use port tokens, or include authentication/authorization
in your service, if you want to expose a private service.
## Choosing a base image
Use `baseImage` to start from a public Linux OCI image, such as `python:3.13`. The default is `ubuntu:26.04`.
For private images, build with your registry credentials first, then pass the returned `imageId` when creating
the sandbox. See [Sandbox images](/compute/sandboxes/images) for examples and how to reuse builds.
## Setting a time to live
A sandbox has a hard timeout and an optional idle timeout. Both **pause** the sandbox, preserving its disk, memory, and
processes for `resume()`. Attached process connections close when it pauses; reconnect after resuming. If the pause
snapshot fails, the sandbox is marked `failed`. Stop a paused sandbox before deleting it.
| Setting | Range | Default |
| - | - | - |
| `maxTtlSeconds` | 60–86,400 seconds | 86,400 seconds (24 hours) |
| `idleTtlSeconds` | 0–86,400 seconds | 0 (disabled) |
```typescript theme={null}
const sandbox = await archil.createSandbox({
maxTtlSeconds: 3600,
idleTtlSeconds: 300,
});
await sandbox.setTimeout(7200);
await sandbox.setTimeout({ timeoutSeconds: 3600, idleTtlSeconds: 60 });
await sandbox.setTimeout({ idleTtlSeconds: 0 });
```
Updating the hard timeout resets its deadline from now, capped at 24 hours after the current session started.
An idle-only update leaves that deadline unchanged and resets the idle countdown if no process connection is open.
Omitted settings stay unchanged. Updates to a paused or stopped sandbox apply to its next powered-on session.
Starting or resuming begins a new session with a fresh timeout.
Open direct process connections prevent idle expiry. Detached processes and service-port traffic do not keep the idle
timer alive. Leave idle timeout disabled for unattended jobs or services that must stay running between connections.
Activity does not extend the hard timeout. `setTimeout()` can extend it only within the session's 24-hour limit.
For REST, use [Set sandbox timeouts](/api-reference/sandboxes/set-sandbox-timeout).
## Opening interactive shells
By default, processes run without allocating a pseudo-terminal (pty). This is usually desirable, since it signals
to the process that it is not in an interactive terminal. However, some commands, such as those that emit color, full-screen
tools like `vim` and `top`, or tools that need interactive input should instead be launched in a pseudo-terminal.
Provide terminal dimensions to launch a process in a pseudo-terminal. The process can then interactively receive input
and stream output to the caller:
```typescript theme={null}
const shell = await sandbox.run("/bin/sh", {
terminal: { cols: 120, rows: 40 },
onOutput: ({ data }) => process.stdout.write(data),
});
await shell.sendInput("ls -la\n");
await shell.sendInput("exit\n");
const { exitCode } = await shell.wait();
```
## Next steps
* [TypeScript SDK](/sdks/typescript#persistent-sandboxes)
* [Python SDK](/sdks/python#persistent-sandboxes)
* [Sandbox images](/compute/sandboxes/images)
* [Create a sandbox](/api-reference/sandboxes/create-sandbox)
* [Create a process connection](/api-reference/sandboxes/create-sandbox-connection)
* [Fork a sandbox](/api-reference/sandboxes/fork-sandbox)
* [Sandbox lifecycle endpoints](/api-reference/sandboxes/pause-sandbox)
* [Sandbox firewall](/compute/sandboxes/firewall)
# Search files
Source: https://docs.archil.com/compute/search-files
Archil disks in AWS regions provide a built-in, high-performance search tool that's effective for
searching across millions of files on your disk, up to 10x faster than tools like `ripgrep`.
```typescript TypeScript theme={null}
import * as archil from "disk";
const d = await archil.getDisk("dsk-abc123");
const { matches } = await d.grep({
directory: "logs",
pattern: "ERROR|FATAL",
recursive: true,
});
```
```python Python theme={null}
import archil
d = archil.get_disk("dsk-abc123")
res = d.grep(directory="logs", pattern="ERROR|FATAL", recursive=True)
```
Teams building the highest-performance AI agents have discovered that agent performance degrades
as you give them more tools to select between at each turn. As a result, teams are increasingly moving
away from specialized products like vector databases for context and towards lowest-common denominator
infrastructure like file storage. However, existing file system tools like `grep` are too slow for
searching across large document sets.
With Archil's built-in `grep` APIs, you can effectively search across millions of documents on your
disk using map-reduce inside of Archil. Archil's `grep` API is priced the same as if you'd manually
run the search on serverless execution, billed as the total compute time needed to perform the
search.
## Managing concurrency
Archil's `grep` API works by performing a massively parallel search across the files stored in
the folder that you're searching. Archil automatically manages the concurrency of each search to
optimize for search latency. If you'd prefer to control this concurrency manually, you can specify
the exact number of workers to use via the `concurrency` option.
```typescript TypeScript theme={null}
const { matches } = await d.grep({
directory: "logs",
pattern: "ERROR|FATAL",
concurrency: 10,
});
```
## Limiting work
For some kinds of search, you do not need an exhaustive set of results. For example, if you're trying
to find the first occurrence of an error, you only need one result. You can manage the amount of work
that Archil's search performs by specifying either the `maxResults` or `maxDurationSeconds` options.
When these options are specified, the search will stop as soon as the result or time limit is reached,
regardless of whether the search was exhaustive.
## Next steps
* [TypeScript SDK: searching files](/sdks/typescript#searching-files) — the `disk.grep` API surface
* [Python SDK: searching files](/sdks/python#searching-files) — the `Disk.grep` API surface
* [API reference: Search Files](/api-reference/serverless-execution/grep-disk) — the raw HTTP endpoint
* [Serverless execution](/compute/serverless-execution) — run arbitrary commands when you need more than search
# Serverless execution
Source: https://docs.archil.com/compute/serverless-execution
Run bash commands directly on an Archil file system.
Archil's serverless execution makes it simple to run untrusted code like bash, Python, and Node.js
directly on the data in your Archil file system **without** needing to manage the lifecycle of
a separate sandbox instance.
Serverless execution is built on full-featured Linux microVMs that are colocated with your
data for maximum security and performance. Each Archil disk can run many commands concurrently,
and because they're fully elastic, you only pay for the time that
your code is actually running. This makes them ideal for tasks such as the bash tool in an AI agent loop.
If you instead need a sandbox that stays alive in-between calls for the purpose of running commands with
long startup times or network services, you can use an Archil [Persistent Sandbox](/compute/sandboxes/introduction).
```typescript TypeScript theme={null}
import * as archil from "disk";
const { disk } = await archil.createDisk({ name: "agent-workspace" });
const { stdout } = await disk.exec("git clone --depth 1 https://github.com/pallets/flask.git");
```
```python Python theme={null}
import archil
from archil import S3Mount
result = archil.create_disk(name="agent-workspace")
res = result.disk.exec("git clone --depth 1 https://github.com/pallets/flask.git")
```
```bash CLI theme={null}
npx disk dsk-abc123 exec "git clone --depth 1 https://github.com/pallets/flask.git"
```
Companies are using Archil's serverless execution for:
* **Running untrusted AI-generated code** on a machine that is close to the context that the AI
is attempting to work on, without risking that code breaking the origin server.
* **Running map-reduce style jobs across large data sets** for data analysis tasks such as full-text
search or aggregation.
* **Using your disk from a different region** in order to avoid excessive network latency between
your application code and the disk, and avoid egress fees.
Archil disks include 1,000 minutes of free serverless execution usage per month, for each terabyte of
active data on the disk. Each invocation is billed with a 100ms minimum runtime.
## Working directory and environment
Each command starts with the working directory set to where your Archil disk is mounted
(`/mnt/archil`). The runtime image includes most common Unix tools, but you can install additional
tools into the environment by directing the binaries to your disk.
## Composing multiple disks into a single command
Often, users need to run commands on multiple disks at once. For example, if you're building an
AI agent which has access to a read-only customer-specific disk and a read-write session-specific
disk, and you want your agent to use documents on both disks.
To run a command against several disks, you instead pass a map of disks into `archil.exec`. Each
disk is automatically mounted to the `/mnt/archil/` path inside the sandbox. For example, in the
code below, the `data` disk is mounted to `/mnt/archil/data` and the `logs` disk is mounted to `/mnt/archil/logs`.
```typescript TypeScript theme={null}
import * as archil from "disk";
const { stdout } = await archil.exec({
disks: {
data: "dsk-abc123",
logs: "dsk-def456",
},
command: "wc -l /mnt/archil/data/*.csv > /mnt/archil/logs/lines.txt",
});
```
```python Python theme={null}
import archil
res = archil.exec(
disks={
"data": "dsk-abc123",
"logs": "dsk-def456",
},
command="wc -l /mnt/archil/data/*.csv > /mnt/archil/logs/lines.txt",
)
```
## Specifying disk options
You may want to run a command in which some disks have reduced permissions, so the code
running cannot modify them. Serverless execution supports specifying whether or not a disk should be
marked as read-only or whether it should be limited to a subdirectory of the disk.
For example, in the code below, the `logs` disk will be mounted to `/mnt/archil/logs`. That directory
will be read-only, and code in the sandbox cannnot modify it. It will only display the contents of
the `app/logs` subdirectory of the disk.
```typescript TypeScript theme={null}
import * as archil from "disk";
const { stdout } = await archil.exec({
disks: {
data: "dsk-abc123",
logs: {
disk: "dsk-def456",
subdirectory: "app/logs",
readOnly: true,
},
},
command: "wc -l /mnt/archil/logs/*.log",
});
```
```python Python theme={null}
import archil
from archil import ExecMountSpec
res = archil.exec(
disks={
"data": "dsk-abc123",
"logs": ExecMountSpec("dsk-def456", subdirectory="app/logs", read_only=True),
},
command="wc -l /mnt/archil/logs/*.log",
)
```
## Consistency with other clients
Serverless execution works like other Archil clients, with the same read-after-write
consistency model. A file written to the disk by one command is immediately visible to the
next command that reads it.
By default, disks are mounted in shared mode, which means that you'll need to
take care to call `archil checkout` before mutating files on the disk. For more details on how to
checkout or checkin files in shared mode, see [Sharing disks](/concepts/sharing-disks).
## Limits
* Each command will timeout after 5 minutes of runtime, but the sandbox itself
will continue running to completion.
* Stdout and stderr are limited to the trailing 128 KiB of output in the response. If you need to
store larger output, we recommend storing the data on the disk and then using the [getObject API](/protocols/s3-api) to retrieve it.
* Users can execute hundreds of commands in parallel, contact us to have your
limits raised to thousands.
## Next steps
* [Search files in parallel with `disk.grep`](/compute/search-files)
* [Build a bash tool for an agent](/guides/ai/bash-tool)
* [TypeScript SDK reference for `disk.exec`](/sdks/typescript#executing-commands)
* [Python SDK reference for `Disk.exec`](/sdks/python#executing-commands)
* [`disk` CLI reference](/reference/disk-cli)
# SQLite
Source: https://docs.archil.com/compute/sqlite
Archil disks in AWS regions provide a native integration with SQLite that enables you to
create and work with an unlimited number of SQLite databases on your disk. This is ideal for
AI agents which store their context as a combination of structured and unstructured data.
The Archil SQLite integration can be used from multiple clients simultaneously, even for writes.
You don't need to manage any resources in order to use the SQLite APIs, and you're only charged
for the compute that's used while queries are running.
The SDK for the Archil SQLite integration is currently available for TypeScript users, in a separate
package, which can you install with any Node package manager:
```bash theme={null}
npm install disk @archildata/sqlite
```
## Create and manage SQLite databases
To create a new SQLite database on your Archil disk, use the `createDatabase` function and pass in the disk and the name of the database file. You can then use the `getDatabase` function to retrieve the database later.
```ts theme={null}
import * as archil from "disk";
import { createDatabase, getDatabase } from "@archildata/sqlite";
const disk = archil.getDisk("dsk-1234");
const db = await createDatabase(disk, "apps/app-1/main.sqlite");
// Retrieve the database on a later request
const db = await getDatabase(disk, "apps/app-1/main.sqlite");
```
`createDatabase` will create a new SQLite database on the disk if it doesn't exist, or return the existing database if it does. `getDatabase` will return an existing database.
## Execute SQL queries
You can execute SQL queries against each database using the `read` and `write` methods. The `read` method is used for queries that return data, while the `write` method is used for queries that modify data.
```ts theme={null}
const db = await createDatabase(disk, "apps/app-1/main.sqlite");
// Create tables
await db.write`CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)`.run();
// Add users
const username = "John Doe";
const email = "john@example.com";
await db.write`INSERT INTO users (name, email) VALUES (${username}, ${email})`.run();
// Read users
const users = await db.read`SELECT * FROM users`.all();
```
Why is there a difference between `read` and `write`? `write` queries need exclusive write access to the database file, while many `read` queries can be executed in parallel.
You can execute multiple queries in a single transaction using the `transaction` method. This ensures that either all queries succeed or none of them are applied.
```ts theme={null}
const [johnResult, janeResult] = await db.transaction([
db.write`INSERT INTO users (name, email) VALUES ('John Doe', 'john@example.com')`.run(),
db.write`INSERT INTO users (name, email) VALUES ('Jane Doe', 'jane@example.com')`.run()
]);
```
# Authorization
Source: https://docs.archil.com/concepts/authorization
Configure who is authorized to access the Archil APIs or disk
Archil supports three different kinds of resources to authorize access to:
* The **Archil disk mount** for mounting the disk on Linux, on macOS, or inside containers.
* The **Archil API** for creating Archil disks or sandboxes
* The **Archil S3 API** for calling the S3-compatible API from AWS SDKs
## Disk mount authorization
Each Archil disk manages which servers are authorized to mount the disk using the `archil` CLI or macOS application. Authorizing
a server to mount the disk gives that server access to the *entire* file system. Even though Archil respects the POSIX permissions
of the files stored on the disk, they should not be used as an isolation boundary for unstrusted workloads running together.
Archil supports two ways to authorize a server to mount a disk:
1. **IAM-based authorization** - recommended when you are mounting a disk located in AWS from the same AWS region
2. **Token-based authorization** - recommended when you need the lowest disk mount latency
You configure the authorized users for your disk on the Archil console, on the disk details page, by selecting **Connetions**, and then
choosing to **Add User**, and selecting either a **Disk Token** or an **IAM User**, or via the [Add Disk User API](/api-reference/disk-users/add-user).
### IAM-based disk authorization
IAM-based disk authorization allows you to configure an AWS IAM role or user ARN that is allowed to mount the disk.
You can use the `archil utils get-iam-role` CLI command to get the IAM role ARN of the current EC2 instance.
You can mount a disk on an IAM-authorized server by running the `archil mount` command without any additional flags.
When using [AWS IAM Roles Anywhere](https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html), you must set the `AWS_EC2_METADATA_SERVICE_ENDPOINT` and `AWS_REGION` environment variables before mounting. These can be set inline with the mount command or exported in your shell session.
### Token-based disk authorization
Token-based disk authorization allows you to create a long-lived bearer token that grants the ability to mount the disk.
You mount a disk using a token by setting the `ARCHIL_MOUNT_TOKEN` environment variable and passing it through sudo.
```bash theme={null}
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/data --region
```
Using `--preserve-env=ARCHIL_MOUNT_TOKEN` passes the disk token through sudo's environment rather than the command line, preventing it from being visible in process listings (`ps aux`). This is more secure than both `sudo -E` (which passes all environment variables) and inline assignment (which exposes the token in the process list).
Archil supports the ability to configure tokens to have a either a TTL (allowing mounts up for a period of time) or to be one-time use.
When the token expires, it can no longer be used to mount the disk, and read/write operations on the disk from that client will fail.
## API authorization
The Archil API allows you to create and manage Archil disks, disk users, and sandboxes -- including grep, SQLite, checkpoint, and S3 APIs.
You can authorize to the Archil API using an API key. You can create an API key on the Archil console, by selecting **API Keys**
from the sidebar and then clicking **Create API Key**.
Archil API keys are scoped to the organization or personal account which created them and a specific Archil region. This means that
an API key created for "aws-us-east-1" will not work to manage disks in "aws-us-west-2". API keys have full access to disks in the
organization or personal account which created them, including the ability to read data, delete disks, and run sandboxes.
## S3 authorization
The Archil API supports S3 APIs such as PutObject, GetObject, DeleteObject, and ListObjectsV2. You only need to create an S3-specific
key if you plan on using out-of-the-box AWS S3 SDKs, which require a SigV4 signing access key and secret access key.
To create an S3 access key and secret access key, use the Archil console and navigate to the disk details page. Then select
**Connetions** from the sidebar, and then click **Add User**. Select **S3** as the user type, and then click **Create User**.
You can then use these keys to access your Archil disk through the AWS CLI:
```bash theme={null}
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
aws s3 ls s3://dsk-abc123/ \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
```
# Branches & Checkpoints
Source: https://docs.archil.com/concepts/branches-and-checkpoints
Snapshot the state of an Archil disk and fork independent, writable copies from any snapshot.
Branches and checkpoints are generally available in all AWS regions. They are available only on disks that are **not** synchronized to a [data source](/concepts/data-sources) — a disk can either synchronize to a data source (such as Amazon S3) or support branches and checkpoints, not both.
Branches and checkpoints let you capture the state of an Archil disk at a moment in time and then explore alternative histories without disturbing the original.
A **checkpoint** is a named, point-in-time snapshot of the filesystem state on a disk. The filesystem state a checkpoint references is immutable.
A **branch** is an independent, writable fork of a disk that starts from a specific checkpoint. Writes on a branch don't affect the source branch.
Branches form a tree. Every branch points back to the checkpoint on its parent it was forked from. Checkpoints, in turn, always belong to a specific branch (or to the root).
If you're familiar with Git, Archil branches and checkpoints map fairly directly to Git branches and Git commits respectively.
For example:
![A timeline showing the root disk with checkpoints chk: x and chk: z, branch 1 forked from chk: x with its own checkpoint chk: y, and branch 2 forked from chk: z.]()
![A timeline showing the root disk with checkpoints chk: x and chk: z, branch 1 forked from chk: x with its own checkpoint chk: y, and branch 2 forked from chk: z.]()
Here, the root has two checkpoints: `x` and `z`. Branch 1 was forked from `x` and branch 2 was forked from `z`. Any mutations to the filesystem state on the root after `x` would not appear in branch 1 and vice versa.
## Working with checkpoints
Checkpoints are taken from a mounted disk and capture the filesystem state at the time the command runs:
```bash theme={null}
# Mount the disk, then take a checkpoint
sudo archil mount myorg/mydisk /mnt/archil --region aws-us-east-1
archil checkpoints create /mnt/archil pre-migration
```
Checkpoint names must be unique within the branch they're taken on. Once created, the state a checkpoint points to never changes.
To see the checkpoints available on a disk:
```bash theme={null}
# Checkpoints on the root
archil checkpoints list dsk-0123456789abcdef --region aws-us-east-1
# Checkpoints on a specific branch
archil checkpoints list dsk-0123456789abcdef --region aws-us-east-1 --branch experiment-1
```
The list is rendered oldest-first, so the most recent checkpoint appears at the bottom.
## Working with branches
Create a branch by naming the checkpoint you want it to fork from.
```bash theme={null}
# Branch from a checkpoint on the root
archil branches --region aws-us-east-1 create dsk-0123456789abcdef experiment-1 \
--from-checkpoint pre-migration
# Branch from a checkpoint that lives on another branch
archil branches --region aws-us-east-1 create dsk-0123456789abcdef experiment-2 \
--from-checkpoint v1 --from-branch experiment-1
```
To use a branch, mount it with `--branch`:
```bash theme={null}
sudo archil mount myorg/mydisk /mnt/branch --region aws-us-east-1 --branch experiment-1
```
A mounted branch behaves exactly like any other Archil disk. You can read, write, take additional checkpoints from it, and create further branches off of those checkpoints. Writes to a branch are isolated from its parent and from any sibling branches.
Because each branch is fully independent, it can be mounted on any client — including a different VM from the one using its parent — and two branches of the same disk can be mounted on separate VMs at the same time. This makes branches a natural fit for fan-out workloads such as giving each agent or CI job its own writable fork.
A branch is not a security boundary. Credentials are scoped to the entire disk, including all of its branches and checkpoints — anyone who can mount the disk can access every branch and checkpoint on it. To isolate untrusted workloads from each other (for example, one agent or tenant per fork), give each its own disk rather than relying on branches. See [Disk Users](/concepts/disk-users).
For the full set of CLI flags, see [`archil checkpoints`](/reference/archil-cli#checkpoints) and [`archil branches`](/reference/archil-cli#branches) in the CLI reference.
# Data Sources
Source: https://docs.archil.com/concepts/data-sources
Configure bidirectional synchronization for Archil disks to existing data sets
Archil disks support optional, bidirectional synchronization to a data source. This is useful when you have
an existing data set stored in cloud object storage, like Amazon S3, that you want to expose to agents as
a native Linux file system.
If, instead, you need to simultaneously use strongly consistent data from both the file system and an
S3 API for a net-new data set, we instead recommend that you use Archil's built-in support for the
[S3 API](/protocols/s3-api) to access your data.
Disks which synchronize to a backing data source do not support [checkpoints and branching](/concepts/branches-and-checkpoints).
## Supported data sources
Archil supports a wide variety of data sources out of the box. Each data source has specific configuration requirements and credential setup procedures.
For detailed configuration instructions for each supported data source, see the dedicated pages:
* [**S3-compatible Object Storage**](/data-sources/s3-object-storage) - including Amazon S3, Google Cloud Storage, Cloudflare R2, and other providers
## Consistency
Archil is [strongly, read-after-write consistent](/details/consistency) for all clients connected through the
file system interface or users of the Archil [S3 API](/protocols/s3-api).
For performance and efficiency reasons, writes to an Archil disk are replayed asynchrously to the backing data
source.
For example, if you create a new file `hello.txt` on an Archil disk using `touch hello.txt`, other clients connected to the same Archil disk will immediately see the new file. Clients of the Archil [S3 API](/protocols/s3-api) will also see the new file immediately. A backing S3 bucket configured as a data source will see the new
file with eventual consistency, usually within 10s of seconds.
To reduce the number of operations on backing object storage, Archil automatically combines multiple writes to
individual files into a single PutObject call ot the data source. The specific amount of time that it takes Archil
to replay a write into a backing data source depends on the specific workload and the compleixty of the operations.
Before writes appear in the backing object storage, they are stored durably on Archil's SSD storage servers. Archil
redundantly stores data across multiple servers, in multiple Availability Zones, so the failure of any individual
node (or an entire Availability Zone) to does not result in data loss. See [Data durability](/details/architecture#data-durability) for details.
Archil does not offer a way for users to force a synchronization to the backing data source. If you find yourself
needing to force synchronizing to the backing bucket, we instead recommend moving S3 API clients to the Archil S3
API, which is strongly consistent with changes in the file system.
Synchronization from a data source is bidirectional. To reduce read latencies (often by up to 30x), Archil caches
active data from a data source on our high-performance SSD storage servers. As a result, newly created object on the backing storage system can take minutes to appear in the corresponding directory on disk.
## Concurrent writes
Archil supports concurrently modifying the disk from both the file system and from the origin data source -- as
long as these writes occur to different locations on the diks. If you attempt to write to the same file or
directory which is being actively modified via the file system, it may result in undefined behavior including data
loss or corruption.
If you need to concurrently modify the same directory from the file system and the S3 API, we recommend using Archil's built-in S3 API which supports concurrent modification.
## Atomicity
Some data sources, such as Amazon S3, do not have atomic operations that mirror each of the possible POSIX file
operations (for example, directory renames). When executing these kinds of operations against an Archil disk,
you may notice partial results for the operations when reading directly from the underlying data source.
Reads from the Archil disk itself are always strongly consistent.
## Availability
If the configured data source becomes unreachable while a disk is mounted, writes are not lost: Archil buffers
them durably in the disk's SSD storage cache and flushes them to the data source once it is reachable again.
This SSD cache for unacknowledged writes has a fixed size to maintain the durability of the disk, and if you
exceed this size, the disk will stop accepting write operations.
Reads served from cache are unaffected; a read that misses the cache and requires the data source blocks until
it can be satisfied.
## POSIX metadata
When Archil synchronizes a disk to a data source, file contents are written as ordinary objects at the matching key.
Archil will internally track the POSIX metadata for each file, including its mode bits, ownership information, and any
extended attributes. These are not visible by reading the bucket directly.
Symbolic links are stored as a text file that contains just the path that the link points to. Hard links are stored
as a full copy of the linked data.
## Deleting a disk
Deleting an Archil disk does not delete the data in its data source. An Archil disk is a synchronized
view over the bucket you own — when you delete the disk, Archil stops synchronizing and discards its cached
copy, but the objects already written to your data source remain in your bucket, under your control.
## Deleting a data source
If you delete the data source while an Archil disk remains connected to it, the associated disk will stop
accepting read, write, or mount operations. This process is eventually consistent, and some operations may
continue to function after the deletion ofr a data source for a short period of time.
# Metering
Source: https://docs.archil.com/concepts/metering
How Archil measures active data in the high-speed storage layer
Archil disks use a high-speed, durable cache to enhance performance when working with data from
[data sources](/concepts/data-sources). You are only charged for data actively stored in this
cache layer. For pricing details, see [Pricing & Billing](/administration/billing).
## File data
When reading data from a data source, Archil automatically caches it in the high-speed storage
layer. Cached data remains active for up to 1 hour after your last access.
Writes are buffered in the high-speed storage layer for coalescing before being flushed to the
data source. Recently written data remains active for up to 1 hour after it has been flushed.
The amount of active data may not exactly match the amount you've read or written. Archil
prefetches and caches data ahead of sequential reads, which can result in additional cached data.
Rapid overwrites to the same file will also result in additional active data while Archil
asynchronously coalesces and flushes the writes.
## File metadata
Metadata accesses (e.g., `find`, `ls`, `stat`) do not cause file data to become active and incur
no metering charges.
Metadata writes (e.g., rename, `chmod`) are counted as 32 KiB of active data per operation until
the change is propagated to the data source.
Taking ownership of a path with [`archil checkout`](/reference/archil-cli#checkout) lets Archil
coalesce the many small attribute updates a workload produces into larger chunks before metering
them. For example, `git clone`-ing a 15 MB repository into a checked-out directory is metered as
roughly fifteen 1 MiB writes rather than the thousands of individual metadata operations the client
issues.
## How billing is calculated
Archil meters the amount of active data each minute. Your monthly bill is the time-weighted
average of active data over the billing period, charged at \$0.20 per GiB-month.
For example, if you store 100 GiB for half of a month, you are billed for 50 GiB-months.
You can monitor your active data usage in the [Archil console](https://console.archil.com) on
each disk's **Usage** tab, or on the [Billing](https://console.archil.com/billing) page for an
account-wide view.
## Branches and checkpoints
[Branches and checkpoints](/concepts/branches-and-checkpoints) are copy-on-write. Forking a branch
or taking a checkpoint does not duplicate the underlying data — you are billed once for the shared
base data, plus only the unique data each branch or checkpoint adds.
For example, a 100 GiB parent disk with 50 branches that each modify \~5 GiB of unique data is
metered as roughly **350 GiB total** — the shared 100 GiB base plus the \~250 GiB of unique
per-branch changes — not as 51 full copies (\~5,100 GiB). A checkpoint retains its unique data for
as long as it exists; deleting a checkpoint removes that data from your bill.
# Sharing Disks
Source: https://docs.archil.com/concepts/sharing-disks
Archil disks can be shared between multiple servers simultaneously
Archil supports sharing disks between multiple connected users simultaneously. This easily allows you to support workflows
where multiple agents are collaborating on the same dataset, or if multiple servers need to have live access to the same
data set.
Archil delivers the local-like disk performance by matching the semantics of xfs, ext4, or zfs and enabling connected clients
to execute file system operations locally. This allows Archil to exceed the performance of comparable NFS or 9p systems, in which
file system operations must always be performed on the remote server. Archil requires that clients hold a write-lock, called a
"delegation", in order to operate in this higher-performance mode. In the fullness of time, Archil will dynamically assign and
manage delegations for optimal performance. Today, users must manually manage these delegations.
Archil supports multiple mount modes, which change how delegations work.
* **Exclusive mode** - By default ([`archil mount`](/reference/archil-cli#mount)), mounting an Archil disk puts it in exclusive mode. This locks the entire file
system for the machine which mounted. That machine will receive local-like performance that matches an EBS volume, but other
machines will not be able to access the disk until its unmounted.
* **Shared mode** - If you mount the disk in shared mode (`archil mount --shared`), no lock is taken out on the disk. By default,
the entire disk will appear read-only to the client until it manually requests a delegation ([`archil checkout `](/reference/archil-cli#checkout)). With a
delegation, the client will have exclusive write access to the specified `` and receive local-like performance.
* **Conditional mode** - If you mount the disk in conditional mode (`archil mount --conditional`), you enable the disk to execute
operations remotely, like an NFS or 9p file system. This means that all files (which are not write-locked by another client) are
immediately writeable, but the disk performance will be reduced to that of a remote file system. Users in conditional mode can
continue to use `archil checkout` to get exclusive write access and higher performance to subdirectories on the disk.
We intend to make **Conditional mode** the default for new disks soon. The remainder of this article will describe how Shared mode
and delegations work.
## Selecting a mode on serverless execution
By default, disks in [serverless execution](/compute/serverless-execution) are mounted in **Shared mode** (`archil mount --shared`) to enable multiple sandboxes to easily access
the same data.
You can change the disk into **Exclusive mode** by first executing `archil checkout /mnt/archil` as part of your serverless execution
command.
You can select **Conditional mode** by using the more verbose serverless execution API, like below:
```ts theme={null}
archil.exec({
data: {
disk: "dsk-def456",
conditional: true,
},
});
```
In this example, the disk will be mounted to `/mnt/archil/data` instead of `/mnt/archil`.
## Automatic delegations in shared mode
When mounting in **Shared mode**, some operations automatically grant a write-lock on the file,
to reduce the number of times that you need to call `archil checkout`. These operations are:
* Creating a new file inside of a directory that does not have a delegation
* Opening a file that is not covered by a delegation for writing (delegation automatically released on close)
* Creating a new directory
These operations will automatically grant a delegation to the client who ran the operation. If you want
other clients to have write-access to these resources, you will need to manually call [`archil checkin`](/reference/archil-cli#checkin) on
the resource after creation.
These rules lead to a common pitfall with Archil in which it's possible to create a file or directory, but a removal
operation will fail with `EROFS: Read-only file system`. This is because of the automatic delegation granting on
file/folder creation. Deleting a file or folder is a directory-level operation, and requires a delegation on the parent
directory of the file being removed.
These rules apply only to clients in **Shared mode**. Clients mounting in **Conditional mode** will never have delegations
automatically issued to them.
## Understanding delegations
Each delegation is an exclusive write-lock on a part of an Archil disk for a single client. When you call `archil checkout `,
all files and folders underneath that path will be included as part of the write delegation. This will prevent other write operations,
including [S3 PutObject](/protocols/s3-api) operations, from completing against those paths.
For example, consider a file system with the following structure:
```
/
├── group/
│ └── models/
│ ├── model1.txt
│ └── model2.txt
└── data/
└── data.txt
```
If one client calls `archil checkout group/` to get exclusive write-access to the "group" directory, it will also have exclusive
write access to the `group/models/` directory and its contents. Other clients will be able to continue to read these files and folders
but will not be able to write to them, or call PutObject. Unless it's mounted in **conditional mode**, this client would not be
able to edit any other part of the disk. Other clients are simultaneously able to call PutObject to add files to the `data/` directory
or get exclusive write access over that directory by calling `archil checkout data/`.
When your client is operating in **conditional mode**, write operations are executed on the server-side, and multiple clients can
simultaneously read and write to any part of the file system without a delegation.
## Releasing delegations
Delegations represent an exclusive write-lock on a part of the Archil disk. As a result, they need to be manually released
by clients to signal that they are no longer performing writes to that portion of the disk. You can do this by either unmounting
the disk from the server which requested the delegation or by manually calling `archil checkin `.
## Forcing ownership
In some cases, a client may crash or poweroff without cleanly unmounting or calling `archil checkin` to release a delegation.
If this happens, you will not automatically be able to write to the location where the client's delegation was held. Instead,
you will need to force release the delegation. This will cause any in-progress writes from the original client to be lost. This
mirrors "force detaching" a block device, such as an EBS or Hyperdisk disk.
You can force checkout a delegation by calling `archil checkout --force`, and you can force mount a disk into exclusive
mode by calling `archil mount --force`.
You can also remove delegations using the Archil API, if you don't have the disk mounted locally:
```typescript Typescript theme={null}
const disk = await archil.getDisk("dsk-abcdef")
for (const delegation of await disk.listDelegations()) {
// If the client which issued the delegation is no longer
// connected, revoke the delegation.
if (delegation.isOrphaned) {
await disk.revokeDelegation(delegation)
}
}
```
# S3 Object Storage
Source: https://docs.archil.com/data-sources/s3-object-storage
Configure S3-compatible storage providers as data sources for your Archil disk
Before Archil can read or write to your bucket, the bucket needs to grant Archil access. This is a one-time setup per bucket, regardless of how you use the disk afterwards (mount, `disk exec`, anything else).
The Archil console walks you through this when you create a disk in the UI. This page documents what the console is doing, for setting things up by hand or scripting it.
## Amazon S3
[ Amazon S3](https://aws.amazon.com/s3/) supports two ways to authorize Archil:
* **Bucket resource policy** — recommended. Archil's IAM role assumes access to your bucket via a resource policy you attach to the bucket. No long-lived credentials change hands.
* **Static IAM credentials** — paste an access key and secret key from an IAM user that has read/write access to the bucket. Simpler to set up, but you're responsible for rotating the credentials.
### Bucket resource policy
In the [AWS S3 console](https://console.aws.amazon.com/console/), select your bucket, open the **Permissions** tab, and edit the **Bucket policy**. Add the following statement, substituting `YOUR-BUCKET-NAME` and your disk's session ID for `YOUR-DISK-SESSION-ID` (shown in the [Archil console](https://console.archil.com), e.g. `asi_us-east-1_tl02r28ami6phda70d1pv`):
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowArchilAccess",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::904233114299:role/archil-s3.prod.us-east-1"
},
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME",
"arn:aws:s3:::YOUR-BUCKET-NAME/*"
],
"Condition": {
"StringEquals": {
"aws:PrincipalTag/archilSessionId": "YOUR-DISK-SESSION-ID"
}
}
}
]
}
```
The `archilSessionId` condition scopes the grant to a single Archil disk, so the bucket is only reachable by that one disk — every other disk in your organization is denied. This is the right default if you point one disk at the bucket.
The action list above is the minimum Archil needs: `s3:ListBucket` for bucket existence checks and listings, `s3:GetObject` for reads, `s3:PutObject` for writes (including multipart uploads), and `s3:DeleteObject` for object removal.
Archil doesn't call `AbortMultipartUpload` directly, so it's not required in the policy. However, on rare failure paths a multipart upload can be left incomplete, and the parts continue to incur storage cost until they're cleaned up. We recommend either adding `s3:AbortMultipartUpload` to the action list, or attaching an [S3 lifecycle rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html) that aborts incomplete multipart uploads after a few days.
The exact `Principal` ARN varies by Archil region — use the table below to find yours, or copy the pre-filled policy from the [Archil console](https://console.archil.com).
| Cloud | Region | Principal ARN |
| - | - | - |
| AWS | us-east-1 | `arn:aws:iam::904233114299:role/archil-s3.prod.us-east-1` |
| AWS | us-east-2 (preview) | `arn:aws:iam::787372716928:role/archil-s3.prod.aws.us-east-2` |
| AWS | us-west-2 | `arn:aws:iam::795289710871:role/archil-s3.prod.aws.us-west-2` |
| AWS | eu-west-1 | `arn:aws:iam::794252461707:role/archil-s3.prod.aws.eu-west-1` |
| GCP | us-central1 | `arn:aws:iam::025016447135:role/archil-s3.prod.gcp.us-central1` |
#### Authorizing every disk in your organization
If you point many disks at the same bucket — for example, one disk per tenant or per workspace — swap the condition to match on `archilAccountId` instead. That's your Archil organization ID (shown in the [Archil console](https://console.archil.com), e.g. `org_3CjfHc6i0vXj24vu0Wpl8twU3nk`) and it's the same for every disk you create:
```json theme={null}
"Condition": {
"StringEquals": {
"aws:PrincipalTag/archilAccountId": "YOUR-ARCHIL-ACCOUNT-ID"
}
}
```
With this condition every current and future disk in your organization can use the bucket without you having to edit the policy again. The trade-off is that any disk in your organization can reach this bucket, so use it when you control all the disks under that organization.
### Server-side encryption with KMS (SSE-KMS)
If your bucket is encrypted with **SSE-S3** (the default `AES256` option), nothing extra is needed — the bucket policy above is sufficient.
If your bucket is encrypted with **SSE-KMS** using a customer-managed KMS key, you also need to grant Archil's IAM role permission to use that key. Open the key in the [AWS KMS console](https://console.aws.amazon.com/kms/) (the key's ARN is shown at the top of its detail page), edit the **Key policy**, and add the following statement — substituting `YOUR-KMS-KEY-ARN`:
```json theme={null}
{
"Sid": "AllowArchilToUseKey",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::904233114299:role/archil-s3.prod.us-east-1"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey",
"kms:DescribeKey"
],
"Resource": "YOUR-KMS-KEY-ARN"
}
```
`kms:Decrypt` lets Archil read objects, `kms:GenerateDataKey` lets it write new objects under envelope encryption, and `kms:DescribeKey` is occasionally used during client setup.
SSE-KMS with the AWS-managed `aws/s3` key does **not** work for cross-account access. AWS-managed keys can't be shared outside the account that owns them, so a bucket encrypted with `aws/s3` is unreachable from Archil's role. To use SSE-KMS with Archil, the bucket must use a [customer-managed KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk).
Archil never sets `ServerSideEncryption` headers on its requests, so whatever default encryption you configure on the bucket is what objects land with. You don't need to tell Archil which key you're using.
### Static credentials
If you'd rather not edit the bucket policy, create an IAM user with `s3:*` permissions on the bucket and pass its access key ID and secret to Archil when you create the disk:
```typescript TypeScript theme={null}
import * as archil from "disk";
await archil.createDisk({
name: "my-disk",
mounts: [{
type: "s3",
bucketName: "my-bucket",
accessKeyId: "AKIA...",
secretAccessKey: "...",
}],
});
```
```python Python theme={null}
import archil
from archil import S3Mount
archil.create_disk(
name="my-disk",
mounts=[S3Mount(
bucket_name="my-bucket",
access_key_id="AKIA...",
secret_access_key="...",
)],
)
```
## Google Cloud Storage
[ Google Cloud Storage](https://cloud.google.com/storage) authorizes Archil through static AWS-compatible HMAC credentials.
1. Open the [Google Cloud Storage console](https://console.cloud.google.com/storage/).
2. Click **Settings**, then **Interoperability**.
3. Under **Service account HMAC**, click **Create a key for another service account**.
4. Grant the **Cloud Storage** → **Storage Object Admin** role to the new service account.
5. Record the **Access key** and **Secret** for the new service account HMAC key.
Pass them when you create the disk:
```typescript TypeScript theme={null}
await archil.createDisk({
name: "my-disk",
mounts: [{
type: "gcs",
bucketName: "my-bucket",
accessKeyId: "GOOG1EABCD1234567890",
secretAccessKey: "1234567890abcdef1234567890ABCD/EF",
}],
});
```
```python Python theme={null}
from archil import GCSMount
archil.create_disk(
name="my-disk",
mounts=[GCSMount(
bucket_name="my-bucket",
access_key_id="GOOG1EABCD1234567890",
secret_access_key="1234567890abcdef1234567890ABCD/EF",
)],
)
```
## Cloudflare R2
[ Cloudflare R2](https://www.cloudflare.com/r2/) authorizes Archil through static AWS-compatible credentials.
1. Open the [Cloudflare console](https://dash.cloudflare.com/login).
2. Browse to **R2** and click **Manage R2 API Tokens**.
3. Create a new token with **Object Read & Write** permissions.
4. Record the **Access Key ID**, **Secret Access Key**, and the default **endpoint** (looks like `https://.r2.cloudflarestorage.com`).
Pass them when you create the disk:
```typescript TypeScript theme={null}
await archil.createDisk({
name: "my-disk",
mounts: [{
type: "r2",
bucketName: "my-bucket",
accessKeyId: "R2_ACCESS_KEY_ID",
secretAccessKey: "R2_SECRET_ACCESS_KEY",
endpoint: "https://.r2.cloudflarestorage.com",
}],
});
```
```python Python theme={null}
from archil import R2Mount
archil.create_disk(
name="my-disk",
mounts=[R2Mount(
bucket_name="my-bucket",
access_key_id="R2_ACCESS_KEY_ID",
secret_access_key="R2_SECRET_ACCESS_KEY",
bucket_endpoint="https://.r2.cloudflarestorage.com",
)],
)
```
## Generic S3-compatible storage
Many providers expose S3-compatible APIs and work with Archil out of the box. To configure one:
1. Get the S3-compatible API endpoint from your provider.
2. Create or obtain access credentials (Access Key ID and Secret Access Key) with read/write/list/delete permissions on the bucket.
3. When creating your disk, select **Generic S3 Compatible** as the data source type and provide the endpoint plus credentials.
Tested with:
* [ MinIO](https://min.io/)
* [ DigitalOcean Spaces](https://www.digitalocean.com/products/spaces)
* [ Wasabi](https://wasabi.com/)
* [ Backblaze B2](https://www.backblaze.com/b2/) (with S3-compatible API)
* And many others
# Architecture
Source: https://docs.archil.com/details/architecture
Overview of Archil disk architecture
Archil disks deliver infinite, high-performance storage and provide instant, local-like access to massive data
sets, wherever they're stored. Archil is designed as a high-speed caching layer that sits between your
servers and your data sources, like Amazon S3, providing a fully POSIX-compatible interface to your data.
With Archil's custom file protocol, developers can achieve high-speed access to shared data, even if it requires
lots of small files.
## Core components
Connect to disks using the Archil client
High-performance, durable caching layer
Your S3-compatible storage where data is durably stored
The instances mounting Archil disks (client instances) and the data sources run in your existing cloud accounts. The Archil storage system runs as a service
in Archil-managed cloud accounts. If your organization requires that all software be hosted in your own cloud account,
[contact us](mailto:support@archil.com) to learn more about our BYOC options for organizations with 100 TiB of data
or more.
### Archil file protocol
The Archil client implements a new, custom storage protocol that enables full POSIX-compliance and
high-performance for small-file workloads, such as git. The Archil client supports all Linux distributions.
For more information, see the reference documentation for the [Archil client](/reference/archil-cli).
### Archil storage system
Archil disks are hosted on a shared cloud service, the Archil storage system. This centralized, durable cache
stores frequently accessed data and recently written data. The cache provides near-infinite storage that automatically
expands to the working set size of your application. The Archil storage service is designed to provide sub-millisecond
latency for data access.
Because the Archil cache is shared across all clients, you can achieve low-latency access to reference data
from multiple servers simultaneously.
The Archil storage system efficiently translates POSIX-like requests into other protocols, such as Amazon S3, even when
those operations (such as rename) are not supported by the underlying data source. Archil disks are always strongly
read-after-write consistent for all connected clients. Synchronization between the Archil storage system and the underlying
data sources is eventually completed, and generally completes in less than 5 minutes.
### Security
Archil encrypts all data at rest (AES-256) and in transit (TLS 1.3), with access control via IAM
roles and scoped API keys. Archil is SOC 2 certified and HIPAA/GDPR compliant. For full details,
see [Security & Compliance](/details/security).
### Data durability
Archil is designed to provide 99.999% (5 nines) durability for newly written data before it is synchronized to S3, and a
**99.99% (4 nines) uptime SLA** for disk availability.
Archil's caching layer automatically stores data redundantly within multiple Availability Zones in the disk's region.
A successful `fsync()` guarantees that the written data has been committed redundantly across these Availability Zones;
the 5-nines durability applies from the moment `fsync()` returns, before the data has synchronized to your data source.
There is no single point of failure that can cause data loss. An Archil disk is not a single server or cache node — if an
individual node, or even an entire Availability Zone, fails before your data has synchronized to the data source, the disk
continues serving your data from the remaining Availability Zones and synchronization completes normally. No data is lost,
and no action is required on your part.
# Client caching
Source: https://docs.archil.com/details/caching
How the Archil client caches data in memory, and the controls for warming, sizing, and refreshing it.
Archil caches your data at two distinct layers:
* The **Archil storage system cache** is a durable, multi-Availability-Zone caching layer that runs server-side, between your clients and your [data source](/concepts/data-sources). It's where your active data is durably stored (a successful `fsync()` commits to it) and where it's [metered](/concepts/metering), and it serves data with sub-millisecond latency. See [Architecture](/details/architecture) for how it fits in.
* The **client-side cache** is an in-memory cache inside each Archil client — one per mount. It holds recently used data so that repeated access is served locally, without even a round-trip to the storage system.
This page is about the **client-side cache**: what it holds, how it's sized and evicted, and the commands you use to warm, refresh, and tune it.
## What the client caches
The client maintains several in-memory caches, all populated on demand as your application reads and writes:
* **File data** — the contents of files. This is the bulk of the cache.
* **File and directory attributes** — metadata such as size, timestamps, and permissions (the result of a `stat`).
* **Directory listings** — the entries returned by `readdir` (i.e., `ls`), so repeated traversals of the same directory don't round-trip to the server.
* **Extended attributes** — when [xattrs](/reference/archil-cli#mount) are enabled.
When your application requests data that isn't in the client cache, the client fetches it from the Archil storage system — which serves it from its durable cache (sub-millisecond) or, on a miss there, from the backing data source (for Amazon S3, typically 10–30 ms) — and then caches the result in memory for future requests.
## Cache size and eviction
The client's in-memory data cache is bounded by two settings on [`archil mount`](/reference/archil-cli#mount):
* `--max-cache-mb` — the hard ceiling on the client cache size. Default: 1/4 of available RAM or 2048 MiB, whichever is smaller.
* `--target-cache-mb` — the size the client aims to keep the cache near, leaving headroom for bursts of new reads. Default: 75% of the maximum.
When the cache fills, the client evicts older entries to make room for new data. Two kinds of entries are never evicted:
* **Unflushed (dirty) writes** — data you've written that hasn't yet been committed to the storage system is always preserved locally, so eviction can never lose your writes.
* **Entries still in use** — anything an application currently has open is kept until it's released.
Because the client cache is bounded, reading or pre-fetching more data than `--max-cache-mb` can hold will evict earlier entries before you use them. Size the cache for your working set when you plan to keep a large data set hot on a client.
For example, to give a read-heavy client a larger cache:
```bash theme={null}
archil mount /mnt/archil --region aws-us-east-1 \
--max-cache-mb 16384 --target-cache-mb 12288
```
## Freshness and consistency
Serving reads from the client cache is what lets a client avoid contacting the storage system on every operation — which also means a client can briefly serve slightly stale data after another client changes a file. The client revalidates cached entries with the storage system so that changes become visible within seconds for file data and attributes. Directory listings are cached for longer; you can tune their lifetime per directory with [`archil set-cache-expiry`](/reference/archil-cli#set-cache-expiry).
For the full model — including how `fsync` and cache flushes provide strong read-after-write consistency between connected clients — see [Consistency](/details/consistency).
## Warming the client cache
By default the client cache fills lazily as data is read, so the first read of any file on a client pays the fetch latency. When you know which files you'll need — loading a model, starting a database, or running a batch job over a known set of inputs — you can warm them ahead of time so the first read is already local.
There are two ways to warm the client cache:
* **[`archil prefetch ...`](/reference/archil-cli#prefetch)** — pre-fetch files or directories (recursively) into the cache of an already-mounted disk. The command queues the work and returns immediately; fetching continues in the background.
* **[`archil mount --pre-fetch `](/reference/archil-cli#mount)** — warm a comma-separated set of paths automatically as soon as the disk is mounted.
```bash theme={null}
# Warm a model directory on an already-mounted disk
archil prefetch /mnt/archil models/llama-3
# Or warm it as part of the mount
archil mount /mnt/archil --region aws-us-east-1 --pre-fetch models/llama-3
```
Pre-fetched data shares the client cache with normal reads, so it counts against `--max-cache-mb` and is subject to the same eviction described above.
The client also reads ahead automatically: when it detects sequential reads of a file, it fetches up to 4 MiB beyond
them. Change the window with [`mount --read-ahead `](/reference/archil-cli#mount), or pass `0` to disable it.
## Refreshing the client cache
To force a client to drop cached entries and re-read from the storage system — for example, when another client has updated files on a [`--shared`](/concepts/shared-disks) mount — use [`archil invalidate-cache`](/reference/archil-cli#invalidate-cache). Pass the mount root to refresh everything, or a single file path to refresh just that file. In-flight (dirty) writes are always preserved.
```bash theme={null}
# Refresh the whole mount
archil invalidate-cache /mnt/archil
# Refresh a single file (siblings keep their cache)
archil invalidate-cache /mnt/archil/shared-data/report.csv
```
## Controls at a glance
| Goal | Control |
| - | - |
| Set the maximum client cache size | [`mount --max-cache-mb`](/reference/archil-cli#mount) |
| Set the target client cache size | [`mount --target-cache-mb`](/reference/archil-cli#mount) |
| Warm files into the client cache after mount | [`prefetch`](/reference/archil-cli#prefetch) |
| Warm files into the client cache at mount time | [`mount --pre-fetch`](/reference/archil-cli#mount) |
| Tune sequential read-ahead | [`mount --read-ahead`](/reference/archil-cli#mount) |
| Drop and re-read cached data | [`invalidate-cache`](/reference/archil-cli#invalidate-cache) |
| Tune directory-listing cache lifetime | [`set-cache-expiry`](/reference/archil-cli#set-cache-expiry) |
# Solution Comparison
Source: https://docs.archil.com/details/comparison
Archil is designed to replace block storage
When designing a new application for the cloud, picking a storage layer shouldn't be the hard part.
Archil is designed to be the simplest storage solution that delivers good performance for the
workloads people actually run in the cloud: applications, databases, model training, data pipelines,
and the long tail of stateful systems that need a real file system underneath them.
You don't need to capacity-plan, choose between block storage and file storage, or run a replication
or backup tier. Disks are mountable from any server, data lives in S3 in its native object format,
and performance is good enough that you can stop thinking about your storage layer and focus on
your application. For most workloads, Archil is the right default.
## When you might want something else
Archil is a network file system, so the cache-miss path always includes a network round-trip. A few
cases where a different tool is a better fit:
* **You need microsecond latency on every operation regardless of working set size.** Use local
NVMe.
* **You already operate a Lustre cluster with HPC tooling and storage admins.** Stay on Lustre.
* **Your application speaks S3 natively and doesn't need POSIX or shared access across servers.**
Just use S3.
## How Archil compares to other solutions
The sections below explain how Archil differs from each of the common alternatives in more detail.
### Block storage (e.g. EBS or Hyperdisk)
Block storage, such as EBS or Hyperdisk, are designed to provide high-performance, low-latency storage for your applications. While they provide
full POSIX compatibility, they require manual capacity provisioning in advance, which can be challenging to predict accurately. They also cannot be shared
between multiple servers and do not synchronize with object storage like S3. This means each server needs its own dedicated disk and must independently
download data from S3, leading to redundant data transfers and higher costs. When you need to scale up your storage, you have to manually provision
additional capacity and pay for the full provisioned amount regardless of actual usage.
### Native FUSE adapters (e.g. S3FS and Mountpoint)
Existing FUSE adapters, such as S3FS and Mountpoint for Amazon S3, are not designed to support full POSIX semantics. For example, Mountpoint for Amazon
S3 does not support file locks, random writes, renames, or symbolic links. In order to use these tools, you need to instrument your file applications
to understand whether or not they use these unsupported operations, which can be difficult and error-prone. In addition, these tools do not provide
shared caching for reference data sets to multiple clients, so large clusters may need to re-download the same data from S3 repeatedly. In comparison,
Archil is designed to support all standard POSIX file operations, which means that it's out-of-the-box compatible with all file applications.
### Block FUSE adapters (e.g. JuiceFS and ObjectiveFS)
Other disk adapters for S3, such as JuiceFS, ObjectiveFS, or ZeroFS, require that you store data in your S3 bucket in a proprietary block format. This means
that you cannot use your existing data sets in S3 with these tools, and these buckets aren't compatible with any existing S3 workflows that you use. Archil
stores data in S3 in its native format, so you can simultaneously use and share the data from both a disk and S3. You can also use Archil to connect to
existing data sets in S3.
### NFS file storage (e.g. Amazon EFS and Amazon S3 Files)
NFS-based file systems, such as Amazon EFS and the newer Amazon S3 Files, are serverless and can be
mounted from many servers at once, but the NFS protocol introduces two fundamental limitations that
make them unsuitable for many workloads.
The first limitation is metadata latency. NFS clients cannot cache the result of mutating metadata
operations, so every file creation, rename, or attribute change must round-trip to the server before
it can be acknowledged. As a result, metadata-heavy workloads — small files, source trees, package
installs, container image extracts, and most interactive applications — are typically an order of
magnitude slower on NFS than on local or block storage like EBS. Archil's custom file protocol allows
clients to take exclusive write access to portions of the file system, so creates and writes can be
acknowledged locally without a server round-trip. This delivers metadata latency comparable to EBS
or local NVMe, even for the small-file workloads that NFS handles poorly.
The second limitation is per-client throughput. NFS clients resolve DNS once at mount time and send
every subsequent request to a single IP address, which caps a single client at roughly 3 GiB/s
regardless of how much network bandwidth the underlying instance has. This is well below the NIC
capacity of large GPU instances, making NFS a bottleneck for model training and other
high-throughput workloads. Archil's protocol allows a single client to talk directly to multiple
storage servers in parallel — similar to how parallel file systems like Lustre achieve scale-out
throughput — so a single client can saturate the NIC of even the largest GPU instances.
Archil also stores data in S3 in its native object format, so the same data is accessible through
both the file system and the underlying S3 bucket without any proprietary on-disk layout.
### Shared file storage (e.g. Alluxio and FSx for Lustre)
Some solutions, like Alluxio or FSx for Lustre, require that you either deploy and manage your own infrastructure or that you provision capacity. It's
often difficult to predict the specific storage needs of an application, especially if the application can have unexpected or bursty needs. Archil is a
fully managed, serverless solution that automatically scales with your application. You don't need to provision capacity or deploy infrastructure, and you're
only billed for what you use.
***
To estimate your costs, try the [pricing calculator](https://archil.com/pricing).
# Consistency
Source: https://docs.archil.com/details/consistency
Understanding Archil's consistency model between clients and data sources
Consistency in Archil is divided into two different models:
* Strong "disk consistency" for clients which are connected directly to the Archil disk
* Eventual "synchronization consistency" for clients which are connected directly to the attached data sources
![Consistency Model]()
## Strong disk consistency
When clients are connected directly to the Archil disk, Archil offers strong read-after-write consistency. Specifically, any time a client issues an
"fsync" operation to the file system, or Archil proactively flushes its cache, the client will push all latest writes to the Archil disk. A successful
`fsync` means the data is durably committed — stored redundantly across multiple Availability Zones, as described in
[Data durability](/details/architecture#data-durability). When another client issues
a read request to the disk, it will see the latest writes immediately. Operations such as rename are atomic — another client's next read of the
affected path returns either the old state or the new state, never a partial result.
### Client-side caching
To enhance performance, the Archil client aggressively caches data and directory information from the server. Therefore, not all reads will result
in a request to the server -- where data is strongly consistent. This can result in delays of seconds before new files or changes to existing files are
reflected in a second disk client.
The client-side cache configuration is adjustable using the `archil set-cache-expiry` command. Read more about the command [here](/reference/archil-cli#set-cache-expiry).
You can also force a client to drop its cached entries and re-read from the server with [`archil invalidate-cache`](/reference/archil-cli#invalidate-cache).
## Eventual synchronization consistency
Archil disks provide eventual consistency between the file system view and the backing data source. For example, an object that is created directly in
Amazon S3 will not appear in the Archil disk (via tools like `ls`) for a few minutes after creation. Similarly, a file that is created in the Archil disk
(using a tool such as `touch`) can take tens of seconds to appear when calling the S3 API directly (such as by using `aws s3 ls`).
### Durability during synchronization
The synchronization delay affects only *visibility* through the data source's APIs — data that has been committed to the disk is not at risk
while it waits to synchronize. The Archil disk is itself a highly available, durable storage system, not a single server: committed data is
stored redundantly across multiple Availability Zones, with no single point of failure that can cause data loss. If an Archil node — or an
entire Availability Zone — fails before data has synchronized to the data source, no data is lost: the disk continues serving it from the
remaining Availability Zones and completes synchronization normally. See
[Data durability](/details/architecture#data-durability) for the full durability design.
# Performance
Source: https://docs.archil.com/details/performance
Archil Performance Specifications
Archil disks are designed to provide enterprise-grade performance while synchronizing with data sets in S3.
### Caching
Archil intelligently caches data to maximize the performance of your disk. If your application requests data that's not
in the cache, Archil will fetch the data from the backing bucket and then cache it for future requests. Archil keeps this data in
cache for up to 1 hour after the final read or write operation to the data. Archil may cache up to 10 MiB of additional data
from the backing bucket than requested.
### Latency
Archil disks are designed to provide consistent, sub-millisecond time-to-first-byte latencies for all cached
read and write operations. Uncached read latencies depend on the specific backing storage provider. For example,
for Amazon S3, uncached reads are typically in the 10-30ms range.
Mounting a disk is fast — `archil mount` typically completes in around 100ms — so disks can be attached on demand
in latency-sensitive and ephemeral environments such as per-request sandboxes.
### Scalability
By default, Archil disks can provide up to 10 GB/s of throughput and 10,000 IOPS across all connected clients.
To request higher throughput or IOPS limits for your disk, please contact us at [support@archil.com](mailto:support@archil.com).
# Security & Compliance
Source: https://docs.archil.com/details/security
How Archil protects your data, and what we commit to contractually
Archil stores your data in **your** cloud storage bucket, encrypted end-to-end, and never in a
proprietary format. This page describes how we protect your data and what we're prepared to sign.
## Data ownership
Your data lives in your S3 bucket (or other [data source](/concepts/data-sources)) in its native
format. Archil reads and writes to your bucket on your behalf — we do not copy your data into a
separate storage system.
If you stop using Archil, your data stays in your bucket, fully accessible through the S3 API.
There is no lock-in and no export process.
## Hosting
Archil runs on AWS infrastructure in the [region](/reference/regions) you select for your disk.
Data does not leave that region unless you explicitly configure cross-region replication.
Archil's storage layer is designed for **99.999% (five nines) durability**, achieved through
replication across multiple Availability Zones and continuous synchronization to your data source
(S3 provides 99.999999999% durability). Formal uptime SLAs are available on
[Enterprise plans](/administration/billing).
## Encryption
All data is encrypted at rest and in transit.
* **At rest** — Data in Archil's storage layer is encrypted with AES-256-GCM via AWS KMS. Data
in your S3 data source inherits your bucket's encryption settings, so your existing key
management policies apply.
* **In transit** — All client-to-service connections use TLS 1.3. Unencrypted connections are
rejected.
Encryption keys are managed by Archil's key management infrastructure and rotated on a regular
schedule.
## SOC 2
Archil undergoes ongoing SOC 2 Type II audits covering security, availability, and confidentiality.
Audit reports are available through the [Archil Trust Center](https://security.archil.com/).
If you need a copy of our most recent report for your procurement or compliance review,
[contact us](mailto:security@archil.com).
## Data protection (GDPR & CCPA)
Archil is GDPR and CCPA compliant. We provide **Data Processing Agreements (DPAs)** to
[Enterprise plan](/administration/billing) customers. The DPA covers data residency,
[subprocessors](/legal/subprocessors), and breach notification commitments.
Data residency is respected by default — Archil does not move data across regions without explicit
configuration.
To request a DPA, [contact us](mailto:security@archil.com).
## HIPAA
Archil supports workloads subject to HIPAA requirements. We sign **Business Associate Agreements
(BAAs)** with [Enterprise plan](/administration/billing) customers.
To request a BAA, [contact us](mailto:security@archil.com).
## Access control
Archil provides layered access control to match your environment:
* **AWS IAM roles** — Clients authenticate with their existing IAM roles. No additional
credentials needed for EC2, Lambda, or ECS. See [Disk Users](/concepts/disk-users).
* **Disk tokens** — For non-AWS or cross-region access, a [disk token](/concepts/disk-users#disk-token-authorization)
grants a client mount access to one specific disk. Disk tokens are hashed before storage —
Archil never stores them in plaintext.
* **API keys** — The [Control Plane API](/api-reference/introduction) uses account-scoped API keys
for managing disks, disk tokens, and users programmatically. API keys are separate from disk tokens.
* **POSIX permissions** — Standard file permissions (owner, group, other) are enforced on all
mounted disks.
## Security testing
Archil partners with [Casco Security](https://casco.com/) for monthly penetration
testing across our infrastructure. We also run continuous static analysis on every code change.
## Vulnerability disclosure
If you discover a security vulnerability in Archil, please report it to
[security@archil.com](mailto:security@archil.com). We target acknowledgment within 24 hours.
We offer bounties for qualifying vulnerability reports — email us for details.
## Private networking
For organizations requiring private connectivity (e.g., AWS PrivateLink),
[contact us](mailto:support@archil.com) to discuss options.
# Application Compatibility
Source: https://docs.archil.com/details/support
Archil is designed for full POSIX-compatibility
Archil's FUSE-based file system is designed to be fully compatible with *all* applications which run on a local, POSIX
file system. If you discover behavior that you believe deviates from POSIX, please contact us by emailing
[support@archil.com](mailto:support@archil.com), and we will address the underlying issue. We are continuously testing
our adherence to POSIX, using the same test suite as the XFS file systems.
## POSIX compatibility details
Archil disks are designed to be broadly compatible with POSIX semantics, even when the underlying data sources are not POSIX
compatible, like Amazon S3. When needed, the [Archil storage service](/details/architecture) emulates file operations on
data sources. Today, Archil supports the following POSIX features:
* All read and write operations, including random writes and appends
* Symbolic links (`ln -s`)
* Hard Links (`ln`)
* Atomic renames, including directory renames (`mv`)1
* POSIX locks, including `flock` and `fcntl`2
* Anonymous files (unlink after opening)
* File modes
* File ownership
* Extended attributes (with the `--enable-xattrs` flag)
**1.** A rename is atomic — a reader on another client sees either the old name or the new name, never a partial result. Archil's storage system is fully consistent; a client's cached view can lag briefly and refreshes with [`archil invalidate-cache`](/reference/archil-cli#invalidate-cache).
**2.** POSIX locks (`flock`/`fcntl`) are honored between processes on a single client, not across clients that share a disk. Today only one client can open a file for writing, so cross-client lock contention doesn't arise; coordinate access across clients with the [checkout/checkin ownership model](/concepts/shared-disks).
**We are actively working on adding support for the following features:**
* File ACLs
* `renameat2` flags, such as `RENAME_WHITEOUT`
## Application support
As a result of our broad compatibility with POSIX, Archil is compatible with all applications which run on
a local, POSIX file system. This includes:
* **Version control systems**, such as Git (including large repositories, `git clone`, `git checkout`, and `git status`), Mercurial, and Subversion
* **Databases**, such as PostgreSQL, MySQL, and SQLite
* **Build tools and compilers**, such as Make, CMake, GCC, and Clang
* **Archive and compression tools**, such as `tar`, `gzip`, `zip`, and `rsync`
* **Editors and IDEs**, such as Vim, Emacs, and VS Code
* **Containers** — [mount in a container](/mounting/containers) via a bind mount from the host or FUSE inside the container
## Limits
* **Path length** — on a disk synchronized to a data source, a file's full path can't exceed the data source's object-key length limit (1,024 bytes for Amazon S3, including the disk's bucket prefix). Longer paths fail with `ENAMETOOLONG`.
* **File names** — on a disk synchronized to a data source, file and directory names must be valid UTF-8.
# Introduction
Source: https://docs.archil.com/getting-started/introduction
Archil provides the infrastructure that makes it simple to connect real-world AI
applications to their data. Out of the box, Archil provides everything that you need
to build production-scale stateful agents or agent platforms for your organization.
You can use Archil to:
* **Build and give agents access to petabyte-scale context**, using our [infinite, shared disks](/concepts/shared-disks)
and optionally synchronizing that data from sources like [Amazon S3 object storage](/data-sources/s3-object-storage).
* **Run untrusted AI-generated code against context** with tools based on [serverless execution](/compute/serverless-execution) or traditional [persistent sandboxes](/compute/sandboxes/introduction).
* **Share artifacts between agents and humans** with a strongly consistent [REST API](/protocols/s3-api) that allows
you to easily embed file watching and uploading into any website.
```typescript Vercel AI SDK theme={null}
import { generateText } from "ai";
import * as archil from "disk";
import { createDiskTools } from "@archildata/ai-sdk";
const disk = await archil.getDisk('disk-1234');
const result = await generateText({
model: "anthropic/claude-sonnet-4.6",
prompt: "What files do you have access to?",
tools: createDiskTools(disk),
});
```
```typescript Archil Typescript SDK theme={null}
import * as archil from "disk";
const myDisk = await archil.getDisk('disk-1234');
// Easily hand off between bash and REST
//
await myDisk.exec('curl google.com > google.txt');
console.log(await myDisk.getObject('google.txt'))
```
If you're building agents, the easiest way to get started with Archil is with one of our
integrations with popular agent frameworks like [AI SDK](/integrations/ai-sdk), [eve](/integrations/eve),
[Mastra](https://www.npmjs.com/package/@archildata/mastra), or [LangChain](https://www.npmjs.com/package/@archildata/langchain).
For more control, use our [TypeScript SDK](/sdks/typescript) or [Python SDK](/sdks/python) directly.
## Features
Archil's storage primitives support a rich set data management features to make it
simple to build complex, agentic workflows.
* [Branches and checkpointing](/concepts/branches-and-checkpoints) make it possible to have simultaneous agents run on the same data set, and pick the best output.
* [Native SQLite support](/compute/sqlite) allows you to easily create databases on your file system, and serverlessly read and write to them from anywhere.
* [Immutable log support](/protocols/s3-api#appending) simplifies how you build log-like abstractions on your disk.
Archil also integrates directly with most sandbox providers (such as [Daytona](https://www.daytona.io/docs/en/mount-external-storage/#mount-an-archil-disk),
[e2b](https://e2b.dev/docs/storage/archil), and [Superserve](https://docs.superserve.ai/storage/archil)),
and can be mounted from [Kubernetes clusters](/reference/csi-driver) or provisioned with [Terraform](/reference/terraform).
## Pricing
Archil's storage and sandboxes are fully elastic, and you only pay for the resources that you're actively using.
For example, if you're using an Archil disk that synchronizes to an S3 bucket that you own, you only pay Archil
for the data that is being accessed (either recently read or written to). When your data isn't being accessed, it
only lives in your S3 bucket, and you pay Archil nothing.
Archil's storage also comes with **free sandbox usage**. For every 1 TB in Active Data, you receive 1,000 minutes of
free sandbox compute. This means that most agentic workloads using Archil no longer have a compute bill.
# Quickstart
Source: https://docs.archil.com/getting-started/quickstart
This page will walk you through how to set up an Archil disk and build a real agent
with the [AI SDK](https://ai-sdk.dev) in less than 5 minutes that uses real tools,
like `bash`, against your data. With Archil, you only pay for the infrastructure that you
use, which means that this entire tutorial should fit inside of our free plan.
For this tutorial, let's build an agent that writes a storybook for us, creates a nicely
formatted PDF document of our book, and delivers an audit-log of every tool that it used to
create the book -- **without the use of a separate sandbox**.
This tutorial does not walk through mounting the Archil file system. If you already have
a server you want to attach Archil to, see [Mount on Linux (FUSE)](/mounting/linux) (or [macOS](/mounting/macos) / [in a container](/mounting/containers)).
Sign in to [the Archil console](https://console.archil.com/api-keys), create an API key from the
**API keys** page, and export it alongside a key for your model provider:
```bash theme={null}
export ARCHIL_API_KEY=key-...
export ARCHIL_REGION=aws-us-east-1
export ANTHROPIC_API_KEY=sk-ant-...
```
Create your API key in the region closest to where your users are, for this tutorial, you need
to use one of the Archil AWS regions where sandboxes are offered (`aws-us-east-1`, `aws-us-west-2`, or `aws-eu-west-1`); see [Region availability](/reference/regions).
In your project directory, install the required dependencies for this tutorial:
```bash theme={null}
npm install ai @ai-sdk/anthropic disk @archildata/ai-sdk
```
Archil provides the file system workspace for agents to store files, manipulate context, and run complex commands -- while providing
concurrent access to that workspace from your web application for humans. Each Archil account supports an unlimited number of file systems
so we recommend making one for each agent that you plan to run. Make one for our tutorial agent:
```bash npx theme={null}
npx disk create tutorial-agent-disk
```
```typescript Typescript theme={null}
import * as archil from "disk";
const { disk: agentDisk } = await archil.createDisk({ name: "tutorial-agent-disk" });
```
The result includes a token that you can use to mount the disk directly. You do not need it for this tutorial.
If you have an existing S3 or GCS bucket you'd like the agent to work against, you can specify it as a [data source](/concepts/data-sources)
when creating your disk -- see [S3 Object Storage](/data-sources/s3-object-storage). When specifying a data source, your disk
will mirror the contents of your bucket.
Let's build our agent to now write us a short story. We use a [ToolLoopAgent](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool-loop-agent)
from the AI SDK to iteratively call tools until we accomplish our goal. We'll go ahead and give the agent some instructions to use Archil
and connect our disk as a set of tools in [conditional mode](/concepts/sharing-disks).
```typescript theme={null}
import { anthropic } from "@ai-sdk/anthropic";
import { ToolLoopAgent } from "ai";
import { createDiskTools } from "@archildata/ai-sdk";
// Mount the disk as a workspace, opting the agent into our new,
// conditional mode semantics.
const workspace = archil.workspace({
workspace: { disk: agentDisk, conditional: true },
});
const agent = new ToolLoopAgent({
model: anthropic("claude-sonnet-5"),
instructions: [
"You are a writing assistant. Save your work as you go.",
"Everything happens in /mnt/archil/workspace — use that path for bash and for the file tools.",
"Each bash command runs in a fresh container, so anything you install belongs in the workspace if you want it next time.",
].join(" "),
tools: createDiskTools(workspace),
});
await agent.generate({
prompt: "Write a short story, format it nicely and save it as story.pdf",
});
```
That's the whole integration. `createDiskTools` automatically gives the agent a set of tools to manipulate data on the disk
without a separate sandbox which includes: read/write/delete files, run bash, glob, and grep.
You can view your story in the [Archil console](https://console.archil.com/), or by downloading the contents of the file:
```typescript theme={null}
import { writeFile } from "node:fs/promises";
await writeFile("story.pdf", await workspace.getObject("workspace/story.pdf"));
```
Next, let's extend our agent to make it ready for production by building an audit log of each tool call that was made while it ran.
We can easily build an append-only log on Archil by using the `AppendObject` function of our extended [S3 API](/protocols/s3-api#appending).
We can use the [onToolExecutionEnd](https://ai-sdk.dev/docs/ai-sdk-core/lifecycle-callbacks#ontoolexecutionend)
lifecycle hook in the AI SDK to record each tool call as it happens.
Replace the `agent.generate` line in your code with the following:
```typescript theme={null}
const auditPath = `.audit/${Date.now()}.jsonl`;
await agent.generate({
prompt: "Write a short story, format it nicely and save it as story.pdf",
onToolExecutionEnd: async ({ toolCall, toolOutput, toolExecutionMs }) => {
const entry = {
tool: toolCall.toolName,
input: toolCall.input,
ms: Math.round(toolExecutionMs),
result: toolOutput.type === "tool-result" ? toolOutput.output : toolOutput.error,
};
await agentDisk.appendObject(auditPath, `${JSON.stringify(entry)}\n`);
},
});
```
Now, you can view your audit log by reading the contents of the `auditPath` file.
```typescript theme={null}
const audit = new TextDecoder().decode(await agentDisk.getObject(auditPath));
for (const line of audit.trim().split("\n")) {
const { tool, ms, input } = JSON.parse(line);
console.log(`${tool.padEnd(11)} ${String(ms).padStart(6)}ms ${JSON.stringify(input).slice(0, 60)}`);
}
```
Which will output the audit log entries to the console that look like this:
```
run_bash 202ms {"command":"which pandoc wkhtmltopdf weasyprint pdflatex ens...
run_bash 3571ms {"command":"pip install --quiet reportlab 2>&1 | tail -20; e...
write_file 55ms {"path":"/mnt/archil/workspace/story.txt","content":"The Lig...
run_bash 3241ms {"command":"cd /mnt/archil/workspace && pip install --quiet...
```
You now know the basics of how to build agents with Archil with persistent disks that can scale to any size, have multiple
agents work in parallel, and provide strong-consistency for human clients who might interact using a web application via our
[S3-compatible API](/protocols/s3-api).
## Next steps
Mount several disks as one workspace, and scope what the agent can reach.
The runtime behind `run_bash` — fan out across containers, compose disks, no lifecycle to manage.
Branch a disk per agent, run them in parallel, and keep the best result.
Keep a sandbox warm across tool calls when the agent needs a long-lived machine.
Use the same disk as a POSIX filesystem from your own machines.
View Archil pricing and plan details.
# Bash tool for agents
Source: https://docs.archil.com/guides/ai/bash-tool
Give an AI agent a persistent file system and a bash tool, using disk.exec and the AI SDK.
This guide shows how to wire up an AI agent with a persistent workspace disk and a bash tool backed by [serverless execution](/compute/serverless-execution). The agent can `ls`, `cat`, `grep`, write files, run scripts — anything bash can do — against data that lives in your S3 bucket.
The whole thing is about 30 lines.
## 1. Create a workspace disk
```typescript TypeScript theme={null}
import * as archil from "disk";
const { disk } = await archil.createDisk({
name: "agent-workspace",
mounts: [{ type: "s3", bucketName: "agent-bucket" }],
});
```
```python Python theme={null}
import archil
from archil import S3Mount
disk = archil.create_disk(
name="agent-workspace",
mounts=[S3Mount(bucket_name="agent-bucket")],
).disk
```
The disk is immediately usable — the file system grows and shrinks with your bucket and survives every agent run.
## 2. Wrap `disk.exec` as a tool
Steps 2 and 3 below use [the Vercel AI SDK](https://sdk.vercel.ai/) (TypeScript). The pattern is identical from Python — wrap `Disk.exec` as a tool in whatever agent framework you use (for example, tool use with the [Anthropic Python SDK](/sdks/python)). Only the framework glue changes; the `Disk.exec` call is the same.
Using the Vercel AI SDK, an `exec` tool is a handful of lines:
```typescript theme={null}
import { generateText, tool } from "ai";
import { z } from "zod";
const bash = tool({
description: "Run a shell command inside the workdir.",
inputSchema: z.object({
command: z.string(),
}),
execute: async ({ command }) => {
const { stdout, stderr, exitCode } = await disk.exec(command);
return `exit ${exitCode}\n${stdout}${stderr ? `\n${stderr}` : ""}`;
},
});
```
A few things worth calling out about the tool shape:
* **Keep the description minimal.** The agent doesn't need to know it's on Archil, or that this is a "workspace disk" — those are implementation details that cost reasoning tokens. `"Run a shell command inside the workdir."` is enough.
* **Return a single string, not an object.** Prefix with `exit N` and let the model read the output. Splitting stdout and stderr into fields forces the model to think about which field to read first; most of the time it doesn't matter.
* **Don't over-describe the argument.** `z.string()` without a `.describe()` is fine — the model knows what a shell command is.
Under the hood, every call spins up a container with the file system as the working directory, runs the command, and returns the result.
## 3. Run the agent
```typescript theme={null}
const result = await generateText({
model: "openai/gpt-5.2",
prompt:
"List everything on the disk, then find every line in app.log that mentions ERROR.",
tools: { bash },
maxSteps: 10,
});
console.log(result.text);
```
The agent will:
1. Call `bash({ command: "ls" })` to discover what's there.
2. Call `bash({ command: "grep ERROR app.log" })` to find the errors.
3. Summarize the result.
Because the disk persists, running the same agent tomorrow sees the same files plus anything it wrote. You can also mount the same disk from a local laptop (`archil mount ...`) to inspect what the agent has been doing.
## Patterns
### Persistent agent memory
Write to a known path and the next agent run sees it:
```typescript TypeScript theme={null}
await disk.exec("echo 'learned fact' >> memory.jsonl");
```
```python Python theme={null}
disk.exec("echo 'learned fact' >> memory.jsonl")
```
### Fan-out across the bucket
A map-reduce across every file in a directory:
```typescript TypeScript theme={null}
const files = (await disk.exec("ls logs")).stdout.trim().split("\n");
const results = await Promise.all(
files.map((f) => disk.exec(`wc -l logs/${f}`)),
);
```
```python Python theme={null}
import asyncio
from archil import Archil
async def main():
async with Archil() as client:
d = await client.disks.get.aio("dsk-abc123")
files = (await d.exec.aio("ls logs")).stdout.strip().split("\n")
results = await asyncio.gather(*(d.exec.aio(f"wc -l logs/{f}") for f in files))
asyncio.run(main())
```
Each exec gets its own container — this scales horizontally without touching your local compute. If the agent's goal is to *search* for a pattern rather than run an arbitrary command per file, reach for [`disk.grep`](/compute/search-files) instead — it fans the same work out in a single call and returns structured matches.
### Running a local agent, too
If the agent runs on your laptop but wants the same workspace, mount the disk alongside:
```bash theme={null}
archil mount aws-us-east-1/agent-workspace /mnt/data
```
Reads and writes from the local mount are read-after-write consistent with anything the `exec`-backed bash tool does.
## Next steps
* [Serverless execution concepts](/compute/serverless-execution)
* [TypeScript SDK reference](/sdks/typescript)
* [Python SDK reference](/sdks/python)
* [Multi-agent systems guide](/guides/ai/multi-agent-systems)
# Vector search
Source: https://docs.archil.com/guides/ai/faiss
Build high-performance vector similarity search with FAISS using Archil for shared embeddings and scalable vector databases
[FAISS](https://faiss.ai/) (Facebook AI Similarity Search) is a library for efficient similarity search and clustering of dense vectors. This guide demonstrates how to use Archil disks to store and share FAISS indices across multiple servers, enabling scalable vector search for AI applications like recommendation systems, semantic search, and similarity matching.
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk that you want to use for storing your FAISS indices and embeddings.
```bash theme={null}
# Mount your Archil disk
sudo mkdir -p /mnt/archil
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region aws-us-east-1 --shared
# Create the directory for FAISS data
sudo mkdir -p /mnt/archil/faiss
sudo chown -R $USER:$USER /mnt/archil/faiss
```
The `--shared` flag enables multiple servers to access the same vector indices simultaneously.
## Install dependencies
Create a Python virtual environment and install FAISS with dependencies:
```bash theme={null}
cd /mnt/archil
mkdir -p faiss
cd faiss
python -m venv faiss-env
source faiss-env/bin/activate
# Install FAISS and related packages
pip install faiss-cpu # or faiss-gpu for GPU support
pip install numpy sentence-transformers
pip install datasets pandas scikit-learn
```
## Set up directory structure
Create directories for organizing your vector search data:
```bash theme={null}
mkdir -p /mnt/archil/faiss/{indices,embeddings,data,models}
```
## Understanding FAISS index types
FAISS offers several index types, each optimized for different use cases. Choosing the right index depends on your dataset size, memory constraints, accuracy requirements, and search speed needs.
**IndexFlatL2** and **IndexFlatIP** provide exact search results by comparing the query against every vector in the database.
**When to use**: Small to medium datasets (\< 1M vectors), when you need 100% accuracy
**Characteristics**:
* **Memory usage**: 4 bytes × dimension × number of vectors
* **Search time**: Linear with dataset size
* **Accuracy**: Perfect (exact results)
* **Training required**: No
```python theme={null}
# Best for: High accuracy requirements, smaller datasets
index = faiss.IndexFlatL2(dimension) # L2 distance
# or
index = faiss.IndexFlatIP(dimension) # Inner product (cosine similarity)
```
**Pros**: Perfect accuracy, simple to use, no training required
**Cons**: Slow for large datasets, high memory usage
**IndexIVFFlat** uses k-means clustering to partition the vector space, then searches only the most relevant partitions.
**When to use**: Medium to large datasets (1M-100M vectors), when you can trade some accuracy for speed
**Characteristics**:
* **Memory usage**: 4 bytes × dimension + 8 bytes per vector
* **Search time**: Depends on `nprobe` parameter (number of clusters to search)
* **Accuracy**: High (90-99% depending on `nprobe`)
* **Training required**: Yes (needs representative sample to learn clusters)
```python theme={null}
# Best for: Balanced speed/accuracy, most common choice for large datasets
quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFFlat(quantizer, dimension, nlist=100) # 100 clusters
index.nprobe = 10 # Search 10 clusters (higher = more accurate but slower)
```
**Rule of thumb for nlist**: Use `sqrt(n)` clusters where `n` is your dataset size. For 1M vectors, use \~1000 clusters.
**Pros**: Good speed/accuracy balance, scalable to large datasets
**Cons**: Requires training, parameter tuning needed
**IndexHNSWFlat** builds a graph structure that enables very fast approximate search with high accuracy.
**When to use**: When you need the fastest possible search with high accuracy
**Characteristics**:
* **Memory usage**: 4 bytes × dimension + additional graph overhead (\~50-100 bytes per vector)
* **Search time**: Logarithmic with dataset size
* **Accuracy**: Very high (95-99%)
* **Training required**: No
```python theme={null}
# Best for: Real-time applications requiring fast search
index = faiss.IndexHNSWFlat(dimension, M=32) # M = connections per node
index.hnsw.efConstruction = 200 # Build-time search depth
index.hnsw.efSearch = 100 # Query-time search depth
```
**HNSW Parameters**:
* `M`: Number of graph connections (16-64, higher = more accurate but more memory)
* `efConstruction`: Build-time exploration depth (100-800)
* `efSearch`: Query-time exploration depth (adjust for speed/accuracy trade-off)
**Pros**: Fastest search, high accuracy, no training required
**Cons**: Higher memory usage, longer build time
For very large datasets where memory is a constraint:
**IndexIVFPQ** (Product Quantization): Compresses vectors to \~8-32 bytes each
```python theme={null}
# Best for: Very large datasets with memory constraints
quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFPQ(quantizer, dimension, nlist=1000, m=8, nbits=8)
```
**IndexScalarQuantizer**: Reduces precision to 4-8 bits per dimension
```python theme={null}
# Best for: Moderate compression with good accuracy
index = faiss.IndexScalarQuantizer(dimension, faiss.ScalarQuantizer.QT_8bit)
```
**When to use**: Very large datasets (>10M vectors) with memory constraints
**Pros**: Significant memory savings, still reasonably fast
**Cons**: Reduced accuracy, more complex setup
### Decision matrix
| Dataset Size | Memory Priority | Speed Priority | Accuracy Priority | Recommended Index |
| - | - | - | - | - |
| \< 100K | Any | Any | Exact | `IndexFlatL2` |
| 100K - 1M | Low | High | High | `IndexHNSWFlat` |
| 100K - 1M | High | Medium | High | `IndexIVFFlat` |
| 1M - 10M | Low | High | High | `IndexHNSWFlat` |
| 1M - 10M | High | Medium | Medium | `IndexIVFPQ` |
| > 10M | High | Medium | Medium | `IndexIVFPQ` |
| > 100M | Very High | Low | Low | `IndexIVFPQ` with aggressive compression |
### Performance comparison
Here's what you can expect for a 1M vector dataset (128D):
| Index Type | Build Time | Memory Usage | Search Time (1-NN) | Accuracy |
| - | - | - | - | - |
| IndexFlatL2 | 1s | 512MB | 50ms | 100% |
| IndexIVFFlat | 30s | 520MB | 2ms | 95-99% |
| IndexHNSWFlat | 120s | 600MB | 0.5ms | 95-99% |
| IndexIVFPQ | 45s | 80MB | 3ms | 85-95% |
For deeper understanding of FAISS index types:
**Official FAISS documentation**:
* [Guidelines to choose an index](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)
* [Basic indexes documentation](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes)
**Research papers**:
* [Billion-scale similarity search with GPUs](https://arxiv.org/abs/1702.08734) - Original FAISS paper
* [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://arxiv.org/abs/1603.09320) - HNSW algorithm
**Practical guides**:
* [FAISS Performance Tuning](https://github.com/facebookresearch/faiss/wiki/How-to-make-Faiss-run-faster)
* [Memory vs Accuracy Trade-offs](https://github.com/facebookresearch/faiss/wiki/Lower-memory-footprint)
## Build a vector search system
Create a Python script for your FAISS-based vector search:
```python expandable theme={null}
# vector_search.py
import os
import numpy as np
import faiss
import pickle
from sentence_transformers import SentenceTransformer
from typing import List, Tuple, Dict
import json
# Configure paths on Archil disk
INDICES_PATH = "/mnt/archil/faiss/indices"
EMBEDDINGS_PATH = "/mnt/archil/faiss/embeddings"
DATA_PATH = "/mnt/archil/faiss/data"
MODELS_PATH = "/mnt/archil/faiss/models"
class ArchilVectorSearch:
def __init__(self, model_name="sentence-transformers/all-MiniLM-L6-v2"):
# Initialize embedding model (cached on Archil disk)
self.model = SentenceTransformer(
model_name,
cache_folder=MODELS_PATH
)
self.dimension = self.model.get_sentence_embedding_dimension()
# Initialize FAISS index
self.index = None
self.documents = []
self.metadata = []
def create_index(self, index_type="flat"):
"""Create a new FAISS index"""
if index_type == "flat":
# Exact search (L2 distance)
self.index = faiss.IndexFlatL2(self.dimension)
elif index_type == "ivf":
# Approximate search with IVF (Inverted File)
quantizer = faiss.IndexFlatL2(self.dimension)
self.index = faiss.IndexIVFFlat(quantizer, self.dimension, 100)
elif index_type == "hnsw":
# Hierarchical Navigable Small World
self.index = faiss.IndexHNSWFlat(self.dimension, 32)
else:
raise ValueError(f"Unknown index type: {index_type}")
print(f"Created {index_type} index with dimension {self.dimension}")
def add_documents(self, documents: List[str], metadata: List[Dict] = None):
"""Add documents to the vector index"""
print(f"Encoding {len(documents)} documents...")
# Generate embeddings
embeddings = self.model.encode(documents, show_progress_bar=True)
embeddings = embeddings.astype('float32')
# Add to index
if self.index.ntotal == 0 and hasattr(self.index, 'train'):
print("Training index...")
self.index.train(embeddings)
self.index.add(embeddings)
# Store documents and metadata
self.documents.extend(documents)
if metadata:
self.metadata.extend(metadata)
else:
self.metadata.extend([{"id": i} for i in range(len(documents))])
print(f"Added {len(documents)} documents. Total: {self.index.ntotal}")
def search(self, query: str, k: int = 5) -> List[Tuple[str, float, Dict]]:
"""Search for similar documents"""
if self.index is None or self.index.ntotal == 0:
return []
# Encode query
query_embedding = self.model.encode([query]).astype('float32')
# Search
distances, indices = self.index.search(query_embedding, k)
# Format results
results = []
for i, (distance, idx) in enumerate(zip(distances[0], indices[0])):
if idx < len(self.documents):
results.append((
self.documents[idx],
float(distance),
self.metadata[idx]
))
return results
def save_index(self, index_name: str):
"""Save index and metadata to Archil disk"""
index_path = os.path.join(INDICES_PATH, f"{index_name}.index")
metadata_path = os.path.join(INDICES_PATH, f"{index_name}_metadata.pkl")
documents_path = os.path.join(INDICES_PATH, f"{index_name}_documents.pkl")
# Save FAISS index
faiss.write_index(self.index, index_path)
# Save metadata and documents
with open(metadata_path, 'wb') as f:
pickle.dump(self.metadata, f)
with open(documents_path, 'wb') as f:
pickle.dump(self.documents, f)
print(f"Index saved to {index_path}")
def load_index(self, index_name: str):
"""Load index and metadata from Archil disk"""
index_path = os.path.join(INDICES_PATH, f"{index_name}.index")
metadata_path = os.path.join(INDICES_PATH, f"{index_name}_metadata.pkl")
documents_path = os.path.join(INDICES_PATH, f"{index_name}_documents.pkl")
if not os.path.exists(index_path):
raise FileNotFoundError(f"Index not found: {index_path}")
# Load FAISS index
self.index = faiss.read_index(index_path)
# Load metadata and documents
with open(metadata_path, 'rb') as f:
self.metadata = pickle.load(f)
with open(documents_path, 'rb') as f:
self.documents = pickle.load(f)
print(f"Loaded index with {self.index.ntotal} vectors")
def create_sample_dataset():
"""Create a sample dataset for demonstration"""
documents = [
"Machine learning is a subset of artificial intelligence",
"Deep learning uses neural networks with multiple layers",
"Natural language processing helps computers understand text",
"Computer vision enables machines to interpret visual information",
"Reinforcement learning trains agents through rewards and penalties",
"Supervised learning uses labeled data for training",
"Unsupervised learning finds patterns in unlabeled data",
"Transfer learning adapts pre-trained models to new tasks",
"Feature engineering improves model performance",
"Cross-validation helps evaluate model generalization"
]
metadata = [
{"category": "AI", "topic": "ML Basics", "id": i}
for i in range(len(documents))
]
return documents, metadata
if __name__ == "__main__":
# Initialize vector search system
vs = ArchilVectorSearch()
# Create sample dataset
documents, metadata = create_sample_dataset()
# Create and populate index
vs.create_index("flat")
vs.add_documents(documents, metadata)
# Save to Archil disk
vs.save_index("ml_concepts")
# Demonstrate search
queries = [
"neural networks and deep learning",
"training with labeled examples",
"computer understanding of images"
]
for query in queries:
print(f"\nQuery: {query}")
results = vs.search(query, k=3)
for i, (doc, distance, meta) in enumerate(results, 1):
print(f"{i}. Distance: {distance:.4f}")
print(f" Document: {doc}")
print(f" Metadata: {meta}")
```
## Load sample data
Create a script to load and index a larger dataset:
```python expandable theme={null}
# load_dataset.py
from datasets import load_dataset
from vector_search import ArchilVectorSearch
def load_wikipedia_data():
"""Load Wikipedia dataset for indexing"""
# Load a subset of Wikipedia articles
try:
dataset = load_dataset("wikipedia", "20220301.simple", split="train[:1000]")
except Exception as e:
print(f"Error loading dataset: {e}")
print("Try reducing the dataset size or check your internet connection")
return [], []
documents = []
metadata = []
for i, article in enumerate(dataset):
# Use article text (truncated for demo)
text = article['text'][:1000] # First 1000 characters
documents.append(text)
metadata.append({
"title": article['title'],
"id": article['id'],
"url": article['url']
})
return documents, metadata
if __name__ == "__main__":
# Initialize vector search
vs = ArchilVectorSearch()
# Load Wikipedia data
print("Loading Wikipedia dataset...")
documents, metadata = load_wikipedia_data()
# Create IVF index for better performance with large datasets
vs.create_index("ivf")
vs.add_documents(documents, metadata)
# Save to shared storage
vs.save_index("wikipedia_1k")
print("Wikipedia index created and saved to Archil disk")
```
## Multi-server search service
Create a simple search service that multiple servers can run:
```python expandable theme={null}
# search_service.py
from vector_search import ArchilVectorSearch
import json
from http.server import HTTPServer, BaseHTTPRequestHandler
import urllib.parse
class SearchHandler(BaseHTTPRequestHandler):
def __init__(self, *args, vector_search=None, **kwargs):
self.vector_search = vector_search
super().__init__(*args, **kwargs)
def do_GET(self):
if self.path.startswith('/search'):
# Parse query parameters
parsed = urllib.parse.urlparse(self.path)
params = urllib.parse.parse_qs(parsed.query)
query = params.get('q', [''])[0]
k = int(params.get('k', ['5'])[0])
if query:
results = self.vector_search.search(query, k)
response = {
"query": query,
"results": [
{
"document": doc,
"distance": dist,
"metadata": meta
}
for doc, dist, meta in results
]
}
else:
response = {"error": "No query provided"}
self.send_response(200)
self.send_header('Content-type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps(response, indent=2).encode())
else:
self.send_response(404)
self.end_headers()
def run_search_service(index_name="ml_concepts", port=8000):
# Load index from Archil disk
vs = ArchilVectorSearch()
vs.load_index(index_name)
# Create handler with vector search instance
handler = lambda *args, **kwargs: SearchHandler(*args, vector_search=vs, **kwargs)
# Start server
server = HTTPServer(('0.0.0.0', port), handler)
print(f"Search service running on port {port}")
print(f"Try: curl 'http://localhost:{port}/search?q=neural%20networks&k=3'")
server.serve_forever()
if __name__ == "__main__":
run_search_service()
```
## Performance optimization
### Index types comparison
```python expandable theme={null}
# benchmark_indices.py
import time
from vector_search import ArchilVectorSearch, create_sample_dataset
def benchmark_index_types():
documents, metadata = create_sample_dataset()
index_types = ["flat", "ivf", "hnsw"]
for index_type in index_types:
print(f"\nBenchmarking {index_type} index:")
vs = ArchilVectorSearch()
vs.create_index(index_type)
# Time index creation
start = time.time()
vs.add_documents(documents, metadata)
build_time = time.time() - start
# Time search
query = "machine learning algorithms"
start = time.time()
results = vs.search(query, k=5)
search_time = time.time() - start
print(f" Build time: {build_time:.4f}s")
print(f" Search time: {search_time:.4f}s")
print(f" Results: {len(results)}")
if __name__ == "__main__":
benchmark_index_types()
```
## Monitoring and maintenance
Monitor your FAISS indices on Archil:
```bash theme={null}
# Check index files
ls -la /mnt/archil/faiss/indices/
# Monitor disk usage
du -sh /mnt/archil/faiss/*
# Check Archil status
archil status /mnt/archil
```
## Advanced features
### Batch processing
```python theme={null}
def batch_add_documents(vs, documents, batch_size=1000):
"""Add documents in batches for memory efficiency"""
for i in range(0, len(documents), batch_size):
batch = documents[i:i+batch_size]
vs.add_documents(batch)
print(f"Processed batch {i//batch_size + 1}")
```
### Index merging
```python theme={null}
def merge_indices(index_names, output_name):
"""Merge multiple FAISS indices"""
vs = ArchilVectorSearch()
vs.create_index("flat")
for index_name in index_names:
temp_vs = ArchilVectorSearch()
temp_vs.load_index(index_name)
# Add documents from loaded index
vs.add_documents(temp_vs.documents, temp_vs.metadata)
vs.save_index(output_name)
print(f"Merged {len(index_names)} indices into {output_name}")
```
## Benefits of FAISS with Archil
* **Shared indices**: Multiple servers access the same vector indices without duplication
* **Fast loading**: Indices load quickly from Archil's high-speed cache
* **Scalable storage**: Store large vector databases that exceed local disk capacity
* **Multi-model support**: Share different embedding models across your infrastructure
* **Cost efficient**: Eliminate the need to replicate large indices across servers
## Next steps
* Explore GPU acceleration with `faiss-gpu`
* Implement real-time index updates
* Add index compression techniques
* Integrate with production search systems
* Scale to billion-vector datasets
This setup provides a robust foundation for building production vector search systems with FAISS and Archil's shared storage capabilities.
# Multi-agent systems
Source: https://docs.archil.com/guides/ai/multi-agent-systems
Build multi-agent AI systems with LangChain and LangGraph using Archil for persistent memory and stateless agent architecture
[LangChain](https://python.langchain.com/) is a framework for developing applications powered by language models, while [LangGraph](https://langchain-ai.github.io/langgraph/) extends LangChain with graph-based orchestration for complex multi-agent workflows. This guide demonstrates how to use Archil disks to provide persistent memory storage for stateless agents, enabling scalable multi-agent systems with shared state across multiple servers.
## Overview
LangGraph enables sophisticated multi-agent architectures where specialized agents (Research, Writer, Reviewer) work together as nodes in a graph. With Archil, you can:
* **Persist agent memory** across restarts and deployments
* **Share state** between multiple agent instances running on different servers
* **Store checkpoints** for complex workflows using LangChain's checkpointer libraries
* **Scale horizontally** while maintaining consistent agent memory
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk for storing your agent memory and checkpoints.
```bash theme={null}
# Mount your Archil disk with shared access
sudo mkdir -p /mnt/archil
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region aws-us-east-1 --shared
# Create directories for agent data
sudo mkdir -p /mnt/archil/langchain/{checkpoints,memory,agents,models}
sudo chown -R $USER:$USER /mnt/archil/langchain
```
The `--shared` flag enables multiple servers to access the same agent memory simultaneously, crucial for distributed multi-agent systems.
## Install dependencies
Set up your Python environment with LangChain, LangGraph, and database drivers:
```bash theme={null}
cd /mnt/archil/langchain
python -m venv langchain-env
source langchain-env/bin/activate
# Core LangChain and LangGraph packages
pip install langchain langgraph langchain-openai
# SQLite for lightweight checkpointing (built into Python)
# No additional database drivers needed - using file-based storage
# Additional utilities
pip install python-dotenv pydantic
# Set your OpenAI API key
export OPENAI_API_KEY="your-api-key-here"
```
## Multi-agent architecture setup
LangGraph uses a "divide-and-conquer" approach where specialized agents handle different aspects of complex tasks. Here's how to set up a multi-agent system with Research, Writer, and Reviewer agents:
```python expandable theme={null}
# /mnt/archil/langchain/multi_agent_system.py
import os
from typing import TypedDict, List
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, END
from datetime import datetime
# Define the shared state structure
class AgentState(TypedDict):
messages: List[str]
research_data: str
draft_content: str
final_content: str
feedback: str
iteration: int
# Initialize the language model
llm = ChatOpenAI(model="gpt-4", temperature=0.7)
def research_agent(state: AgentState) -> AgentState:
"""Research agent gathers information on the topic"""
prompt = f"""You are a research agent. Based on the user's request: {state['messages'][-1]}
Conduct thorough research and provide key findings, statistics, and relevant information.
Focus on accuracy and comprehensiveness."""
response = llm.invoke([SystemMessage(content=prompt)])
return {
**state,
"research_data": response.content,
"messages": state["messages"] + [f"Research completed: {response.content[:100]}..."]
}
def writer_agent(state: AgentState) -> AgentState:
"""Writer agent creates content based on research"""
prompt = f"""You are a writing agent. Using this research data:
{state['research_data']}
Create well-structured, engaging content that addresses the original request:
{state['messages'][0]}
Make it informative and well-organized."""
response = llm.invoke([SystemMessage(content=prompt)])
return {
**state,
"draft_content": response.content,
"messages": state["messages"] + [f"Draft completed: {len(response.content)} characters"]
}
def reviewer_agent(state: AgentState) -> AgentState:
"""Reviewer agent evaluates and provides feedback"""
prompt = f"""You are a review agent. Evaluate this content:
{state['draft_content']}
Original request: {state['messages'][0]}
Research data: {state['research_data'][:200]}...
Provide specific feedback on:
1. Accuracy and completeness
2. Structure and clarity
3. Suggestions for improvement
If the content is good enough, say "APPROVED". Otherwise, provide detailed feedback."""
response = llm.invoke([SystemMessage(content=prompt)])
# Determine if we need another iteration
is_approved = "APPROVED" in response.content.upper()
return {
**state,
"feedback": response.content,
"final_content": state["draft_content"] if is_approved else "",
"iteration": state["iteration"] + 1,
"messages": state["messages"] + [f"Review completed: {'Approved' if is_approved else 'Needs revision'}"]
}
def should_continue(state: AgentState) -> str:
"""Determine the next step in the workflow"""
if state["final_content"]:
return "end"
elif state["iteration"] >= 3: # Max 3 iterations
return "end"
else:
return "revise"
# Create the workflow graph
workflow = StateGraph(AgentState)
# Add nodes
workflow.add_node("research", research_agent)
workflow.add_node("write", writer_agent)
workflow.add_node("review", reviewer_agent)
# Define the flow
workflow.set_entry_point("research")
workflow.add_edge("research", "write")
workflow.add_edge("write", "review")
workflow.add_conditional_edges(
"review",
should_continue,
{
"revise": "write",
"end": END
}
)
# Compile the workflow
app = workflow.compile()
```
## Supervisor pattern implementation
The supervisor pattern coordinates multiple agents and manages the overall workflow. Here's how to implement it with LangGraph:
```python expandable theme={null}
# /mnt/archil/langchain/supervisor_pattern.py
from typing import Literal
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, END
# Define available agents
members = ["Researcher", "Writer", "Reviewer"]
system_prompt = (
"You are a supervisor tasked with managing a conversation between the"
" following workers: {members}. Given the following user request,"
" respond with the worker to act next. Each worker will perform a"
" task and respond with their results and status. When finished,"
" respond with FINISH."
)
options = ["FINISH"] + members
# Create supervisor prompt
prompt = ChatPromptTemplate.from_messages([
("system", system_prompt),
MessagesPlaceholder(variable_name="messages"),
("system", "Given the conversation above, who should act next? Or should we FINISH? Select one of: {options}"),
]).partial(options=str(options), members=", ".join(members))
llm = ChatOpenAI(model="gpt-4")
def supervisor_agent(state):
"""Supervisor decides which agent should act next"""
supervisor_chain = prompt | llm
response = supervisor_chain.invoke(state)
return {"next": response.content}
# Create supervisor workflow
supervisor_graph = StateGraph(AgentState)
supervisor_graph.add_node("supervisor", supervisor_agent)
supervisor_graph.add_node("research", research_agent)
supervisor_graph.add_node("write", writer_agent)
supervisor_graph.add_node("review", reviewer_agent)
# Define conditional routing
def route_to_agent(state) -> Literal["research", "write", "review", "__end__"]:
"""Route to the appropriate agent based on supervisor decision"""
next_agent = state.get("next", "").lower()
if next_agent == "finish":
return "__end__"
elif "researcher" in next_agent:
return "research"
elif "writer" in next_agent:
return "write"
elif "reviewer" in next_agent:
return "review"
else:
return "__end__"
supervisor_graph.set_entry_point("supervisor")
supervisor_graph.add_conditional_edges("supervisor", route_to_agent)
# Add edges back to supervisor after each agent
for member in ["research", "write", "review"]:
supervisor_graph.add_edge(member, "supervisor")
# Compile the supervisor workflow
supervisor_app = supervisor_graph.compile()
```
## Checkpointer integration
LangGraph supports persistent state management through checkpointers. With Archil, you can use simple file-based checkpointing or SQLite for lightweight, persistent storage:
### File-based checkpointer
Simple JSON file-based checkpointing for lightweight persistence:
```python expandable theme={null}
# /mnt/archil/langchain/file_checkpointer.py
import json
import os
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, Optional
class FileCheckpointer:
"""Simple file-based checkpointer using Archil persistent storage"""
def __init__(self, checkpoint_dir="/mnt/archil/langchain/checkpoints"):
self.checkpoint_dir = Path(checkpoint_dir)
def save_checkpoint(self, thread_id: str, state: Dict[str, Any]):
"""Save agent state to a JSON file"""
checkpoint_file = self.checkpoint_dir / f"{thread_id}.json"
checkpoint_data = {
"state": state,
"timestamp": datetime.utcnow().isoformat(),
"thread_id": thread_id
}
# Atomic write using temporary file
temp_file = checkpoint_file.with_suffix('.tmp')
with open(temp_file, 'w') as f:
json.dump(checkpoint_data, f, indent=2)
# Atomic rename
temp_file.rename(checkpoint_file)
def load_checkpoint(self, thread_id: str) -> Optional[Dict[str, Any]]:
"""Load agent state from JSON file"""
checkpoint_file = self.checkpoint_dir / f"{thread_id}.json"
if checkpoint_file.exists():
try:
with open(checkpoint_file, 'r') as f:
data = json.load(f)
return data.get("state")
except (json.JSONDecodeError, IOError):
return None
return None
def list_checkpoints(self) -> list:
"""List all available checkpoints"""
return [f.stem for f in self.checkpoint_dir.glob("*.json")]
def delete_checkpoint(self, thread_id: str):
"""Delete a checkpoint"""
checkpoint_file = self.checkpoint_dir / f"{thread_id}.json"
checkpoint_file.unlink(missing_ok=True)
# Usage with LangGraph
file_checkpointer = FileCheckpointer()
# Custom workflow compilation with file checkpointer
class FileCheckpointWorkflow:
def __init__(self, workflow, checkpointer):
self.workflow = workflow
self.checkpointer = checkpointer
def invoke(self, initial_state: dict, config: dict):
thread_id = config.get("configurable", {}).get("thread_id")
if thread_id:
# Load existing state if available
saved_state = self.checkpointer.load_checkpoint(thread_id)
if saved_state:
# Merge with initial state
initial_state.update(saved_state)
# Run the workflow
result = self.workflow.invoke(initial_state)
if thread_id:
# Save the final state
self.checkpointer.save_checkpoint(thread_id, result)
return result
# Use compiled workflow with file checkpointer
checkpoint_app = FileCheckpointWorkflow(app, file_checkpointer)
# Run with persistent state
config = {"configurable": {"thread_id": "agent_session_1"}}
result = checkpoint_app.invoke({
"messages": ["Write a comprehensive guide about renewable energy"],
"research_data": "",
"draft_content": "",
"final_content": "",
"feedback": "",
"iteration": 0
}, config)
```
### SQLite checkpointer
SQLite-based checkpointing with history tracking and structured queries:
```python expandable theme={null}
# /mnt/archil/langchain/sqlite_checkpointer.py
import sqlite3
import json
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, Optional
class SQLiteCheckpointer:
"""SQLite-based checkpointer using Archil persistent storage"""
def __init__(self, db_path="/mnt/archil/langchain/checkpoints/agent_checkpoints.db"):
self.db_path = Path(db_path)
self._init_db()
def _init_db(self):
"""Initialize the SQLite database and tables"""
with sqlite3.connect(self.db_path) as conn:
conn.execute("""
CREATE TABLE IF NOT EXISTS checkpoints (
thread_id TEXT PRIMARY KEY,
state TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""")
conn.execute("""
CREATE TABLE IF NOT EXISTS checkpoint_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
thread_id TEXT NOT NULL,
state TEXT NOT NULL,
checkpoint_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (thread_id) REFERENCES checkpoints (thread_id)
)
""")
conn.commit()
def save_checkpoint(self, thread_id: str, state: Dict[str, Any], keep_history: bool = True):
"""Save agent state to SQLite database"""
state_json = json.dumps(state)
with sqlite3.connect(self.db_path) as conn:
# Save current state
conn.execute("""
INSERT OR REPLACE INTO checkpoints (thread_id, state, updated_at)
VALUES (?, ?, ?)
""", (thread_id, state_json, datetime.utcnow().isoformat()))
# Optionally save to history
if keep_history:
conn.execute("""
INSERT INTO checkpoint_history (thread_id, state)
VALUES (?, ?)
""", (thread_id, state_json))
conn.commit()
def load_checkpoint(self, thread_id: str) -> Optional[Dict[str, Any]]:
"""Load agent state from SQLite database"""
with sqlite3.connect(self.db_path) as conn:
cursor = conn.execute(
"SELECT state FROM checkpoints WHERE thread_id = ?",
(thread_id,)
)
row = cursor.fetchone()
if row:
return json.loads(row[0])
return None
def get_checkpoint_history(self, thread_id: str, limit: int = 10) -> list:
"""Get checkpoint history for a thread"""
with sqlite3.connect(self.db_path) as conn:
cursor = conn.execute("""
SELECT state, checkpoint_time FROM checkpoint_history
WHERE thread_id = ?
ORDER BY checkpoint_time DESC
LIMIT ?
""", (thread_id, limit))
return [
{"state": json.loads(row[0]), "timestamp": row[1]}
for row in cursor.fetchall()
]
def list_threads(self) -> list:
"""List all thread IDs with checkpoints"""
with sqlite3.connect(self.db_path) as conn:
cursor = conn.execute("SELECT thread_id, updated_at FROM checkpoints ORDER BY updated_at DESC")
return [{"thread_id": row[0], "last_updated": row[1]} for row in cursor.fetchall()]
def delete_checkpoint(self, thread_id: str):
"""Delete a checkpoint and its history"""
with sqlite3.connect(self.db_path) as conn:
conn.execute("DELETE FROM checkpoint_history WHERE thread_id = ?", (thread_id,))
conn.execute("DELETE FROM checkpoints WHERE thread_id = ?", (thread_id,))
conn.commit()
# Usage with LangGraph
sqlite_checkpointer = SQLiteCheckpointer()
# Enhanced workflow with SQLite checkpointer
class SQLiteCheckpointWorkflow:
def __init__(self, workflow, checkpointer):
self.workflow = workflow
self.checkpointer = checkpointer
def invoke(self, initial_state: dict, config: dict):
thread_id = config.get("configurable", {}).get("thread_id")
if thread_id:
# Load existing state if available
saved_state = self.checkpointer.load_checkpoint(thread_id)
if saved_state:
initial_state.update(saved_state)
# Run the workflow
result = self.workflow.invoke(initial_state)
if thread_id:
# Save checkpoint with history
self.checkpointer.save_checkpoint(thread_id, result, keep_history=True)
return result
def get_thread_history(self, thread_id: str):
"""Get the execution history for a thread"""
return self.checkpointer.get_checkpoint_history(thread_id)
# Use compiled workflow with SQLite checkpointer
checkpoint_app = SQLiteCheckpointWorkflow(app, sqlite_checkpointer)
# Run with persistent state and history tracking
config = {"configurable": {"thread_id": "renewable_energy_research_1"}}
result = checkpoint_app.invoke({
"messages": ["Write a comprehensive guide about renewable energy"],
"research_data": "",
"draft_content": "",
"final_content": "",
"feedback": "",
"iteration": 0
}, config)
# View execution history
history = app.get_thread_history("renewable_energy_research_1")
print(f"Found {len(history)} checkpoints in history")
```
## Stateless agent memory persistence
Create a comprehensive memory management system for stateless agents:
```python expandable theme={null}
# /mnt/archil/langchain/agent_memory.py
import json
import os
from datetime import datetime
from typing import Dict, List, Any
from pathlib import Path
class ArchiledAgentMemory:
"""Persistent memory system for stateless agents using Archil storage"""
def __init__(self, base_path="/mnt/archil/langchain/memory"):
self.base_path = Path(base_path)
def save_agent_memory(self, agent_id: str, memory_data: Dict[str, Any]):
"""Save agent memory to persistent storage"""
agent_dir = self.base_path / agent_id
memory_file = agent_dir / "memory.json"
# Load existing memory and merge
existing_memory = self.load_agent_memory(agent_id)
existing_memory.update(memory_data)
existing_memory["last_updated"] = datetime.utcnow().isoformat()
with open(memory_file, 'w') as f:
json.dump(existing_memory, f, indent=2)
def load_agent_memory(self, agent_id: str) -> Dict[str, Any]:
"""Load agent memory from persistent storage"""
memory_file = self.base_path / agent_id / "memory.json"
if memory_file.exists():
with open(memory_file, 'r') as f:
return json.load(f)
return {}
def save_conversation_history(self, session_id: str, messages: List[Dict]):
"""Save conversation history"""
session_dir = self.base_path / "conversations" / session_id
history_file = session_dir / "history.json"
with open(history_file, 'w') as f:
json.dump({
"messages": messages,
"created_at": datetime.utcnow().isoformat()
}, f, indent=2)
def load_conversation_history(self, session_id: str) -> List[Dict]:
"""Load conversation history"""
history_file = self.base_path / "conversations" / session_id / "history.json"
if history_file.exists():
with open(history_file, 'r') as f:
data = json.load(f)
return data.get("messages", [])
return []
# Usage example
memory_manager = ArchiledAgentMemory()
# Save agent-specific memory
memory_manager.save_agent_memory("research_agent_1", {
"expertise_areas": ["renewable_energy", "climate_science"],
"recent_queries": ["solar panel efficiency", "wind turbine technology"],
"learned_facts": {
"solar_efficiency_2024": "Modern panels achieve 22-26% efficiency",
"wind_capacity_growth": "Global capacity grew 77% in 2023"
}
})
# Load memory for stateless agent
agent_memory = memory_manager.load_agent_memory("research_agent_1")
```
## Complete multi-agent example
Here's a complete example that ties everything together:
```python expandable theme={null}
# /mnt/archil/langchain/complete_example.py
import os
from dotenv import load_dotenv
# Load environment variables
load_dotenv()
# Initialize components
memory_manager = ArchiledAgentMemory()
sqlite_checkpointer = SQLiteCheckpointer()
# Compile the complete workflow
complete_app = SQLiteCheckpointWorkflow(supervisor_app, sqlite_checkpointer)
def run_multi_agent_task(user_request: str, session_id: str = None):
"""Run a complete multi-agent task with persistent memory"""
if not session_id:
session_id = f"session_{int(datetime.utcnow().timestamp())}"
# Load conversation history
conversation_history = memory_manager.load_conversation_history(session_id)
# Initial state
initial_state = {
"messages": [user_request],
"research_data": "",
"draft_content": "",
"final_content": "",
"feedback": "",
"iteration": 0,
"next": ""
}
# Configuration for persistent checkpoints
config = {"configurable": {"thread_id": session_id}}
try:
# Run the workflow
result = complete_app.invoke(initial_state, config)
# Save conversation history
conversation_history.append({
"user_request": user_request,
"result": result,
"timestamp": datetime.utcnow().isoformat()
})
memory_manager.save_conversation_history(session_id, conversation_history)
# Update agent memories based on the interaction
memory_manager.save_agent_memory("research_agent", {
"last_research_topic": user_request,
"research_quality_score": len(result.get("research_data", "")) / 100
})
return result
except Exception as e:
print(f"Error in multi-agent task: {e}")
return {"error": str(e)}
# Example usage
if __name__ == "__main__":
result = run_multi_agent_task(
"Create a comprehensive analysis of renewable energy trends in 2024",
session_id="renewable_energy_analysis_1"
)
print("Final Content:")
print(result.get("final_content", "No content generated"))
```
## Advanced features and best practices
### Scaling across multiple servers
When running agents across multiple servers, use Archil's shared disk capabilities:
```python expandable theme={null}
# /mnt/archil/langchain/distributed_agents.py
import fcntl
import time
from pathlib import Path
class DistributedAgentCoordinator:
"""Coordinate agents across multiple servers using file-based locking"""
def __init__(self, coordination_path="/mnt/archil/langchain/coordination"):
self.coordination_path = Path(coordination_path)
def acquire_agent_lock(self, agent_type: str, timeout: int = 30):
"""Acquire exclusive lock for an agent type"""
lock_file = self.coordination_path / f"{agent_type}.lock"
start_time = time.time()
while time.time() - start_time < timeout:
try:
fd = os.open(lock_file, os.O_CREAT | os.O_EXCL | os.O_RDWR)
return fd
except FileExistsError:
time.sleep(0.1)
raise TimeoutError(f"Could not acquire lock for {agent_type}")
def release_agent_lock(self, fd: int, agent_type: str):
"""Release agent lock"""
os.close(fd)
lock_file = self.coordination_path / f"{agent_type}.lock"
lock_file.unlink(missing_ok=True)
# Usage in distributed setup
coordinator = DistributedAgentCoordinator()
def distributed_research_agent(state: AgentState) -> AgentState:
"""Research agent with distributed coordination"""
lock_fd = coordinator.acquire_agent_lock("research")
try:
# Load any shared research cache
cache_file = Path("/mnt/archil/langchain/cache/research_cache.json")
if cache_file.exists():
with open(cache_file, 'r') as f:
research_cache = json.load(f)
else:
research_cache = {}
# Perform research (implementation from earlier)
result = research_agent(state)
# Update shared cache
research_cache[state['messages'][-1][:50]] = result['research_data']
with open(cache_file, 'w') as f:
json.dump(research_cache, f)
return result
finally:
coordinator.release_agent_lock(lock_fd, "research")
```
### Monitoring and observability
Track agent performance and system health:
```python expandable theme={null}
# /mnt/archil/langchain/monitoring.py
import json
import time
from datetime import datetime
from pathlib import Path
class AgentMonitor:
"""Monitor agent performance and system health"""
def __init__(self, metrics_path="/mnt/archil/langchain/metrics"):
self.metrics_path = Path(metrics_path)
def log_agent_execution(self, agent_name: str, execution_time: float,
success: bool, input_size: int, output_size: int):
"""Log agent execution metrics"""
metrics = {
"timestamp": datetime.utcnow().isoformat(),
"agent_name": agent_name,
"execution_time": execution_time,
"success": success,
"input_size": input_size,
"output_size": output_size
}
# Append to daily metrics file
date_str = datetime.utcnow().strftime("%Y-%m-%d")
metrics_file = self.metrics_path / f"agent_metrics_{date_str}.jsonl"
with open(metrics_file, 'a') as f:
f.write(json.dumps(metrics) + '\n')
def get_agent_performance_summary(self, days: int = 7) -> dict:
"""Get performance summary for the last N days"""
summary = {
"total_executions": 0,
"success_rate": 0.0,
"average_execution_time": 0.0,
"agent_breakdown": {}
}
# Process metrics files for the specified period
# Implementation would read and aggregate metrics
return summary
# Decorator for monitoring agent functions
def monitor_agent(agent_name: str):
def decorator(func):
def wrapper(state):
monitor = AgentMonitor()
start_time = time.time()
try:
input_size = len(str(state))
result = func(state)
output_size = len(str(result))
execution_time = time.time() - start_time
monitor.log_agent_execution(
agent_name, execution_time, True, input_size, output_size
)
return result
except Exception as e:
execution_time = time.time() - start_time
monitor.log_agent_execution(
agent_name, execution_time, False, 0, 0
)
raise
return wrapper
return decorator
# Apply monitoring to agents
@monitor_agent("research")
def monitored_research_agent(state: AgentState) -> AgentState:
return research_agent(state)
```
## Configuration management
Store configuration in Archil for consistent settings across deployments:
```python expandable theme={null}
# /mnt/archil/langchain/config.py
import json
from pathlib import Path
from typing import Dict, Any
class ArchiledConfig:
"""Configuration management using Archil persistent storage"""
def __init__(self, config_path="/mnt/archil/langchain/config"):
self.config_path = Path(config_path)
self.config_file = self.config_path / "agent_config.json"
# Initialize with defaults if config doesn't exist
if not self.config_file.exists():
self._create_default_config()
def _create_default_config(self):
"""Create default configuration"""
default_config = {
"agents": {
"research": {
"model": "gpt-4",
"temperature": 0.3,
"max_tokens": 2000,
"timeout": 30
},
"writer": {
"model": "gpt-4",
"temperature": 0.7,
"max_tokens": 3000,
"timeout": 45
},
"reviewer": {
"model": "gpt-4",
"temperature": 0.2,
"max_tokens": 1500,
"timeout": 30
}
},
"workflow": {
"max_iterations": 3,
"checkpoint_interval": 1,
"enable_monitoring": True
},
"storage": {
"checkpointer_type": "postgres",
"memory_retention_days": 30,
"cache_size_mb": 100
}
}
with open(self.config_file, 'w') as f:
json.dump(default_config, f, indent=2)
def get_config(self, section: str = None) -> Dict[str, Any]:
"""Get configuration section or entire config"""
with open(self.config_file, 'r') as f:
config = json.load(f)
return config.get(section) if section else config
def update_config(self, section: str, updates: Dict[str, Any]):
"""Update configuration section"""
with open(self.config_file, 'r') as f:
config = json.load(f)
if section not in config:
config[section] = {}
config[section].update(updates)
with open(self.config_file, 'w') as f:
json.dump(config, f, indent=2)
# Usage
config_manager = ArchiledConfig()
agent_config = config_manager.get_config("agents")
workflow_config = config_manager.get_config("workflow")
```
## Testing your setup
Validate your LangChain/LangGraph integration with Archil:
```bash theme={null}
# Test the complete setup
cd /mnt/archil/langchain
source langchain-env/bin/activate
# Run a simple test
python -c "
from complete_example import run_multi_agent_task
result = run_multi_agent_task('Write a brief summary of solar energy benefits')
print('Success!' if result.get('final_content') else 'Failed!')
"
# Check persistent storage
ls -la /mnt/archil/langchain/
ls -la /mnt/archil/langchain/memory/
ls -la /mnt/archil/langchain/checkpoints/
```
## Best practices
1. **Memory Management**: Regularly clean up old conversation histories and agent memories to prevent storage bloat
2. **Error Handling**: Implement robust error handling and recovery mechanisms for distributed agent failures
3. **Security**: Store API keys and sensitive configuration in environment variables, not in Archil storage
4. **Performance**: Use appropriate checkpointer backends based on your performance requirements (file-based for simplicity, SQLite for structured queries and history)
5. **Monitoring**: Implement comprehensive logging and monitoring to track agent performance and system health
6. **Scaling**: Use Archil's shared disk capabilities to enable horizontal scaling of your agent systems
With this setup, you have a robust, scalable multi-agent system that leverages Archil's persistent storage capabilities to maintain state across deployments and servers, enabling truly stateless agent architectures with persistent memory.
# AWS Lambda
Source: https://docs.archil.com/guides/compute/aws-lambda
Use Archil disks from AWS Lambda functions
AWS Lambda functions are ephemeral -- they spin up, run your code, and disappear. Any data written to
`/tmp` is lost between invocations. Archil gives your Lambda functions persistent, shared storage backed
by S3 without managing EFS or writing directly to the S3 API.
Lambda doesn't support FUSE mounts, so the natural fit is the [`disk` SDK](/sdks/typescript) — it's pure
JavaScript, has no native dependencies, and gives you [`disk.grep`](/compute/search-files) for parallel
search and [`disk.exec`](/compute/serverless-execution) for arbitrary commands, both run directly against
the filesystem without pulling data through the Lambda.
## Install the SDK
Add `disk` to your Lambda project:
```bash theme={null}
npm install disk
```
## Lambda handler
This handler runs a command against an Archil disk and returns the result — no data is ever copied
through the Lambda:
```typescript theme={null}
import * as archil from "disk";
archil.configure({
apiKey: process.env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
export const handler = async (event: { diskId: string; query: string }) => {
const d = await archil.getDisk(event.diskId);
const { matches, stoppedReason } = await d.grep({
directory: "logs",
pattern: event.query,
recursive: true,
});
return {
statusCode: matches.length > 0 ? 200 : 404,
body: JSON.stringify({ matches, stoppedReason }),
};
};
```
The `disk.grep` call fans the search out across Archil-managed containers with the filesystem mounted;
only the matching lines come back to the Lambda. This is dramatically cheaper and faster than streaming
the underlying log files through the function, and it scales across many containers as `logs` grows
instead of being capped by a single one. For arbitrary commands beyond search, use
[`disk.exec`](/compute/serverless-execution) instead.
## IAM and credentials
`disk` calls the Archil control plane over HTTPS, which means it needs an API key. Set
`ARCHIL_API_KEY` (and optionally `ARCHIL_REGION`) on the Lambda function as an environment variable
or, better, pull it from AWS Secrets Manager during cold start.
The serverless `exec` containers themselves don't need any AWS credentials configured on the Lambda
side — they run inside Archil's infrastructure with the disk's own data-source credentials.
## Cold start considerations
`disk` is a thin HTTP client, so cold starts are dominated by the Lambda runtime itself rather than
SDK initialization. To minimize end-to-end latency:
* **Hoist `archil.configure` and `archil.getDisk` out of the handler** so warm invocations skip both:
```typescript theme={null}
import * as archil from "disk";
archil.configure({
apiKey: process.env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
const diskPromise = archil.getDisk(process.env.DISK_ID!);
export const handler = async (event: { query: string }) => {
const d = await diskPromise;
const { matches } = await d.grep({ directory: "logs", pattern: event.query, recursive: true });
return { statusCode: 200, body: JSON.stringify(matches) };
};
```
* **Select a Lambda region close to the disk's Archil region** to minimize round-trip latency on
control-plane calls.
* **Increase Lambda memory** if cold starts are slow — Lambda allocates CPU proportionally to memory,
and TLS setup is CPU-bound.
## Reading and writing files inside Lambda
If the Lambda needs to manipulate file contents directly (rather than dispatching work to the
filesystem), the simplest approach is still to do it through `disk.exec`:
```typescript theme={null}
const d = await archil.getDisk(diskId);
await d.exec(`echo ${JSON.stringify(JSON.stringify(event))} > /events/${Date.now()}.json`);
const { stdout } = await d.exec(`cat /config.json`);
```
For the rare case where you need a low-level filesystem client running *inside* the Lambda — for
example, to stream a large file's bytes through the Lambda — see the
[`@archildata/native` package](/sdks/typescript#low-level-protocol-access). It only ships binaries
for Linux x64 and arm64 (both Lambda architectures are supported).
# Cloudflare Workers
Source: https://docs.archil.com/guides/compute/cloudflare-workers
Access Archil disks from Cloudflare Workers
Cloudflare Workers run at the edge with no persistent filesystem. Archil gives your Workers
read/write access to S3-backed storage with the [`disk` SDK](/sdks/typescript) — a pure-JavaScript
control-plane client with no native dependencies, which works inside the Workers runtime.
The pattern: a Worker dispatches work to [`disk.exec`](/compute/serverless-execution), which runs
the command in a container with the filesystem mounted. The Worker stays small and stateless; the
heavy lifting happens server-side, against the disk.
## Install the SDK
```bash theme={null}
npm install disk
```
## Worker example
This Worker accepts a query parameter, runs a parallel search against an Archil disk via `disk.grep`,
and returns the matching lines:
```typescript theme={null}
import * as archil from "disk";
interface Env {
ARCHIL_API_KEY: string;
DISK_ID: string;
}
export default {
async fetch(request: Request, env: Env): Promise {
archil.configure({
apiKey: env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
const url = new URL(request.url);
const query = url.searchParams.get("q") ?? "";
try {
const d = await archil.getDisk(env.DISK_ID);
if (request.method === "PUT") {
const path = url.pathname.replace(/[^a-zA-Z0-9/_.-]/g, "");
const body = await request.text();
// disk.exec doesn't provide stdin, so base64-encode the body into the
// command itself. For larger payloads, write via a mounted client instead.
const b64 = Buffer.from(body).toString("base64");
await d.exec(
`mkdir -p $(dirname ${path}) && echo ${b64} | base64 -d > ${path}`,
);
return new Response("OK", { status: 200 });
}
// GET: search the disk in parallel
const { matches } = await d.grep({
directory: "logs",
pattern: query,
recursive: true,
});
return Response.json(matches, { status: matches.length > 0 ? 200 : 404 });
} catch (err: any) {
return new Response(err.message, { status: 500 });
}
},
};
```
## Configuration
Set your Archil API key as a Worker secret:
```bash theme={null}
npx wrangler secret put ARCHIL_API_KEY
```
In your `wrangler.toml`, enable the Node.js compatibility flag (the SDK depends on standard Node
APIs but no native modules):
```toml theme={null}
compatibility_flags = ["nodejs_compat"]
```
## Archil with R2
If you're already using Cloudflare R2 for object storage, you can use R2 as Archil's
[data source](/concepts/data-sources). Archil keeps a managed cache in front of R2, so reads from
your Workers go to fast cached storage instead of round-tripping to R2 on every request — and you
get filesystem semantics on top of the bucket.
Create a disk backed by your R2 bucket and your Workers can `exec` against it.
## Performance tips
* **Pick the right Archil region.** `disk.exec` runs in the Archil region, not at the edge — select
a region close to where the bulk of your traffic originates, or where your data already lives.
* **Push work into `exec` or `grep`, not into the Worker.** Workers have CPU-time limits; the
serverless runtime doesn't. If a request requires scanning many files, run it through
[`disk.grep`](/compute/search-files) (for search) or `disk.exec` (for arbitrary commands) to keep
the Worker fast and avoid streaming gigabytes through the edge.
* **Reuse `getDisk()` across requests** in long-lived isolates by caching the returned `Disk` at
module scope.
# Jupyter Notebooks
Source: https://docs.archil.com/guides/data-science/jupyter-serverless
Learn how to run Jupyter notebooks with PyTorch and MNIST training directly on Amazon S3 using Archil
[Jupyter Notebook](https://jupyter.org/) is an open-source web application that allows you to create and share documents containing live code, equations, visualizations, and narrative text. Archil makes it simple to run Jupyter notebooks in a completely serverless manner, storing all data and notebooks directly on Amazon S3 while maintaining interactive performance through intelligent caching.
All notebooks, datasets, and trained models automatically stored in S3, persisting across compute sessions
Pay only for high-speed storage when active. Data remains in S3 at standard costs when idle
Scale compute resources up or down without data migration or storage limits
Multiple team members access same datasets and notebooks by mounting the same Archil disk
This guide will walk you through setting up a serverless Jupyter environment for data science workflows, including training a PyTorch model on the MNIST dataset with all data stored in S3.
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk that will serve as your data science workspace.
## Mount your Archil disk
Mount your Archil disk to create your data science workspace:
```bash theme={null}
# Create mount directory
sudo mkdir -p /mnt/archil
# Mount Archil disk
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region aws-us-east-1
# Create datascience workspace directories
sudo mkdir -p /mnt/archil/datascience/notebooks
sudo mkdir -p /mnt/archil/datascience/datasets
sudo mkdir -p /mnt/archil/datascience/models
sudo mkdir -p /mnt/archil/datascience/venv
sudo chown -R $USER:$USER /mnt/archil/datascience
```
## Set up Python virtual environment on Archil disk
Create a virtual environment directly on your Archil disk to ensure all dependencies persist in S3:
```bash theme={null}
# Create virtual environment on Archil disk
cd /mnt/archil/datascience
python3 -m venv venv
# Activate the virtual environment
source venv/bin/activate
# Upgrade pip and install data science packages
pip install --upgrade pip
pip install jupyter torch torchvision matplotlib numpy pandas
# Verify installation
python -c "import torch; print(f'PyTorch version: {torch.__version__}')"
```
## Configure Jupyter to use the virtual environment
Set up Jupyter to use your virtual environment and Archil workspace:
```bash theme={null}
# Generate Jupyter config (while venv is activated)
jupyter notebook --generate-config
# Create Jupyter configuration
cat > ~/.jupyter/jupyter_notebook_config.py << 'EOF'
c.NotebookApp.notebook_dir = '/mnt/archil/datascience/notebooks'
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.port = 8888
c.NotebookApp.open_browser = False
c.NotebookApp.allow_root = True
EOF
```
## Download the MNIST training notebook
Instead of creating the notebook from scratch, download our pre-built tutorial notebook:
```bash theme={null}
# Navigate to notebooks directory
cd /mnt/archil/datascience/notebooks
# Download the MNIST PyTorch tutorial notebook
curl -O https://s3.amazonaws.com/archil-client/docs/artifacts/guides/data-science/mnist-pytorch-tutorial.ipynb
# Verify the download
ls -la mnist-pytorch-tutorial.ipynb
```
## Start Jupyter and run the tutorial
Start Jupyter notebook server with your virtual environment:
```bash theme={null}
# Make sure you're in the Archil disk and virtual environment is activated
cd /mnt/archil/datascience
source venv/bin/activate
# Start Jupyter notebook
jupyter notebook --notebook-dir=/mnt/archil/datascience/notebooks
```
Open your browser and navigate to the Jupyter interface. You'll see the `mnist-pytorch-tutorial.ipynb` notebook ready to run.
The notebook includes:
* **Data loading**: Downloads MNIST dataset directly to your Archil disk
* **Model definition**: Simple neural network for digit classification
* **Training loop**: Complete training pipeline with progress tracking
* **Evaluation**: Model accuracy assessment on test data
* **Persistence**: Automatic model saving to S3 via Archil
* **Visualization**: Training progress and prediction examples
## Advanced workflows
### Loading pre-trained models
```python theme={null}
# Load a previously saved model
checkpoint = torch.load('../models/mnist_model.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
print(f'Loaded model with accuracy: {checkpoint["accuracy"]:.2f}%')
```
### Working with larger datasets
```python expandable theme={null}
# For larger datasets, you can stream data directly from S3
# The Archil cache will intelligently manage frequently accessed data
# Example: Custom dataset class for large image datasets
class LargeImageDataset(torch.utils.data.Dataset):
def __init__(self, data_dir, transform=None):
self.data_dir = data_dir # Points to Archil-mounted directory
self.transform = transform
self.image_files = os.listdir(data_dir)
def __len__(self):
return len(self.image_files)
def __getitem__(self, idx):
# Files are loaded on-demand from S3 via Archil cache
img_path = os.path.join(self.data_dir, self.image_files[idx])
# ... load and process image
```
## Monitoring and optimization
You can monitor your data science workspace and optimize performance by checking disk usage and managing your virtual environment dependencies as needed.
## Cleanup
When you're done with your session, you can safely stop Jupyter:
```bash theme={null}
# Stop Jupyter
# Ctrl+C in the terminal where Jupyter is running
```
Your data remains safely stored in S3 and can be accessed again by mounting the same disk in future sessions.
## Next steps
This tutorial demonstrated the basics of serverless Jupyter notebooks with PyTorch. You can extend this setup for more complex workflows:
* **Distributed training** across multiple compute instances
* **Hyperparameter tuning** with automated experiment tracking
* **Model serving** by deploying trained models from S3
* **Data pipelines** that process large datasets stored in object storage
All while maintaining the serverless benefits of Archil's intelligent caching and S3 integration.
# MySQL
Source: https://docs.archil.com/guides/databases/mysql
Run MySQL on an Archil disk for elastic, S3-backed storage
[MySQL](https://www.mysql.com/) is an open-source relational database management system (RDBMS) that's
popular for its performance, ease of use, and extensive ecosystem.
This guide walks you through running MySQL on an Archil disk. MySQL doesn't need to know
anything about Archil or S3 -- it just reads and writes files to its data directory as usual. By pointing
that data directory at an Archil mount, you get:
* **Elastic storage**: no need to provision or resize disks as your database grows
* **S3 durability**: all data is automatically synchronized to S3 in the background
* **Pay for what you use**: you're only billed for active data, not provisioned capacity
* **Portability**: unmount from one server and mount on another to move your database
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk.
## Install MySQL
The exact instructions for installing MySQL will vary depending on your operating system. For example, on Amazon Linux 2023,
you can install MySQL with the following commands:
```bash theme={null}
# Download the MySQL community Yum repository
sudo wget https://dev.mysql.com/get/mysql80-community-release-el9-1.noarch.rpm
sudo dnf install -y mysql80-community-release-el9-1.noarch.rpm
# Import the MySQL GPG key
sudo rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2023
# Install MySQL
sudo dnf install -y mysql-community-server mysql-community-client
```
Mount your Archil disk to an easy to access directory.
```bash theme={null}
# Mount Archil disk
sudo mkdir -p /mnt/archil
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region aws-us-east-1
# Create the directory for MySQL data
sudo mkdir -p /mnt/archil/mysql
sudo chown -R mysql:mysql /mnt/archil/mysql
```
Set the MySQL data directory to the Archil disk. Open the file `/etc/my.cnf` in your favorite editor.
Under the `[mysqld]` section, identify the line that starts with `datadir=` and edit the path to look
like the following:
```bash theme={null}
datadir=/mnt/archil/mysql
```
Initialize the MySQL database in your data directory, and start the MySQL service.
```bash theme={null}
sudo mysqld --initialize --user=mysql --datadir=/mnt/archil/mysql
sudo systemctl start mysqld
```
Note the temporary password that was generated for the root user by looking through the initialization log.
```bash theme={null}
sudo grep 'temporary password' /var/log/mysqld.log
```
Change your root password to something more secure.
```bash theme={null}
mysql --connect-expired-password -u root -p -e "ALTER USER 'root'@'localhost' IDENTIFIED BY 'YOUR_NEW_PASSWORD';"
```
By default, the MySQL password policy requires that you set a password of at least 8 characters that contains at least
one uppercase letter, one lowercase letter, one number, and one special character.
Validate that MySQL is running on Archil by executing the following command, logging in with your new password.
```bash theme={null}
mysql -u root -p -e "SELECT @@datadir;"
```
This should output the data directory you previously set up, `/mnt/archil/mysql`.
## (Optional) Run a benchmark to validate performance
First, download and compile the `sysbench` tool.
```bash theme={null}
# Install development dependencies for sysbench
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y mysql-devel openssl-devel glibc-static libstdc++-static
# Download and compile sysbench
git clone https://github.com/akopytov/sysbench.git
cd sysbench
./autogen.sh
./configure
make -j
sudo make install
```
Create a database and write test data to it.
```bash theme={null}
mysql -u root -p -e "CREATE DATABASE sbtest;"
sysbench /usr/local/share/sysbench/oltp_read_write.lua \
--mysql-host=127.0.0.1 \
--mysql-user=root \
--mysql-password='YourPassword' \
--mysql-db=sbtest \
--tables=4 --table-size=100000 prepare
```
Finally, run the test:
```bash theme={null}
# Run the sysbench test
sysbench /usr/local/share/sysbench/oltp_read_write.lua \
--mysql-host=127.0.0.1 \
--mysql-user=root \
--mysql-password='YourPassword' \
--mysql-db=sbtest \
--threads=4 --time=30 run
# Cleanup the sysbench test
sysbench /usr/local/share/sysbench/oltp_read_write.lua \
--mysql-host=127.0.0.1 \
--mysql-user=root \
--mysql-password='YourPassword' \
--mysql-db=sbtest cleanup
```
# PostgreSQL
Source: https://docs.archil.com/guides/databases/postgres
Run PostgreSQL on an Archil disk for elastic, S3-backed storage
[PostgreSQL](https://www.postgresql.org/) is an open-source relational database management system (RDBMS) that's
popular for its reliability, feature set, and community support.
This guide walks you through running PostgreSQL on an Archil disk. PostgreSQL doesn't need to know
anything about Archil or S3 -- it just reads and writes files to its data directory as usual. By pointing
that data directory at an Archil mount, you get:
* **Elastic storage**: no need to provision or resize disks as your database grows
* **S3 durability**: all data is automatically synchronized to S3 in the background
* **Pay for what you use**: you're only billed for active data, not provisioned capacity
* **Portability**: unmount from one server and mount on another to move your database
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk.
## Install PostgreSQL
The exact instructions for installing PostgreSQL will vary depending on your operating system. For example, on Amazon Linux 2023,
you can install PostgreSQL with the following command:
```bash theme={null}
sudo dnf update -y
sudo dnf install -y postgresql15 postgresql15-server
```
Next, mount your Archil disk to an easy to access directory.
```bash theme={null}
# Mount Archil disk
sudo mkdir -p /mnt/archil
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region aws-us-east-1
# Create the directory for PostgreSQL data
sudo mkdir -p /mnt/archil/pgdata
sudo chown -R postgres:postgres /mnt/archil/pgdata
```
Then, edit the PostgreSQL service files to point to the Archil disk. On Amazon Linux 2023, use the `systemctl` command
```bash theme={null}
sudo systemctl edit postgresql
```
Add the following lines to the file:
```bash theme={null}
[Service]
Environment=PGDATA=/mnt/archil/pgdata
```
Then, initialize the PostgreSQL database, and start the service.
```bash theme={null}
sudo postgresql-setup --initdb
sudo systemctl start postgresql
```
You can validate that postgres is successfully running on Archil by executing the following command:
```bash theme={null}
sudo -u postgres psql -c 'SHOW data_directory;'
```
This should output the PGDATA directory you previously set up, `/mnt/archil/pgdata`.
## Connect and use your database
Now that PostgreSQL is running on Archil, you can connect to it and use it normally with the `psql` command or from
an application. PostgreSQL is unaware of Archil -- it reads and writes files to its data directory like it normally
would. Archil handles caching those files on fast SSD storage and asynchronously synchronizing them to S3.
When your database is idle, you're not charged for Archil's high-speed cache -- you only pay for the data stored in S3.
## (Optional) Run pgbench benchmarks to validate performance
First, install the `pgbench` tool.
```bash theme={null}
sudo dnf install -y postgresql15-contrib
```
Then, initialize the benchmark database:
```bash theme={null}
sudo -u postgres createdb benchdb
sudo -u postgres pgbench -i -s 10 benchdb
```
Finally, you can run a basic benchmark to validate performance.
```bash theme={null}
sudo -u postgres pgbench -c 10 -j 2 -T 30 benchdb
```
This will run a benchmark for 30 seconds with 10 connections and 2 threads.
# SQLite
Source: https://docs.archil.com/guides/databases/sqlite
Run SQLite on an Archil disk for elastic, S3-backed storage
[SQLite](https://www.sqlite.org/index.html) is an open-source in-process SQL database engine that makes it
easy to build portable, self-contained databases. Because SQLite stores each database in a single file, it
works naturally with Archil -- just create your `.db` file on an Archil mount.
SQLite doesn't need to know anything about Archil or S3. It reads and writes files as usual, and Archil
handles caching on fast SSD storage and synchronizing to S3 in the background. This is especially useful for
organizations that create a SQLite database per user -- you only pay for the databases that are actively being
accessed.
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk.
## Install SQLite
On Amazon Linux 2023, you can install SQLite with the following command:
```bash theme={null}
sudo dnf install sqlite -y
```
Next, mount your Archil disk to an easy to access directory.
```bash theme={null}
sudo mkdir -p /mnt/archil
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region aws-us-east-1
```
Now, you can create a SQLite database on Archil. Simply cd to the Archil disk and create a new database file.
```bash theme={null}
cd /mnt/archil
sqlite3 my_database.db
```
You can use the SQLite database as normal.
## (Optional) Run a benchmark to validate performance
You can benchmark SQLite performance using the `speedtest1` tool from the SQLite authors.
Install and compile the tool.
```bash theme={null}
git clone https://github.com/sqlite/sqlite.git
cd sqlite
./configure --all
make clean speedtest1
```
Then, run a benchmark.
```bash theme={null}
sudo ./speedtest1 /mnt/archil/performance_test.db
```
# Logging
Source: https://docs.archil.com/guides/observability/logging
Using Archil to send application logs directly to S3 without code changes
Configure your applications with robust logging using Archil disks that automatically flow to Amazon S3 storage. Simply mount an Archil disk and point your regular application loggers to it - no S3 APIs required.
Write to normal filesystem paths and logs automatically sync to S3
Each instance creates its own directory for proper ownership separation
Never run out of capacity because your logs are going to S3
Use your existing logging setup - Python logging, winston, etc.
## Create and mount an Archil disk
First, create an Archil disk for your logging infrastructure. Follow the [Quickstart Guide](/getting-started/quickstart) to create a disk, then mount it in shared mode to allow multiple instances to write logs simultaneously.
### Mount the logging disk
```bash theme={null}
# Create the mount directory
sudo mkdir -p /mnt/logs
# Mount the Archil disk in shared mode
sudo archil mount /mnt/logs --region aws-us-east-1 --shared
```
### Create instance-specific directories
Each application instance should create its own directory within the mounted filesystem. We recommend doing this as part of a startup script or on application startup:
```bash theme={null}
# Create instance-specific log directory using hostname
INSTANCE_DIR="/mnt/logs/$(hostname)"
mkdir -p "$INSTANCE_DIR"
# Or use a custom instance identifier
INSTANCE_ID="${INSTANCE_ID:-$(date +%s)-$$}"
INSTANCE_DIR="/mnt/logs/instance-${INSTANCE_ID}"
mkdir -p "$INSTANCE_DIR"
```
## Application configuration
Configure your application loggers to write to the mounted Archil filesystem. Your logs will automatically flow to S3 without any S3 API calls.
```python theme={null}
import logging
import os
import socket
import threading
import time
from datetime import datetime
class MinuteRotatingFileHandler(logging.FileHandler):
def __init__(self, log_dir, service_name):
self.log_dir = log_dir
self.service_name = service_name
self.current_minute = None
super().__init__(self._get_current_filename())
def _get_current_filename(self):
now = datetime.utcnow()
timestamp = now.strftime('%Y%m%dT%H_%M_00Z')
return f"{self.log_dir}/{self.service_name}-{timestamp}.log"
def emit(self, record):
now = datetime.utcnow()
current_minute = now.strftime('%Y%m%dT%H_%M')
if self.current_minute != current_minute:
self.close()
self.baseFilename = self._get_current_filename()
self.stream = self._open()
self.current_minute = current_minute
super().emit(record)
def setup_logging():
# Create instance-specific directory
instance_id = socket.gethostname()
log_dir = f"/mnt/logs/{instance_id}"
os.makedirs(log_dir, exist_ok=True)
# Configure standard Python logging
logger = logging.getLogger('app')
logger.setLevel(logging.INFO)
# Create minute-rotating file handler
file_handler = MinuteRotatingFileHandler(log_dir, 'app')
file_handler.setFormatter(
logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
)
logger.addHandler(file_handler)
return logger
# Usage
logger = setup_logging()
logger.info("Application started - logs automatically flowing to S3")
logger.error("Sample error message")
```
```javascript theme={null}
const winston = require('winston');
const fs = require('fs');
const os = require('os');
const path = require('path');
class MinuteRotatingTransport extends winston.transports.File {
constructor(options) {
super(options);
this.logDir = options.logDir;
this.serviceName = options.serviceName;
this.currentMinute = null;
this.updateFilename();
// Check for rotation every 10 seconds
setInterval(() => {
this.checkRotation();
}, 10000);
}
updateFilename() {
const now = new Date();
const timestamp = now.toISOString()
.replace(/[-:]/g, '')
.replace(/\.\d{3}Z$/, '_00Z')
.replace(/T(\d{2})(\d{2})/, 'T$1_$2');
this.filename = path.join(this.logDir, `${this.serviceName}-${timestamp}.log`);
this.currentMinute = timestamp.slice(0, -4); // Remove '_00Z' suffix
}
checkRotation() {
const now = new Date();
const currentMinute = now.toISOString()
.replace(/[-:]/g, '')
.replace(/\.\d{3}Z$/, '')
.replace(/T(\d{2})(\d{2})/, 'T$1_$2')
.slice(0, -2); // Remove seconds
if (this.currentMinute !== currentMinute) {
this.close();
this.updateFilename();
this.open();
}
}
}
function setupLogging() {
// Create instance-specific directory
const instanceId = os.hostname();
const logDir = `/mnt/logs/${instanceId}`;
if (!fs.existsSync(logDir)) {
fs.mkdirSync(logDir, { recursive: true });
}
// Configure Winston logger with minute rotation
const logger = winston.createLogger({
level: 'info',
format: winston.format.combine(
winston.format.timestamp(),
winston.format.printf(({ timestamp, level, message }) => {
return `${timestamp} - ${level}: ${message}`;
})
),
transports: [
new winston.transports.Console(),
new MinuteRotatingTransport({
logDir: logDir,
serviceName: 'app'
})
]
});
return logger;
}
// Usage
const logger = setupLogging();
logger.info('Application started - logs automatically flowing to S3');
logger.error('Sample error message');
```
```go theme={null}
package main
import (
"fmt"
"log"
"os"
"path/filepath"
"sync"
"time"
)
type MinuteRotatingLogger struct {
logDir string
serviceName string
currentFile *os.File
currentMinute string
logger *log.Logger
mutex sync.Mutex
}
func NewMinuteRotatingLogger(logDir, serviceName string) *MinuteRotatingLogger {
mrl := &MinuteRotatingLogger{
logDir: logDir,
serviceName: serviceName,
}
mrl.rotateIfNeeded()
return mrl
}
func (mrl *MinuteRotatingLogger) rotateIfNeeded() {
mrl.mutex.Lock()
defer mrl.mutex.Unlock()
now := time.Now().UTC()
currentMinute := now.Format("20060102T15_04")
if mrl.currentMinute != currentMinute {
if mrl.currentFile != nil {
mrl.currentFile.Close()
}
timestamp := now.Format("20060102T15_04_00Z")
logFile := filepath.Join(mrl.logDir, fmt.Sprintf("%s-%s.log", mrl.serviceName, timestamp))
file, err := os.OpenFile(logFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
log.Fatalf("Failed to open log file: %v", err)
}
mrl.currentFile = file
mrl.currentMinute = currentMinute
mrl.logger = log.New(file, "", log.LstdFlags|log.Lshortfile)
}
}
func (mrl *MinuteRotatingLogger) Printf(format string, v ...interface{}) {
mrl.mutex.Lock()
defer mrl.mutex.Unlock()
mrl.rotateIfNeeded()
mrl.logger.Printf(format, v...)
}
func (mrl *MinuteRotatingLogger) Println(v ...interface{}) {
mrl.mutex.Lock()
defer mrl.mutex.Unlock()
mrl.rotateIfNeeded()
mrl.logger.Println(v...)
}
func setupLogging() *MinuteRotatingLogger {
// Create instance-specific directory
hostname, _ := os.Hostname()
logDir := filepath.Join("/mnt/logs", hostname)
os.MkdirAll(logDir, 0755)
return NewMinuteRotatingLogger(logDir, "app")
}
func main() {
logger := setupLogging()
logger.Println("Application started - logs automatically flowing to S3")
logger.Println("Sample log message")
}
```
```java theme={null}
import java.io.IOException;
import java.net.InetAddress;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.logging.FileHandler;
import java.util.logging.Handler;
import java.util.logging.Logger;
import java.util.logging.LogRecord;
import java.util.logging.SimpleFormatter;
class MinuteRotatingFileHandler extends Handler {
private final Path logDir;
private final String serviceName;
private FileHandler currentHandler;
private String currentMinute;
private final ScheduledExecutorService scheduler;
public MinuteRotatingFileHandler(Path logDir, String serviceName) throws IOException {
this.logDir = logDir;
this.serviceName = serviceName;
this.scheduler = Executors.newScheduledThreadPool(1);
rotateIfNeeded();
// Check for rotation every 10 seconds
scheduler.scheduleAtFixedRate(this::rotateIfNeeded, 10, 10, TimeUnit.SECONDS);
}
private synchronized void rotateIfNeeded() {
try {
LocalDateTime now = LocalDateTime.now(ZoneOffset.UTC);
String minute = now.format(DateTimeFormatter.ofPattern("yyyyMMdd'T'HH_mm"));
if (!minute.equals(currentMinute)) {
if (currentHandler != null) {
currentHandler.close();
}
String timestamp = now.format(DateTimeFormatter.ofPattern("yyyyMMdd'T'HH_mm_00'Z'"));
String logFile = logDir.resolve(serviceName + "-" + timestamp + ".log").toString();
currentHandler = new FileHandler(logFile, true);
currentHandler.setFormatter(new SimpleFormatter());
currentMinute = minute;
}
} catch (IOException e) {
System.err.println("Failed to rotate log file: " + e.getMessage());
}
}
@Override
public synchronized void publish(LogRecord record) {
rotateIfNeeded();
if (currentHandler != null) {
currentHandler.publish(record);
}
}
@Override
public void flush() {
if (currentHandler != null) {
currentHandler.flush();
}
}
@Override
public void close() throws SecurityException {
if (currentHandler != null) {
currentHandler.close();
}
scheduler.shutdown();
}
}
public class LoggingSetup {
public static Logger setupLogging() throws IOException {
// Create instance-specific directory
String hostname = InetAddress.getLocalHost().getHostName();
Path logDir = Paths.get("/mnt/logs", hostname);
Files.createDirectories(logDir);
// Configure Java logging with minute rotation
Logger logger = Logger.getLogger("AppLogger");
MinuteRotatingFileHandler fileHandler = new MinuteRotatingFileHandler(logDir, "app");
logger.addHandler(fileHandler);
return logger;
}
public static void main(String[] args) throws IOException {
Logger logger = setupLogging();
logger.info("Application started - logs automatically flowing to S3");
logger.severe("Sample error message");
}
}
```
## Log organization and management
With Archil, you don't need traditional log rotation because logs are automatically offloaded to S3, where there's infinite capacity. The standardized logging approach implements automatic minute-based rotation across all languages, creating new log files every minute with the consistent naming pattern `service_YYYYMMDDTHH_MM_00Z.log` for natural chronological organization while maintaining optimal query performance.
### S3 lifecycle management
One of the key advantages of using Archil for logging is that your logs automatically flow to S3, giving you access to all of S3's built-in cost optimization features. You can leverage S3's intelligent tiering and lifecycle policies to automatically move your log data to cheaper storage classes over time without any application changes.
Configure lifecycle policies on your S3 bucket to automatically transition logs to cheaper storage classes:
```json theme={null}
{
"Rules": [
{
"ID": "LogRetentionPolicy",
"Status": "Enabled",
"Filter": {
"Prefix": "logs/"
},
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
],
"Expiration": {
"Days": 365
}
}
]
}
```
## Querying logs with Amazon Athena
Once your logs are flowing to S3 through Archil, you can use Amazon Athena to run SQL queries directly against your log data without needing to load it into a database.
### Setting up Athena for log analysis
Create a table in Athena that leverages the standardized filename format for efficient time-based partitioning. The standardized `service_YYYYMMDDTHH_MM_00Z.log` format enables Athena to automatically partition data by time:
```sql theme={null}
CREATE EXTERNAL TABLE application_logs (
timestamp string,
level string,
message string,
module string,
function string,
line int
)
PARTITIONED BY (
year string,
month string,
day string,
hour string,
minute string,
instance_id string
)
STORED AS TEXTFILE
LOCATION 's3://your-bucket/logs/'
TBLPROPERTIES (
'projection.enabled' = 'true',
'projection.year.type' = 'integer',
'projection.year.range' = '2020,2030',
'projection.year.interval' = '1',
'projection.month.type' = 'integer',
'projection.month.range' = '1,12',
'projection.month.interval' = '1',
'projection.day.type' = 'integer',
'projection.day.range' = '1,31',
'projection.day.interval' = '1',
'projection.hour.type' = 'integer',
'projection.hour.range' = '0,23',
'projection.hour.interval' = '1',
'projection.minute.type' = 'integer',
'projection.minute.range' = '0,59',
'projection.minute.interval' = '1',
'projection.instance_id.type' = 'injected',
'storage.location.template' = 's3://your-bucket/logs/${instance_id}/app-${year}${month}${day}T${hour}_${minute}_00Z.log'
)
```
This time-based partitioning approach provides several advantages:
* **Efficient time-range queries**: Athena can skip entire partitions when filtering by time
* **Automatic partition discovery**: No need to manually add partitions as new log files are created
* **Cost optimization**: Only scan the data you need for time-based analysis
* **Scalability**: Performance remains consistent as log volume grows
For simpler use cases where you don't need fine-grained time partitioning, you can use a non-partitioned table:
```sql theme={null}
-- Non-partitioned approach (simpler setup, less efficient for large datasets)
CREATE EXTERNAL TABLE application_logs_simple (
timestamp string,
level string,
message string,
instance_id string,
module string,
function string,
line int
)
STORED AS TEXTFILE
LOCATION 's3://your-bucket/logs/'
```
### Example Athena queries
```sql theme={null}
-- Find all errors in the last 24 hours (leverages time partitioning)
SELECT timestamp, instance_id, message
FROM application_logs
WHERE level = 'ERROR'
AND year = '2024' AND month = '07' AND day = '29'
AND timestamp >= date_format(date_add('hour', -24, now()), '%Y-%m-%d %H:%i:%s')
ORDER BY timestamp DESC
LIMIT 100;
-- Count errors by instance (with partition pruning)
SELECT instance_id, COUNT(*) as error_count
FROM application_logs
WHERE level = 'ERROR'
AND year = '2024' AND month = '07' AND day >= '28'
AND timestamp >= date_format(date_add('day', -1, now()), '%Y-%m-%d')
GROUP BY instance_id
ORDER BY error_count DESC;
```
```sql theme={null}
-- Analyze log volume by hour
SELECT
date_format(from_iso8601_timestamp(timestamp), '%Y-%m-%d %H:00:00') as hour,
COUNT(*) as log_count,
COUNT(DISTINCT instance_id) as active_instances
FROM application_logs
WHERE timestamp >= date_format(date_add('day', -7, now()), '%Y-%m-%d')
GROUP BY date_format(from_iso8601_timestamp(timestamp), '%Y-%m-%d %H:00:00')
ORDER BY hour;
-- Find instances with unusual activity
SELECT
instance_id,
COUNT(*) as total_logs,
COUNT(CASE WHEN level = 'ERROR' THEN 1 END) as error_logs,
COUNT(CASE WHEN level = 'WARN' THEN 1 END) as warn_logs
FROM application_logs
WHERE timestamp >= date_format(date_add('hour', -1, now()), '%Y-%m-%d %H:%i:%s')
GROUP BY instance_id
HAVING COUNT(*) > 1000 OR COUNT(CASE WHEN level = 'ERROR' THEN 1 END) > 10
ORDER BY total_logs DESC;
```
```sql theme={null}
-- Search for specific patterns in log messages
SELECT timestamp, instance_id, level, message
FROM application_logs
WHERE message LIKE '%timeout%'
OR message LIKE '%connection%'
OR message LIKE '%failed%'
AND timestamp >= date_format(date_add('hour', -6, now()), '%Y-%m-%d %H:%i:%s')
ORDER BY timestamp DESC;
-- Analyze function call patterns
SELECT
module,
function,
COUNT(*) as call_count,
COUNT(CASE WHEN level = 'ERROR' THEN 1 END) as error_count
FROM application_logs
WHERE timestamp >= date_format(date_add('day', -1, now()), '%Y-%m-%d')
GROUP BY module, function
HAVING COUNT(*) > 100
ORDER BY call_count DESC;
```
# Firecracker
Source: https://docs.archil.com/guides/virtualization/firecracker
Learn how to run Archil inside of a Firecracker microVM
[Firecracker](https://firecracker-microvm.github.io/) is a lightweight virtualization technology from Amazon that
allows you to run Linux-based applications in a secure and isolated environment. Archil can be run inside of a
Firecracker microVM, giving these user-created containers an infinite disk that's synchronized to an S3 bucket.
By default, the Linux kernel used by Firecracker is configured without support for FUSE, which is required for
Archil. Therefore, you must recompile the kernel with FUSE support.
In AWS, Firecracker can only be run on bare-metal instances. The remainder of this guide assumes that you're running
on one of these instances, such as `c6g.metal`. This guide assumes that you've already installed or compiled the
Firecracker binary.
## Create an Archil disk
First, follow the Archil [Getting Started Guide](/getting-started/quickstart) to create an Archil disk that you wish to attach
to your Firecracker microVM.
## Compile the kernel
First, download the Kernel source code from kernel.org. The most recent version of the Kernel to be included
in the Firecracker CI is 6.1, so use that version. Also clone the Firecracker repository to get the most
up to date kernel configuration.
```bash theme={null}
# Download the kernel source code
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.tar.xz
tar -xvf linux-6.1.tar.xz
# Download the Firecracker source code for the most recent kernel configuration
git clone https://github.com/firecracker-microvm/firecracker.git
```
Move the Kernel configuration you plan to use into the Kernel source code directory.
```bash theme={null}
cp firecracker/resources/guest_configs/microvm-kernel-ci-aarch64-6.1.config linux-6.1/
```
Next, edit the Kernel configuration to enable FUSE support. Open the configuration file in an editor,
identify the line that says `CONFIG_FUSE_FS is not set`, and change it to `CONFIG_FUSE_FS=y`. The following command
should return `CONFIG_FUSE_FS=y`.
```bash theme={null}
grep CONFIG_FUSE_FS linux-6.1/microvm-kernel-ci-aarch64-6.1.config
```
Next, set this configuration as the default for the kernel build, and build the kernel.
```bash theme={null}
cd linux-6.1
cp microvm-kernel-ci-aarch64-6.1.config .config
make -j$(nproc)
cp arch/arm64/boot/Image ../vmlinux-6.1-fuse
```
Move the Kernel configuration you plan to use into the Kernel source code directory.
```bash theme={null}
cp firecracker/resources/guest_configs/microvm-kernel-ci-x86_64-6.1.config linux-6.1/
```
Next, edit the Kernel configuration to enable FUSE support. Open the configuration file in an editor,
identify the line that says `CONFIG_FUSE_FS is not set`, and change it to `CONFIG_FUSE_FS=y`. The following command
should return `CONFIG_FUSE_FS=y`.
```bash theme={null}
grep CONFIG_FUSE_FS linux-6.1/microvm-kernel-ci-x86_64-6.1.config
```
Next, set this configuration as the default for the kernel build, and build the kernel.
```bash theme={null}
cd linux-6.1
cp microvm-kernel-ci-x86_64-6.1.config .config
make bzImage -j$(nproc)
cp arch/x86/boot/bzImage ../vmlinux-6.1-fuse
```
## Starting the microVM
The remainder of this guide follows the
[Firecracker Quick Start Guide](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md).
Create an SSH key, and download the latest rootfs.
```bash theme={null}
ARCH="$(uname -m)"
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest_version=$(basename $(curl -fsSLI -o /dev/null -w %{url_effective} ${release_url}/latest))
CI_VERSION=${latest_version%.*}
latest_ubuntu_key=$(curl "http://spec.ccfc.min.s3.amazonaws.com/?prefix=firecracker-ci/$CI_VERSION/$ARCH/ubuntu-&list-type=2" \
| grep -oP "(?<=)(firecracker-ci/$CI_VERSION/$ARCH/ubuntu-[0-9]+\.[0-9]+\.squashfs)(?=)" \
| sort -V | tail -1)
ubuntu_version=$(basename $latest_ubuntu_key .squashfs | grep -oE '[0-9]+\.[0-9]+')
# Download a rootfs
wget -O ubuntu-$ubuntu_version.squashfs.upstream "https://s3.amazonaws.com/spec.ccfc.min/$latest_ubuntu_key"
# Create an ssh key for the rootfs
unsquashfs ubuntu-$ubuntu_version.squashfs.upstream
ssh-keygen -f id_rsa -N ""
cp -v id_rsa.pub squashfs-root/root/.ssh/authorized_keys
mv -v id_rsa ./ubuntu-$ubuntu_version.id_rsa
# create ext4 filesystem image
sudo chown -R root:root squashfs-root
truncate -s 400M ubuntu-$ubuntu_version.ext4
sudo mkfs.ext4 -d squashfs-root -F ubuntu-$ubuntu_version.ext4
# Verify everything was correctly set up and print versions
echo "Kernel: $(ls vmlinux-* | tail -1)"
echo "Rootfs: $(ls *.ext4 | tail -1)"
echo "SSH Key: $(ls *.id_rsa | tail -1)"
```
The `Kernel:` line should be the kernel you just compiled, `vmlinux-6.1-fuse`. To install the Archil client, you need
additional disk space for the binaries themselves along with SSL root certificates. Expand the size of the rootfs image:
```bash theme={null}
sudo truncate -s 1G ubuntu-$ubuntu_version.ext4
sudo resize2fs ubuntu-$ubuntu_version.ext4
```
Next, start the Firecracker server with the following commands:
```bash theme={null}
API_SOCKET="/tmp/firecracker.socket"
# Remove API unix socket
sudo rm -f $API_SOCKET
# Run firecracker
sudo ./firecracker --api-sock "${API_SOCKET}"
```
In a new terminal (keeping the terminal with the server running open), start the microVM with the following commands:
```bash theme={null}
TAP_DEV="tap0"
TAP_IP="172.16.0.1"
MASK_SHORT="/30"
# Setup network interface
sudo ip link del "$TAP_DEV" 2> /dev/null || true
sudo ip tuntap add dev "$TAP_DEV" mode tap
sudo ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
sudo ip link set dev "$TAP_DEV" up
# Enable ip forwarding
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sudo iptables -P FORWARD ACCEPT
# This tries to determine the name of the host network interface to forward
# VM's outbound network traffic through. If outbound traffic doesn't work,
# double check this returns the correct interface!
HOST_IFACE=$(ip -j route list default |jq -r '.[0].dev')
# Set up microVM internet access
sudo iptables -t nat -D POSTROUTING -o "$HOST_IFACE" -j MASQUERADE || true
sudo iptables -t nat -A POSTROUTING -o "$HOST_IFACE" -j MASQUERADE
API_SOCKET="/tmp/firecracker.socket"
LOGFILE="./firecracker.log"
# Create log file
touch $LOGFILE
# Set log file
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{
\"log_path\": \"${LOGFILE}\",
\"level\": \"Debug\",
\"show_level\": true,
\"show_log_origin\": true
}" \
"http://localhost/logger"
KERNEL="./$(ls vmlinux* | tail -1)"
KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1 pci=off"
ARCH=$(uname -m)
if [ ${ARCH} = "aarch64" ]; then
KERNEL_BOOT_ARGS="keep_bootcon ${KERNEL_BOOT_ARGS}"
fi
# Set boot source
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{
\"kernel_image_path\": \"${KERNEL}\",
\"boot_args\": \"${KERNEL_BOOT_ARGS}\"
}" \
"http://localhost/boot-source"
ROOTFS="./$(ls *.ext4 | tail -1)"
# Set rootfs
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{
\"drive_id\": \"rootfs\",
\"path_on_host\": \"${ROOTFS}\",
\"is_root_device\": true,
\"is_read_only\": false
}" \
"http://localhost/drives/rootfs"
# The IP address of a guest is derived from its MAC address with
# `fcnet-setup.sh`, this has been pre-configured in the guest rootfs. It is
# important that `TAP_IP` and `FC_MAC` match this.
FC_MAC="06:00:AC:10:00:02"
# Set network interface
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{
\"iface_id\": \"net1\",
\"guest_mac\": \"$FC_MAC\",
\"host_dev_name\": \"$TAP_DEV\"
}" \
"http://localhost/network-interfaces/net1"
# Increase machine memory to allow for running apt and Archil
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{
\"vcpu_count\": 1,
\"mem_size_mib\": 512
}" \
"http://localhost/machine-config"
# API requests are handled asynchronously, it is important the configuration is
# set, before `InstanceStart`.
sleep 0.015s
# Start microVM
sudo curl -X PUT --unix-socket "${API_SOCKET}" \
--data "{
\"action_type\": \"InstanceStart\"
}" \
"http://localhost/actions"
# API requests are handled asynchronously, it is important the microVM has been
# started before we attempt to SSH into it.
sleep 2s
KEY_NAME=./$(ls *.id_rsa | tail -1)
# Setup internet access in the guest
ssh -i $KEY_NAME root@172.16.0.2 "ip route add default via 172.16.0.1 dev eth0"
# Setup DNS resolution in the guest
ssh -i $KEY_NAME root@172.16.0.2 "echo 'nameserver 8.8.8.8' > /etc/resolv.conf"
# SSH into the microVM
ssh -i $KEY_NAME root@172.16.0.2
# Use `root` for both the login and password.
# Run `reboot` to exit.
```
## Installing and running Archil
You should now be SSH'd into the new microVM. You can now install the Archil client inside the microVM.
Run the remainder of this guide from the microVM itself. Start by installing the root certificates
to allow for SSL connections for downloading the Archil client and connecting to the Archil disk itself.
```bash theme={null}
apt update
apt install -y ca-certificates
update-ca-certificates
```
Next, install the Archil client.
```bash theme={null}
curl https://archil.com/install | sh
```
Finally, mount the Archil disk.
```bash theme={null}
mkdir -p /mnt/archil
export ARCHIL_MOUNT_TOKEN=""
archil mount /mnt/archil --region aws-us-east-1
```
Your microVM now has access to an unlimited disk that's synchronized to an S3 bucket. Your microVM can instantly
access data from the S3 bucket without having to first download it.
# AI SDK
Source: https://docs.archil.com/integrations/ai-sdk
Context your agent can understand
Agents are extremely capable of performing tasks, but they are limited by the amount of information available in their context. The Archil AI SDK integration gives agents the ability to store and retrieve information from an infinite filesystem, giving them access to the entirety of your data.
It exposes an interface they are already extremely familiar with -- the filesystem. It allows your agent to read files, write files, list directories, search with grep, and execute commands in a sandbox via [serverless execution](/compute/serverless-execution).
Go to the [Archil console](https://console.archil.com) and create a new disk. Also, [create an API key](https://console.archil.com/api-keys) in the same region as your disk. Add the following environment variables to your project:
```sh theme={null}
ARCHIL_API_KEY=
ARCHIL_REGION=
ARCHIL_DISK_ID=
```
```bash theme={null}
npm install disk @archildata/ai-sdk
```
Create tools for your agent to interact with the disk.
```ts theme={null}
import { generateText } from "ai";
import * as archil from "disk";
import { createDiskTools } from "@archildata/ai-sdk";
const disk = await archil.getDisk(process.env.ARCHIL_DISK_ID!);
const result = await generateText({
model: "anthropic/claude-sonnet-4.6",
prompt: "Read the quarterly reports and write a summary of the key findings for each one.",
tools: createDiskTools(disk),
});
```
# eve
Source: https://docs.archil.com/integrations/eve
Context your agent can understand
eve agents are built around the concept of filesystems as a core primitive. Archil now automatically provides the layer connecting them to your data by exposing tools for eve agents to read, write, and execute commands against data, as if it was a local filesystem.
Go to the [Archil console](https://console.archil.com) and create a new disk. Also, [create an API key](https://console.archil.com/api-keys) in the same region as your disk. Create a `.env.local` file in the root of your project with:
```sh theme={null}
ARCHIL_API_KEY=
ARCHIL_REGION=
ARCHIL_DISK_ID=
```
```bash theme={null}
npm install disk @archildata/eve
```
Give your agent tools to read/write from an existing Archil disk. Create `agent/tools/disk.ts`:
```ts theme={null}
import * as archil from "disk";
import { createDiskTools } from "@archildata/eve";
const disk = await archil.getDisk(process.env.ARCHIL_DISK_ID!);
export default createDiskTools(disk);
```
Optionally, you can add Archil as the compute and persistent storage provider for eve agent sessions. Create `agent/sandbox/sandbox.ts`:
```ts theme={null}
import { defineSandbox } from "eve/sandbox";
import { archilBackend } from "@archildata/eve";
export default defineSandbox({
backend: archilBackend({
disk: process.env.ARCHIL_DISK_ID!,
}),
});
```
# just-bash
Source: https://docs.archil.com/integrations/just-bash
[just-bash](https://justbash.dev) is a terminal emulator from Vercel designed to provide agents with an in-process bash emulator with a limited set of tools.
We offer the ability to use Archil disks from `just-bash` for agent environments where you do not have the ability to mount (such as AWS Lambda), and do not want to use a sandbox. For all users of `just-bash`, we recommend moving to [Serverless execution](/compute/serverless-execution), which gives you a full Linux environment to run commands like bash, node, and python on your disk data.
Our `just-bash` integration is built by linking our Typescript package against our native Linux client. As a result, it's only available on `x86_64` and `aarch64` Linux machines.
## Usage
You use the Archil `just-bash` integration through the exported `ArchilFs` file system. In order to give your agent the ability to use the `archil` CLI to manage delegations, we recommend also adding `archil` as a custom command with `createArchilCommand`.
```typescript theme={null}
import { ArchilClient } from '@archildata/client';
import { ArchilFs, createArchilCommand } from "@archildata/just-bash";
import { Bash } from "just-bash";
const client = await ArchilClient.connect({
region: 'aws-us-east-1',
diskName: 'myaccount/agent-workspace',
});
const fs = await ArchilFs.create(client);
const bash = new Bash({
fs,
customCommands: [createArchilCommand(client, fs)],
});
const result = await bash.exec("ls -la /");
console.log(result.stdout);
await bash.exec('echo "hello world" > /greeting.txt');
const cat = await bash.exec("cat /greeting.txt");
console.log(cat.stdout); // "hello world"
```
## Delegations
Writes through `ArchilFs` and the bash executor go through Archil's [delegation system](/concepts/sharing-disks). `writeFile` and `mkdir` handle delegations automatically when possible; for explicit control:
```typescript theme={null}
const inodeId = await fs.resolveInodeId("/mydir");
await client.checkout(inodeId);
await bash.exec('echo "hello" > /mydir/newfile.txt');
await client.checkin(inodeId);
```
# Subprocessors
Source: https://docs.archil.com/legal/subprocessors
Third parties Archil engages to process data on your behalf
*Last updated: April 30, 2026.*
Archil engages the subprocessors listed below to deliver the service. This page is referenced
from our [Data Processing Agreement](/details/security#data-protection-gdpr--ccpa) and is the
authoritative subprocessor list for DPA purposes.
## Customer data subprocessors
These subprocessors process disk contents and associated metadata.
| Subprocessor | Purpose | Location of processing | Entity jurisdiction |
| - | - | - | - |
| Amazon Web Services, Inc. | Disk metadata hosting for every disk1; active disk data hosting when AWS is the selected cloud | [Region you select](/reference/regions) | United States |
| Google LLC | Active disk data hosting when Google Cloud is the selected cloud | [Region you select](/reference/regions) | United States |
| Microsoft Corporation | Active disk data hosting when Microsoft Azure is the selected cloud | [Region you select](/reference/regions) | United States |
Each disk's active data is processed by **exactly one** of AWS, Google Cloud, or Microsoft Azure
— determined by the region you select when creating the disk. The other two are not engaged for
that disk.
**1.** "Disk metadata" means the disk's name, its list of authorized users, and billing information — and nothing more. Your file data, file paths, and file attributes are always stored exclusively in the cloud and region you select for the disk; they are never sent to AWS unless AWS is the cloud you selected. The metadata hosted in AWS does not include the names or contents of files within a disk.
## Regional contracting entities
Where a customer's contracting entity is in the EEA or UK, the applicable subprocessor contracting
entity may be a regional affiliate instead of the US entity listed above:
* **Amazon Web Services, Inc.** → Amazon Web Services EMEA SARL (Luxembourg)
* **Google LLC** → Google Cloud EMEA Limited (Ireland)
* **Microsoft Corporation** → Microsoft Ireland Operations Limited
## Transfers and change notifications
Applicable cross-border transfer mechanisms (such as the Standard Contractual Clauses) and
subprocessor change-notification commitments are set out in the DPA. To be notified of new or
replaced subprocessors, [contact us](mailto:security@archil.com).
## Customer-controlled data sources
If you configure a [data source](/concepts/data-sources), the data source provider (for example,
AWS S3, Google Cloud Storage, or Cloudflare R2) stores the synchronized copy of your data in the
bucket you own. You contract directly with that provider — they are not Archil subprocessors.
# Mount in a container
Source: https://docs.archil.com/mounting/containers
Mount Archil into a container when you need data that survives individual container lifetimes, or if you need to share data
across multiple containers. Containers, unlike regular Linux servers, require special permissions in order to mount external
file systems.
The best way to mount Archil into a container depends on the platform that you're using and the permissions that you have:
* **Local docker containers** support mounting Archil if they are started with `--privileged`, or mount to the host machine
and add Archil to your container as a Docker volume.
* **Kubernetes pods** support mounting Archil via the [CSI driver](/reference/csi-driver), follow instructions to install and use the CSI driver
on K8s deployments
* **Cloud container platforms** (such as AWS Lambda, ECS Fargate, or Modal container sandboxes) often do not support mounting
Archil because they do not give users sufficient permission to mount external file systems. For these users, we recommend
using [Serverless execution](/compute/serverless-execution) to send code that needs the disk to Archil directly.
* **Cloud sandboxes** (such as Daytona, E2B, and Modal VM sandboxes) often support mounting Archil directly. Consult your
provider's documentation for the best way to mount Archil.
The remainder of this article is about how to mount into a local Docker container.
## Mounting into a privileged container
You can easily mount an Archil disk into a privileged container, because it has full access to the host's Linux kernel,
and has permissions to mount external file systems. These steps are the same as mounting on a regular Linux machine.
Add the following lines to your Dockerfile to automatically install the Archil CLI to mount Archil disks, and
create the folder where you plan to mount Archil.
```bash theme={null}
RUN curl -s https://archil.com/install | sh
RUN mkdir /mnt/archil
```
Start your container with the `--privileged` flag to give your container permission to mount external file systems.
```bash theme={null}
docker run --privileged
```
Each container that connects to the Archil disk needs to be authorized. You can managed which servers are authorized
in the Archil console, by navigating to the disk that you want to use, then selecting **Connections** in the left-bar,
and clicking **Add User** to authorize a new server.
Select **Disk Token**, and save the token somewhere you can retrieve it, you will not be able to see it again.
Using the token that you just saved from the console, you can now mount the disk with `archil mount`.
```bash theme={null}
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region
```
## Mounting as a Docker volume
Follow the instructions for either [mounting to Linux](/mounting/linux) or [mounting to macOS](/mounting/macos) to
mount your Archil disk to the host running docker.
For example, for Linux, assuming you have the Archil CLI installed:
```bash theme={null}
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region
```
Once your disk is mounted to the host, you can run your container with the `docker run` command, using the `--volume` flag to bind-mount the disk to the container.
```bash theme={null}
docker run --volume /mnt/archil:/mnt/archil
```
# Mount on Linux
Source: https://docs.archil.com/mounting/linux
You can use the `archil` CLI to mount an Archil disk locally to an already running Linux server -- such as an Amazon EC2 instance,
a microVM sandbox, or a VPS.
Archil is compatible out-of-the-box with all programs that run on Linux, such as git, SQLite, or ffmpeg. For more details, see
[Application compatibility](/details/support).
This article is for servers which are running their own Linux kernel. If you're trying to mount Archil into a container
such as Docker, Podman, or a container platform (such as Amazon ECS) see [Mount in a container](/mounting/containers).
The `archil` client provides support for mounting Archil disks on Linux. First, install it on your server:
```bash theme={null}
curl -s https://archil.com/install | sh
```
Each server that connects to the Archil disk needs to be authorized. You can managed which servers are authorized
in the Archil console, by navigating to the disk that you want to use, then selecting **Connections** in the left-bar,
and clicking **Add User** to authorize a new server.
Select **Disk Token**, and save the token somewhere you can retrieve it, you will not be able to see it again.
Archil disks also support *IAM authentication* for users mounting from within AWS. For more details on how disks
authenticate users, see [Disk users](/concepts/disk-users).
First, create the directory that you intend to use to mount Archil:
```bash theme={null}
sudo mkdir -p /mnt/archil
```
Using the token that you just saved from the console, you can now mount the disk with `archil mount`.
```bash theme={null}
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/archil --region
```
You can now use any Linux program against your Archil disk by navigating to `/mnt/archil`, and running commands.
## Disk ownership
By default, new Archil disks are owned by the root user (Linux `uid:0`). When you mount your disk to a folder such as
`/mnt/archil`, that folder will become owned by the root user.
When not running as the root user, you will see `EACCES: Permission denied` for file operations. You can change the owner
of the root directory with the command:
`sudo chown $(id -u):$(id -g)" /mnt/archil`
The ownership change will be persisted
## Mounting on multiple hosts
By default, your disk is mounted in **exclusive-ownership** mode. If you would like to mount the disk on multiple servers,
instead call `archil mount --shared` and familiarize yourself with the [Shared disk](/concepts/shared-disks) semantics.
When you mount an Archil disk with `--shared`, you need to manually manage write-ownership of different files and folders
on the Archil disk. This requires calling `archil checkout ` before writing and `archil checkin ` after writing,
otherwise these operations will fail with `EROFS: Read-only file system`.
## Ensuring that data is persisted on shutdown
Like all local file systems, Archil automatically stages recently written data in-memory until an explicit `fsync` or a periodic
background flush job writes it to durable storage. This means that recently-written data may not be persisted if a machine
is unexpectedly powered off.
Archil sandboxes automatically manage this persistence on shutdown. On other sandbox providers, make sure that you explicitly
call `archil unmount` before powering off the sandbox.
We recommend **always** using `archil unmount` instead of the built-in Linux `umount` command, which does not automatically flush
data to Archil's servers.
## Mount automatically on startup
If your service automatically launches new instances, such as in a load-based auto scaling group, you'll want to ensure that new
servers automatically mount Archil at startup. In addition to service startup scripts, you can achieve this by adding Archil
to the server's `/etc/fstab` with a line like the following:
```bash theme={null}
archil _netdev,region=,defaults 0 0
```
**Parameters:**
* ``: An owner-qualified disk name (`organization/disk-name` or `employee@email.com/disk-name`) or disk ID (`dsk-0123456789abcdef`).
* ``: Local directory where the disk should be mounted (e.g., `/mnt/archil`).
* `archil`: Filesystem type for Archil disks.
* `_netdev`: Tells systemd this is a network filesystem that needs network connectivity before mount.
* `region=`: The region your disk is in (e.g., `aws-us-east-1`).
Example:
```bash theme={null}
# /etc/fstab
myorg/mydisk /mnt/data archil _netdev,region=aws-us-east-1,defaults 0 0
```
# Mount on macOS
Source: https://docs.archil.com/mounting/macos
In addition to mounting on Linux servers, Archil also supports mounting locally to a macOS machine for occassionally
debugging what is happening on the disk. Mounting Archil on macOS is supported on macOS 26 (Tahoe) and later.
When you mount an Archil disk to a laptop or desktop in an office, you're often introducing >100ms of network latency
between your computer and the Archil service. As a result, this configuration has reduced performance compared to mounting
Archil from an instance in the same cloud region as the Archil file system, and you should not use it for benchmarking
how your application runs against the service.
Mounting Archil on macOS is via a menu bar application. Install it by running the following:
```bash theme={null}
curl -s https://archil.com/install | sh
```
Click the Archil icon in your menu bar and select **Sign in to Archil**. Enter the email address associated with your Archil account, then enter the verification code we send you. Your session persists across restarts.
Once you're signed in, the menu bar lists every disk you have access to — both your personal disks and any disks shared with you through an [organization](/administration/organizations). Disks are grouped by region.
If you don't have a disk yet, create one in the [Archil console](https://console.archil.com) and click **Refresh Disks** in the menu.
Click any disk and select **Mount**. Archil mounts the disk as a native macOS volume, opens it in Finder, and adds a shortcut to your Finder sidebar. From there, you can browse, edit, and copy files just like any other folder.
To unmount cleanly, click the disk in the menu bar and select **Unmount**. Archil waits for any pending writes to flush to your data source before disconnecting. Quitting the app unmounts every active volume automatically.
## Checking out files
The macOS client mounts the disk in a shared mode, so it can be simultaneously used by other computers. In this mode,
calling `archil checkout` and `archil checkin` are not required, but should be used to improve the performance of metadata-heavy
workloads such as `git clone` or `npm install`. For more information on the semantics of `checkin` and `checkout`,
see [Shared disks](/concepts/shared-disks).
You can "check-out" a file or folder for writing by right-clicking it in Finder and selecting "Check out". An indicator
will appear on the file in Finder to show that it's locked exclusively for writing. When you're done working with the
folder or file, you can right-click it and select "Check in".
## Mount via command line
After installing the application and signing in to your Archil account, you can also mount and unmount disks from the
command line. You can mount a disk on macOS by running the following:
```bash theme={null}
sudo /usr/local/bin/archil-mount \
'archil:////?token=&shared=true' \
/Volumes/
```
On macOS, disks *must* be mounted underneath the `/Volumes` directory. You'll need to specify the `region`, `owner`, `disk-name`, and `disk-token` to mount.
To unmount the disk, you can call:
```bash theme={null}
sudo umount /Volumes/
```
The command-line interface on macOS is **unstable and subject to change** — the helper path, URL format, and flags may change in any release. We're actively working to make this a first-class, supported interface. For anything you depend on, prefer the menu bar, or mount from a [Linux client](/mounting/linux) where the [`archil` CLI](/reference/archil-cli) is fully supported.
## Sharing data with Linux servers
Disks mounted on your Mac are the same disks you can mount from Linux servers using the [`archil` CLI](/mounting/linux). Files you create or edit on macOS are immediately visible to any other client mounting the same disk, with full POSIX semantics.
# S3 API
Source: https://docs.archil.com/protocols/s3-api
Read and write Archil disks over an S3-compatible HTTP API — list, download, upload, and delete objects with any S3 client, no mount required.
Archil exposes every disk as an **S3-compatible bucket**. Point `aws s3`, `boto3`, `s5cmd`, or any other S3 client at the Archil endpoint, present per-disk SigV4 credentials, and read or write the disk's contents over HTTPS without installing the [Archil client](/mounting/linux) or running an [`exec`](/compute/serverless-execution).
This S3 view talks directly to the disk, so it's **[strongly read-after-write consistent](/details/consistency)** with every other client. An object you read over the S3 API always reflects the latest write — even one an agent made a moment ago through a [mount](/mounting/linux) or [`exec`](/compute/serverless-execution) — with no propagation delay and no stale cache. You can serve files from a website while an agent is actively editing them and trust that every read is current.
We also recommend using the S3 API to accelerate access to a disk: requests are served from Archil's high-performance, in-region SSD storage rather than going all the way to object storage, making it lower-latency than a general-purpose S3 bucket.
```bash theme={null}
aws s3 ls s3://dsk-abc123/ \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
```
For workloads that need full POSIX semantics (renames, locks, fast random reads), [mount the disk](/mounting/linux) instead — the S3 API isn't optimised for high-throughput streaming.
## Endpoint
The S3 API is served from a per-region endpoint. Use the endpoint that matches the region your disk lives in:
| Region | Code | Endpoint |
| - | - | - |
| AWS US East (N. Virginia) | `aws-us-east-1` | `https://s3.green.us-east-1.aws.prod.archil.com` |
| AWS US West (Oregon) | `aws-us-west-2` | `https://s3.green.us-west-2.aws.prod.archil.com` |
| AWS EU West (Ireland) | `aws-eu-west-1` | `https://s3.green.eu-west-1.aws.prod.archil.com` |
| GCP US Central (Iowa) | `gcp-us-central1` | `https://s3.blue.us-central1.gcp.prod.archil.com` |
The endpoint uses **path-style** addressing (`https:////`); virtual-host style is not supported. Most S3 clients require an explicit flag to force path style:
```bash AWS CLI theme={null}
aws s3 ls s3://dsk-abc123/ \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
```
```python boto3 theme={null}
import boto3
from botocore.config import Config
s3 = boto3.client(
"s3",
endpoint_url="https://s3.green.us-east-1.aws.prod.archil.com",
aws_access_key_id="",
aws_secret_access_key="",
region_name="us-east-1",
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
```
```bash s5cmd theme={null}
s5cmd \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com \
ls s3://dsk-abc123/
```
## Authentication
Each disk has its own set of S3 credentials — there are no account-wide S3 keys. To get credentials, add an **S3 user** to the disk:
1. Open the disk's **Details** page in the [Archil console](https://console.archil.com).
2. Under **Authorized Users**, click **Add User** and select **S3 credentials**.
3. The console shows the **Access Key ID** and **Secret Access Key** exactly once. Save them somewhere secure — the secret cannot be retrieved again.
Credentials are scoped to a single disk: presenting them on a request to a different bucket returns `AccessDenied`. To rotate, add a new S3 user and remove the old one. The S3 API uses standard **AWS Signature Version 4** — any S3 SDK signs correctly out of the box.
S3 credentials are distinct from [disk tokens](/concepts/disk-users#disk-token-authorization) (used by the Archil client when mounting) and from [Control Plane API keys](/api-reference/introduction) (account-level, used to manage disks). They cannot be used in place of either one.
## Bucket name
The bucket name is the **disk ID** as shown in the console (`dsk-…`).
```bash theme={null}
aws s3 ls s3://dsk-abc123/
```
To address a [branch](/concepts/branches-and-checkpoints) instead of the disk's root, append the branch name to the disk ID with a `.` separator. The same disk credentials authorize every branch.
```bash theme={null}
aws s3 ls s3://dsk-abc123.branch-xyz/
```
## Supported operations
| Operation | Method | Notes |
| - | - | - |
| `HeadBucket` | `HEAD /` | Verifies the disk exists and the credentials can read it. |
| `GetBucketLocation` | `GET /?location` | Returns the disk's region. |
| `ListObjectsV2` | `GET /?list-type=2` | Supports `prefix`, `delimiter`, `max-keys`, and `continuation-token`. |
| `GetObject` | `GET //` | Streams the object body. Supports HTTP `Range`. |
| `HeadObject` | `HEAD //` | Returns size, content type, and last-modified. |
| `PutObject` | `PUT //` | Creates or overwrites the file at ``. See [Writing](#writing). |
| `DeleteObject` | `DELETE //` | Removes the file at ``. Idempotent. |
| `DeleteObjects` | `POST /?delete` | Bulk-deletes up to 1,000 keys in one request; returns per-key results. |
| `AppendObject` | `PUT //?append=true` | Appends to an existing file (or creates it). Up to 1 MiB per call — see [Appending](#appending). |
| Multipart upload | `CreateMultipartUpload` / `UploadPart` / `CompleteMultipartUpload` / `AbortMultipartUpload` / `ListParts` / `ListMultipartUploads` | Upload large objects in parts — see [Large uploads](#large-uploads). Each part (except the last) must be ≥ 5 MiB. |
`CopyObject` and `ListBuckets` return `MethodNotAllowed` or `NotImplemented`. For renames or any other POSIX operation, mount the disk as a file system or use [`disk.exec`](/compute/serverless-execution).
### Listing
`ListObjectsV2` accepts the standard query parameters:
```bash theme={null}
aws s3api list-objects-v2 \
--bucket dsk-abc123 \
--prefix logs/ \
--delimiter / \
--max-keys 100 \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
```
The only delimiter that's accepted is `/`. Any other delimiter value returns a `400 InvalidArgument`. This matches how Archil clients see the underlying directory tree — `delimiter=/` collapses each directory into a single `CommonPrefixes` entry, exactly as a real S3 bucket would.
Pagination uses the standard `NextContinuationToken` — pass it back as `continuation-token` to fetch the next page.
### Range requests
`GetObject` honours the HTTP `Range` header end-to-end and returns `206 Partial Content` with a `Content-Range` header:
```bash theme={null}
# First 1 KiB
curl -H "Range: bytes=0-1023" https://s3.green.us-east-1.aws.prod.archil.com/dsk-abc123/data.bin
# Last 500 bytes
curl -H "Range: bytes=-500" https://s3.green.us-east-1.aws.prod.archil.com/dsk-abc123/data.bin
# Open-ended
curl -H "Range: bytes=1048576-" https://s3.green.us-east-1.aws.prod.archil.com/dsk-abc123/data.bin
```
`Accept-Ranges: bytes` is set on every response so clients can discover range support without an explicit probe.
### Content type
`Content-Type` is derived from the object key's extension when one is recognised; otherwise Archil sniffs the first 512 bytes. `HEAD` and `GET` always agree on the same content type for a given key.
### Writing
`PutObject` creates or overwrites the file at the given key, creating any parent directories as needed. The write commits to the disk before the response returns, so the object is immediately visible to every other client — the same [read-after-write consistency](/details/consistency) as a FUSE mount or `disk.exec`. When the disk is synchronized to a [data source](/concepts/data-sources), the new object is flushed back to the underlying bucket like any other write. `DeleteObject` is idempotent — deleting a key that doesn't exist still returns success — and if you send a `Content-MD5` header on a `PutObject`, Archil validates the body against it and rejects a mismatch. To remove many keys at once, `DeleteObjects` (`POST /?delete`) deletes up to 1,000 per request and reports per-key results; `aws s3 rm --recursive` and `s5cmd rm` use it automatically.
```bash theme={null}
# Upload a file
aws s3 cp report.json s3://dsk-abc123/reports/2026-01/data.json \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
# Delete it
aws s3 rm s3://dsk-abc123/reports/2026-01/data.json \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
```
### Large uploads
Multipart uploads are supported, so most S3 clients work unchanged for large files — the AWS CLI and boto3 automatically switch to multipart above their threshold (8 MB by default), uploading parts in parallel and assembling them server-side. `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, `ListParts`, and `ListMultipartUploads` are all available. Each part except the last must be at least **5 MiB**, and an upload may have at most **10,000 parts**.
```bash theme={null}
# Large files upload via multipart automatically — no special flags needed
aws s3 cp big.tar s3://dsk-abc123/backups/big.tar \
--endpoint-url https://s3.green.us-east-1.aws.prod.archil.com
```
A single `PutObject` body is capped at **5 GB**; larger objects (up to **5 TiB**) must go through multipart.
### Appending
`AppendObject` appends bytes to the end of an existing object, or creates it if the key doesn't exist yet — useful for log-style writes that grow a file over time without rewriting it. It's a `PutObject` with an `?append=true` query parameter:
```http theme={null}
PUT /dsk-abc123/logs/app.log?append=true
```
Each call may append at most **1 MiB**; to grow a file further, append in successive calls. Like every other write, an append commits before the response returns and is immediately visible to other clients. The standard CLI tools don't expose append, so it's easiest to use through the SDKs — `append_object` ([Python](/sdks/python#reading-and-writing-objects)) / `appendObject` ([TypeScript](/sdks/typescript#reading-and-writing-objects)).
## What gets exposed
The S3 API exposes the disk's filesystem as it currently exists — every file written through any client (FUSE mount, `disk.exec`, the Console file browser, or `PutObject` over this API) appears as an object with the key set to its POSIX path relative to the disk root, and objects written here show up identically on a mount. Newly written files are visible to the next S3 request after the write commits, with the same [read-after-write consistency](/details/consistency) as any other Archil client.
Symlinks, FIFOs, sockets, and device nodes appear in listings as zero-byte objects and can be deleted, but `GetObject`, `HeadObject`, and `CopyObject` on them return `501 NotImplemented`; read them through a mount. Directories appear as `CommonPrefixes` when you list with `delimiter=/`. There is no separate set of "S3 objects" — the bucket is just a different protocol view of the same data.
## Share URLs
For one-off, ad-hoc downloads, Archil can mint **signed share URLs** that don't require any credentials. Generate one from the console's file browser ("Share" in the row menu), or via the [Control Plane API](/api-reference/disks/create-share-url):
```bash theme={null}
curl -X POST \
-H "Authorization: ${ARCHIL_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"key": "path/to/file.csv", "expiresIn": 3600}' \
"https://control.green.us-east-1.aws.prod.archil.com/api/disks/dsk-abc123/share"
```
`expiresIn` is in seconds and accepts any integer from `1` to `604800` (7 days). The default is `86400` (24 hours). The response contains the share URL:
```json theme={null}
{
"success": true,
"data": {
"url": "https://control.green.us-east-1.aws.prod.archil.com/api/shared/",
"expiresIn": 3600
}
}
```
The returned URL can be opened in any browser — no credentials needed. It streams the file with a sensible `Content-Type` and `Content-Disposition` so files render inline where the browser supports it. HTTP `Range` requests are supported for partial downloads. Share URLs are signed by Archil's internal KMS key, so they survive control-plane restarts and work across replicas.
After the link expires, requests return `401 Unauthorized`. Deleting the underlying file causes the link to return `404 Not Found`.
## Limitations
* **`PutObject` is capped at 5 GB per request.** Larger objects (up to 5 TiB) must use [multipart](#large-uploads). `CopyObject` is unsupported; for renames, mount the disk.
* **Reserved paths are rejected.** Keys under `.archil/` are managed by Archil and can't be written or deleted.
* **Streaming-signed bodies aren't supported.** Configure your client to use `UNSIGNED-PAYLOAD` or a full-body SHA-256 (the default for `aws s3 cp` and `boto3`). A `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` request returns `NotImplemented`.
* **Active mount conflicts return `409`.** If a file is being actively written through a FUSE mount that holds a write [delegation](/concepts/shared-disks#ownership-rules), a conflicting `PutObject` or `DeleteObject` returns `409 Conflict` rather than racing the mounted client.
* **One bucket per credential.** S3 credentials are disk-scoped; cross-disk access requires a separate set of credentials.
* **`/` is the only supported delimiter** for `ListObjectsV2`.
* **Keys must fit the data source's key limit.** On a disk synchronized to a data source, a `PutObject` or `CreateMultipartUpload` key longer than the data source allows (1,024 bytes for Amazon S3, including the disk's bucket prefix) returns `400 KeyTooLongError`.
* **Non-empty directory markers can't be deleted.** `DeleteObject` on `key/` while that directory still has contents returns `400 InvalidRequest`.
* **No `ListBuckets`.** With disk-scoped credentials there is at most one bucket to list, so the operation isn't meaningful.
* **No S3 versioning, object lock, ACLs, tags, lifecycle rules, or bucket policies.** A disk is a filesystem exposed over S3, not a full S3 bucket. `PutObject` overwrites the previous contents at that key in place, and prior versions are not retained. The following APIs return `NotImplemented`:
* Versioning: `GetBucketVersioning`, `PutBucketVersioning`, `ListObjectVersions`
* Object lock: `Put/GetObjectLockConfiguration`, `Put/GetObjectRetention`, `Put/GetObjectLegalHold`
* Object ACLs and tagging
* Lifecycle and bucket policy APIs
To preserve point-in-time state, use [checkpoints and branches](/concepts/branches-and-checkpoints) instead. A checkpoint captures the entire disk at a moment in time and is immutable once created.
* **Throughput.** The S3 API is optimised for low latency and convenience. For sustained, high-throughput reads or writes use the [Archil client](/mounting/linux) — it talks the native protocol and benefits from local caching.
## Related
* [Branches & Checkpoints](/concepts/branches-and-checkpoints) — fork a disk and read a branch over S3 with `dsk-….`
* [Disk Users](/concepts/disk-users) — how authorization works for the disk's other access paths
* [Mounting on Linux](/mounting/linux) — the POSIX-protocol path, with full read/write support
* [Serverless Execution](/compute/serverless-execution) — run code directly on the disk instead of pulling data out
# `archil` CLI
Source: https://docs.archil.com/reference/archil-cli
Reference for the Linux system CLI that mounts Archil disks as local folders.
The `archil` CLI is the Linux system client you install with `curl -s https://archil.com/install | sh` to mount Archil disks on a Linux server. For a step-by-step walkthrough, see [Mount on Linux (FUSE)](/mounting/linux). To verify a versioned Linux artifact before installing it, see [Verify Linux downloads](/reference/verify-linux-downloads). This page is a reference for every command and flag it accepts.
To mount a disk on macOS, use the separate [Archil for macOS](/mounting/macos) app — it's an FSKit-based menu bar app, not the `archil` CLI.
## Global options
```bash theme={null}
archil [OPTIONS]
```
**Global options:**
* `--log-level `: Set log level (trace, debug, info, warn, error) \[default: info]
* `--version`: Print version information
* `--help`: Show help message
## Environment variables
The following environment variables can be used to configure the Archil CLI:
| Variable | Description |
| - | - |
| `ARCHIL_MOUNT_TOKEN` | The [disk token](/concepts/disk-users#disk-token-authorization) used to authenticate a mount when not using AWS IAM. Generate one from the disk's **Details** page in the Archil console. Pair with `sudo --preserve-env=ARCHIL_MOUNT_TOKEN` to pass it securely. |
**Example:**
```bash theme={null}
export ARCHIL_MOUNT_TOKEN=""
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount /mnt/data --region aws-us-east-1
```
Using `--preserve-env=ARCHIL_MOUNT_TOKEN` passes the disk token through sudo's environment rather than the command line, preventing it from being visible in process listings (`ps aux`).
`ARCHIL_MOUNT_TOKEN` is **not** an API key. It grants access to a single disk for mounting; API keys authenticate requests to the [Control Plane API](/api-reference/introduction) and are separate.
## mount
Mount a disk to a local directory.
```bash theme={null}
archil mount --region [OPTIONS]
```
**Arguments:**
* ``: An owner-qualified disk name (`organization/disk-name|employee@email.com/disk-name`) or disk ID (`dsk-0123456789abcdef`). Optionally append `:/path` to mount a [subdirectory](#subdirectory-mounts) as the root (e.g., `myorg/mydisk:/data`).
* ``: Local directory to mount the disk
**Options:**
**Connection:**
* `--region `: Region where the disk is located (e.g., `aws-us-east-1`, `gcp-us-central1`)
**Cache configuration:**
* `--max-cache-mb `: Maximum cache size in MB \[default: 1/4 of available RAM or 2048 MiB, whichever is smaller]
* `--target-cache-mb `: Target cache size in MB \[default: 75% of the maximum cache size]
* `--pre-fetch `: Comma-separated list of paths to pre-fetch into the cache after the mount completes (e.g., `models,data/2024`). Equivalent to running [`archil prefetch`](#prefetch) once the disk is mounted.
* `--read-ahead `: Maximum window the client reads ahead of sequential reads; `0` disables read-ahead \[default: 4194304 (4 MiB)]
**Behavior options:**
* `--shared`: Enable sharing the disk with multiple clients by skipping the checkout of the root directory \[default: false]. Without this flag, the client claims exclusive ownership of the root, and any attempt to mount the same disk from a second location will fail. See [Shared Disks](/concepts/shared-disks).
* `--enable-xattrs`: Enable extended attributes \[default: false]
* `--statistics`: Enable statistics collection \[default: false]
* `--force, -f`: Force claim ownership of the root directory, immediately revoking the delegations of any other clients \[default: false]
* `--branch `: Mount a specific [branch](#branches) of the disk instead of the root
Force claiming ownership of the root directory will immediately revoke the delegations of any other clients.
This can result in data loss for any outstanding writes from other clients to the disk.
**Process & logging:**
* `--no-fork`: Launch without forking \[default: false]
* `--log-dir `: Log to directory instead of journald
* `--max-log-size-mb `: Maximum log file size in MB \[default: 100]
**Examples:**
```bash theme={null}
# Basic mount
archil mount /mnt/archil --region aws-us-east-1
# Mount a subdirectory as the root
archil mount :/data /mnt/data --region aws-us-east-1
# Mount with custom cache settings
archil mount /mnt/archil --region aws-us-east-1 --max-cache-mb 2048 --target-cache-mb 1024
# Mount in shared mode with statistics
archil mount /mnt/archil --region aws-us-east-1 --shared --statistics
# Mount and warm the cache with specific paths
archil mount /mnt/archil --region aws-us-east-1 --pre-fetch models,data/2024
# Mount with custom logging
archil mount /mnt/archil --region aws-us-east-1 --log-dir /var/log/archil --max-log-size-mb 50
```
Always use `archil unmount` to unmount your Archil disks, not the builtin `umount` command. Unlike `umount`,
the `archil unmount` command does not exit until all pending data is flushed to the backing disk. This is particularly
important on system shutdown, in which the Linux kernel will, by default, call `umount` on all devices and shutdown
without waiting for pending Archil writes to be synced.
### Subdirectory mounts
You can mount a subdirectory of a disk instead of the entire root by appending `:/path` to the disk
name, following the same convention as NFS:
```bash theme={null}
# Mount the /data directory as the local root
sudo archil mount myorg/mydisk:/data /mnt/data --region aws-us-east-1
# Mount a nested subdirectory
sudo archil mount myorg/mydisk:/projects/alpha /mnt/alpha --region aws-us-east-1
```
The client resolves the path at mount time by walking from the disk root to the target directory,
following symlinks along the way. The resolved directory becomes the root of the FUSE mount — your
application sees it as `/` and cannot traverse above it via `..`.
The target path must be absolute (starting with `/` after the `:`) and must resolve to a directory
that already exists on the disk.
**Delegations with subdirectory mounts**
When mounting exclusively (without `--shared`), the client checks out ownership of the subdirectory
rather than the disk root. This means different clients can mount different subdirectories of the same
disk simultaneously without [delegation](/concepts/shared-disks) conflicts:
```bash theme={null}
# Client A mounts /models exclusively
sudo archil mount myorg/mydisk:/models /mnt/models --region aws-us-east-1
# Client B mounts /data exclusively — no conflict with Client A
sudo archil mount myorg/mydisk:/data /mnt/data --region aws-us-east-1
```
The subdirectory must exist on the disk before mounting. Create it by first mounting the disk root,
running `mkdir`, and then unmounting with `archil unmount`.
## unmount
Unmount a disk.
```bash theme={null}
archil unmount [OPTIONS]
```
**Arguments:**
* ``: Directory where the disk is mounted
**Options:**
* `--force, -f`: Force unmount operation, skipping fsyncing outstanding writes and releasing any ownership claims \[default: false]
Force unmounting a disk will skip fsyncing outstanding writes. This will result in data loss for any outstanding writes.
**Examples:**
```bash theme={null}
# Normal unmount
archil unmount /mnt/archil
# Force unmount
archil unmount /mnt/archil --force
```
## version
Display version information.
```bash theme={null}
archil version
```
## checkout
Request an ownership claim on a file or directory.
```bash theme={null}
archil checkout [OPTIONS]
```
**Arguments:**
* ``: Path to the file to check out
**Options:**
* `--force, -f`: Force checkout, revoking an existing delegation from another client \[default: false]
Force checkout will revoke ownership that other clients have on parents of the affected file or children of the affected file.
For more details, see the [Shared Disks](/concepts/shared-disks) page.
Force checkout will revoke any existing delegation from another client.
This can result in data loss for any outstanding writes from other clients to the affected file.
**Examples:**
```bash theme={null}
# Normal checkout
archil checkout /mnt/archil/myfile.txt
# Force checkout (with confirmation prompt)
archil checkout /mnt/archil/myfile.txt --force
```
## checkin
Check in a file, releasing ownership of a file or directory.
```bash theme={null}
archil checkin
```
**Arguments:**
* ``: Path to the file to check in
**Examples:**
```bash theme={null}
# Check in a file
archil checkin /mnt/archil/myfile.txt
```
## delegations
View the list of ownership claims that this client has on the disk mounted at ``.
```bash theme={null}
archil delegations
```
**Arguments:**
* ``: Directory where the disk is mounted
**Examples:**
```bash theme={null}
# View delegations
archil delegations /mnt/archil
```
## status
Display status information for a mounted disk, for example, whether or not the disk has failed and is not
accepting writes from this client.
```bash theme={null}
archil status
```
**Arguments:**
* ``: Directory where the disk is mounted
**Examples:**
```bash theme={null}
# View disk status
archil status /mnt/archil
```
## sync
Flush every pending write on a mounted disk to durable storage. Fails if the mount has failed or is read-only, since
unacknowledged writes cannot be flushed.
```bash theme={null}
archil sync
```
**Arguments:**
* ``: Directory where the disk is mounted
## update-token
Replace the disk token used by a live, token-authenticated mount. The new token is read from stdin so it never appears
in the process list.
```bash theme={null}
archil update-token
```
**Arguments:**
* ``: Directory where the disk is mounted
**Examples:**
```bash theme={null}
echo "$NEW_ARCHIL_TOKEN" | archil update-token /mnt/archil
```
## set-cache-expiry
Set the duration of time for which directories are kept cached locally instead of forcing a read from the server. This command configures the readdir cache expiry time for a specific directory.
```bash theme={null}
archil set-cache-expiry --readdir-expiry
```
**Arguments:**
* ``: Path to the directory where cache expiry should be set
**Options:**
* `--readdir-expiry `: Cache expiry duration in seconds
**Notes:**
* This command only applies to directories, not files
* Setting expiry to 0 seconds disables caching for the specified directory
* Cache settings help balance performance and data freshness
* Longer expiry times improve performance but may show stale directory contents
* The path must be on an Archil-mounted filesystem
**Examples:**
```bash theme={null}
# Set directory cache expiry to 5 minutes (300 seconds)
archil set-cache-expiry /mnt/archil/my-directory --readdir-expiry 300
# Disable caching for a directory
archil set-cache-expiry /mnt/archil/real-time-data --readdir-expiry 0
# Set cache expiry to 1 hour (3600 seconds)
archil set-cache-expiry /mnt/archil/static-content --readdir-expiry 3600
```
## invalidate-cache
*Available in client `v0.8.11` and later.*
Drop this client's cached entries (and the kernel's matching attribute, page, and dentry caches) so the next read goes back to the server. Pass the mount root to refresh everything, or a single file path to refresh just that file. Non-root directory paths are rejected. In-flight (dirty) writes are always preserved.
```bash theme={null}
archil invalidate-cache
```
**Arguments:**
* ``: An Archil mount root (whole-mount form) or a regular file inside an Archil mount (per-file form).
**Examples:**
```bash theme={null}
# Whole mount
archil invalidate-cache /mnt/archil
# One file (siblings keep their cache)
archil invalidate-cache /mnt/archil/shared-data/report.csv
```
Stale data on a `--shared` mount typically means another client has updated files on the disk and your client is still serving cached entries. See [Shared Disks](/concepts/shared-disks) for the full ownership and delegation model.
## prefetch
*Available in client `v0.8.11` and later.*
Warm this client's cache by reading paths into it ahead of time, so later reads are served locally instead of fetching from the server. Pass one or more files or directories; directories are fetched recursively. The command queues the work and returns immediately — fetching continues in the background.
```bash theme={null}
archil prefetch ...
```
**Arguments:**
* ``: Directory where the disk is mounted
* `...`: One or more paths to pre-fetch. Each path may be relative to the mount root (e.g., `data/report.csv`) or a full path under the mountpoint (e.g., `/mnt/archil/data/report.csv`) — the mountpoint prefix is stripped automatically, so shell tab-completion works.
**Examples:**
```bash theme={null}
# Pre-fetch a single file
archil prefetch /mnt/archil data/report.csv
# Pre-fetch several paths at once (files and/or directories, fetched recursively)
archil prefetch /mnt/archil models config.yaml data/2024
# Full paths under the mountpoint also work
archil prefetch /mnt/archil /mnt/archil/models
```
Pre-fetched data is read into the same client cache as regular reads, so it counts against your cache size limits (`--max-cache-mb`). If you pre-fetch more than the cache can hold, earlier entries may be evicted before you read them. To warm the cache automatically as soon as a disk is mounted, use [`mount --pre-fetch`](#mount).
## checkpoints
Manage checkpoints: named, point-in-time snapshots of a disk's filesystem state. A [branch](#branches) is always forked from a specific checkpoint. For details, see [Branches & Checkpoints](/concepts/branches-and-checkpoints).
### checkpoints create
Create a checkpoint capturing the current filesystem state of a mounted disk.
```bash theme={null}
archil checkpoints create
```
**Arguments:**
* ``: Local directory where the disk is mounted
* ``: Name for the new checkpoint (must be unique within the current branch)
**Examples:**
```bash theme={null}
# Checkpoint the current state of the disk mounted at /mnt/archil
archil checkpoints create /mnt/archil pre-migration
```
### checkpoints list
List checkpoints on a branch (or on the root if `--branch` is omitted).
```bash theme={null}
archil checkpoints list --region [OPTIONS]
```
**Arguments:**
* ``: Disk ID to list checkpoints for (e.g., `dsk-0123456789abcdef`)
**Options:**
* `--region `: Region where the disk is located (e.g., `aws-us-east-1`)
* `--branch `: Branch name to list checkpoints from. If omitted, lists checkpoints on the root.
* `--json`: Output as JSON \[default: false]
**Examples:**
```bash theme={null}
# List checkpoints on the root
archil checkpoints list dsk-0123456789abcdef --region aws-us-east-1
# List checkpoints on a specific branch
archil checkpoints list dsk-0123456789abcdef --region aws-us-east-1 --branch experiment-1
# Machine-readable output
archil checkpoints list dsk-0123456789abcdef --region aws-us-east-1 --json
```
## branches
Manage branches: independent, writable forks of a disk created from a [checkpoint](#checkpoints). Branches can be mounted with [`archil mount --branch`](#mount). For details, see [Branches & Checkpoints](/concepts/branches-and-checkpoints).
`branches` subcommands require a `--region` flag:
```bash theme={null}
archil branches --region
```
**Options (apply to all `branches` subcommands):**
* `--region `: Region where the disk is located (e.g., `aws-us-east-1`)
### branches create
Create a new branch forking a specific checkpoint.
```bash theme={null}
archil branches --region create --from-checkpoint [--from-branch ]
```
**Arguments:**
* ``: Disk ID to branch from (e.g., `dsk-0123456789abcdef`)
* ``: Name for the new branch
**Options:**
* `--from-checkpoint `: Name of the checkpoint to branch from (required)
* `--from-branch `: Name of the branch the checkpoint is on. Defaults to the root.
**Examples:**
```bash theme={null}
# Create a branch from a checkpoint on the root
archil branches --region aws-us-east-1 create dsk-0123456789abcdef experiment-1 \
--from-checkpoint pre-migration
# Create a branch from a checkpoint that lives on another branch
archil branches --region aws-us-east-1 create dsk-0123456789abcdef experiment-2 \
--from-checkpoint v1 --from-branch experiment-1
```
### branches list
List branches on a disk as a tree. Each node shows the branch name and the checkpoint it was forked from.
```bash theme={null}
archil branches --region list [OPTIONS]
```
**Arguments:**
* ``: Disk ID to list branches for (e.g., `dsk-0123456789abcdef`)
**Options:**
* `--limit `: Cap the number of branches shown in the tree. Excess branches are dropped newest-leaf-first, and a `... N more` marker is rendered beneath the closest still-visible parent to indicate how many direct children were truncated. Ignored when `--json` is set.
* `--json`: Output the full list as JSON, with no truncation \[default: false]
**Examples:**
```bash theme={null}
# Render the full branch tree
archil branches --region aws-us-east-1 list dsk-0123456789abcdef
# Cap the tree to the 10 oldest branches
archil branches --region aws-us-east-1 list dsk-0123456789abcdef --limit 10
# Machine-readable output (full list, never truncated)
archil branches --region aws-us-east-1 list dsk-0123456789abcdef --json
```
# Changelog
Source: https://docs.archil.com/reference/changelog
History of updates to the Archil platform and client
## September 2026
* Sandbox [timeouts](/compute/sandboxes/introduction#setting-a-time-to-live) now pause the sandbox instead of shutting it down. The hard timeout defaults to, and can be set up to, 24 hours; an optional idle timeout pauses a sandbox once no process connection is open. Change either on a live sandbox with `setTimeout` / `set_timeout` or the [Set Sandbox Timeouts](/api-reference/sandboxes/set-sandbox-timeout) endpoint.
* Sandbox [ports](/compute/sandboxes/introduction#hosting-network-services) can be exposed publicly or reached privately with port tokens. Services started with `archil services create` are private unless the sandbox is created with `enable_service_ingress`.
* The [sandbox firewall](/compute/sandboxes/firewall#draining-requests-before-pause) can let in-flight requests to selected hosts finish before a sandbox pauses.
* The [`disk` TypeScript SDK](/sdks/typescript#persistent-sandboxes) added sandbox timeouts and ports (**v1.2.0**), port tokens (**v1.3.0**), private certificate authorities (**v1.4.0**), and request draining (**v1.5.0**). **v1.6.0** requires Node.js 22.19 or later.
* The [`archil` Python SDK](/sdks/python#persistent-sandboxes) added sandbox timeouts, ports, port tokens, and request draining (**v0.14.0**).
* Client **v0.8.40** added [`archil sync`](/reference/archil-cli#sync), [`archil update-token`](/reference/archil-cli#update-token), and adaptive [read-ahead](/details/caching) (`mount --read-ahead`).
* The [S3-compatible API](/protocols/s3-api#what-gets-exposed) now lists and deletes symlinks and other special files, and rejects keys longer than the data source allows with `KeyTooLongError`.
## August 2026
* Preview launch of [persistent sandboxes](/compute/sandboxes/introduction) with persistent disks, interactive shells, services, pause and resume, and forks.
* Added persistent sandbox support to the [`disk` TypeScript SDK](/sdks/typescript#persistent-sandboxes) (**v0.9.0**).
## June 2026
* The [S3-compatible API](/protocols/s3-api#appending) now supports **appends** — `AppendObject` (`PUT //?append=true`) adds bytes to the end of an existing object (or creates it), up to 1 MiB per call, with the same [read-after-write consistency](/details/consistency) as any other write. This makes log-style and incrementally-grown files possible over the object API without rewriting the whole object or mounting. Exposed in the SDKs as `append_object` ([Python](/sdks/python#reading-and-writing-objects)) / `appendObject` ([TypeScript](/sdks/typescript#reading-and-writing-objects)).
* The [Python](/sdks/python#reading-and-writing-objects) and [TypeScript](/sdks/typescript#reading-and-writing-objects) SDKs gained large-object and bulk write operations on the S3-compatible object API. `put_object` / `putObject` now switch to a multipart upload automatically for large bodies — split into parts, uploaded in parallel, and assembled — with tunable `part_size`/`partSize`, `concurrency`, and `multipart_threshold`/`multipartThreshold`. `delete_objects` / `deleteObjects` removes up to 1,000 keys per request (returning per-key results), and the raw multipart lifecycle is available under the opt-in `disk.multipart` namespace. Transient failures (429/5xx/network) are now retried automatically with jittered exponential backoff.
* The [S3-compatible API](/protocols/s3-api) now supports **writes** — `PutObject` to create or overwrite a file and `DeleteObject` to remove one, both with the same [read-after-write consistency](/details/consistency) as any other Archil client. Point any S3 client at your disk to write objects without mounting. `CopyObject` is not supported yet. Writes are also exposed in the SDKs as `put_object` / `delete_object` ([Python](/sdks/python#reading-and-writing-objects)) and `putObject` / `deleteObject` ([TypeScript](/sdks/typescript#reading-and-writing-objects)).
* Launch of the [`archil`](https://pypi.org/project/archil/) Python package — a pure-Python control-plane SDK for creating disks, listing and inspecting them, managing API keys, running serverless commands with `Disk.exec`, and reading and writing disk contents through an S3-compatible object API. Every method works both synchronously and asynchronously via an `.aio` variant. No native dependencies; requires Python 3.10+. See the new [Python SDK reference](/sdks/python).
* Launch of [parallel search](/compute/search-files) — `disk.grep` searches every file under a directory for a regular expression, fanning the listing and matching out across many ephemeral containers so the search scales across many machines instead of being capped by one. You get back structured matches (file, line, text) plus a `stoppedReason` that says whether the search ran to completion or stopped early on your time or result limit. Available in the [`disk` SDK](/sdks/typescript#searching-files) (**v0.8.12**) and the [control plane API](/api-reference/serverless-execution/grep-disk).
* [Branches & checkpoints](/concepts/branches-and-checkpoints) are now generally available in all AWS regions. Capture named, immutable point-in-time snapshots of a disk's filesystem state (checkpoints) and fork independent, writable copies from any checkpoint (branches) without disturbing the original — much like Git commits and branches. Manage them from a mounted disk with the `archil checkpoints` and `archil branches` commands. Available on disks that are not synchronized to a [data source](/concepts/data-sources).
## May 2026
* Added [`archil prefetch`](/reference/archil-cli#prefetch) (client **v0.8.11**) to warm the client cache by reading files and directories into it ahead of time, so later reads are served locally instead of fetching from the server. Paths can also be pre-fetched automatically at mount time with [`mount --pre-fetch`](/reference/archil-cli#mount).
## April 2026
* Launch of the [read-only S3-compatible API](/protocols/s3-api) — point any S3 client (`aws s3 cp`, `boto3`, `s5cmd`) at your Archil disk and read its contents over HTTPS without mounting. Generate per-disk SigV4 credentials in the [Archil console](https://console.archil.com), use the disk ID as the bucket name, and call `HeadBucket`, `ListObjectsV2`, `GetObject` (with HTTP `Range`), or `HeadObject`. The same path also powers signed share URLs for one-off, time-limited file links from the console.
* Launch of [serverless execution](/compute/serverless-execution) — `disk.exec(command)` spins up a container with your file system already mounted, runs an arbitrary shell command to completion, and returns stdout, stderr, exit code, and timing. Pay only for the wall-clock time the command runs (1ms increments, 100ms minimum). Use it to give an agent a bash tool, fan work out across one disk, or get a result without pulling raw data through your client. See the new [Compute](/compute/serverless-execution) section.
* Launch of the [`disk`](https://www.npmjs.com/package/disk) npm package — a pure-JavaScript control-plane SDK and `npx disk` CLI for creating disks, listing and inspecting them, managing API keys, and running serverless commands with `disk exec`. No native dependencies; runs anywhere Node does. See the new [TypeScript SDK reference](/sdks/typescript) and [`disk` CLI reference](/reference/disk-cli).
* Launch of [Archil for macOS](/mounting/macos) — a native menu bar app that mounts your Archil disks as macOS volumes via Apple's FSKit framework. Sign in with email, browse personal and organization disks across regions, and mount them straight into Finder. Requires macOS 26 (Tahoe) or later on Apple Silicon. Install with `curl -s https://archil.com/install | sh`.
## February 2026
* Added [disk management support](/sdks/typescript#managing-disks) to the TypeScript SDK — create, list, and delete disks and API tokens programmatically via `@archildata/client`
* Added [subdirectory mounting](/reference/archil-cli#subdirectory-mounts) via `disk:/path` syntax — mount any subdirectory of a disk as the root of your local mount. Supports independent delegations, symlink resolution, and mount boundary isolation.
* Launch of the [Archil Terraform Provider](https://registry.terraform.io/providers/archil-data/archil/latest) for managing disks, users, and API tokens with Infrastructure as Code. See the [documentation](/reference/terraform) for details.
* Launch of the [Archil Control Plane API](/api-reference/introduction) for programmatic management of disks, mounts, and authentication tokens
## January 2026
* Launch of [`@archildata/just-bash`](https://www.npmjs.com/package/@archildata/just-bash) — just-bash bindings for the Archil filesystem
* Launch of the Archil Node.js client [`@archildata/client`](https://www.npmjs.com/package/@archildata/client)
## December 2025
* Launch of private preview regions for early access to new cloud regions
* Launch of managed buckets support for Google Cloud Storage
* Improved the speed of sequential read operations
* Launched client **v0.7.0** including new caching architecture, auth tokens via the `ARCHIL_AUTH_TOKEN` environment variable, and the preview `--writeback-cache` flag for workloads that perform frequent small writes to a small number of files.
## November 2025
* Launch of support for Azure Blob Storage as a data source
* Launch of managed bucket support for Amazon S3, enabling users to use Archil without a pre-existing S3 bucket
* Active data tracking provides visibility into how much data is actively cached
* Container runtime beta launched with TTY support, environment variables, and file injection
## October 2025
* Launch of scale-out read performance to achieve 8 GiB/s of throughput per client
* Improved Archil's ability to handle network congestion under load
* Launched client **v0.6.4**: Implemented "silly rename" to support unlinking opened file descriptors, added `--sts-region` flag for IAM authentication outside of AWS, improved authentication error messages
* Launch of Archil shells to use bash against Archil disks from the web console
## September 2025
* Launched client **v0.6.1**: Improved a rare issue related to dirty listing
* Launched client **v0.6.0**: Added additional tools to debug memory usage and performance
* Launched client **v0.5.0**: Reduced memory copies for faster writes, added `--read-only` mount flag, improved error messages
* Launched client **v0.4.0**: Disabled extended attributes by default to improve write performance (use `--enable-xattrs`), added `--nconnect` to work around hyperscale single-stream throughput limitations
## August 2025
* Launch Archil in Google Cloud us-central1 region
* Support the usage of cross-region data sources for Archil disks
* Launched client **v0.3.0**: Enables the ability to mount an Archil disks by name, and the ability to mount an Archil disk in a different cloud region
## July 2025
* Launch of the Archil file explorer to browse and manage Archil disks
* CSI driver improvements with better mount lifecycle management and ReadWriteMany (RWX) support
* Launched client **v0.2.0**: Initial preview release of the Archil Client
# Client Release Notes
Source: https://docs.archil.com/reference/client-release-notes
Release history for the Archil client
## v0.9.2
### Fixed
* `archil checkout` of a file or directory created less than about a second earlier no longer fails with "does not exist".
* On conditional mounts with a checked-out directory, deleting, renaming or hard-linking a file no longer fails with "Invalid cross-device link" after its other hard links outside the delegation have been removed.
* Closed a race in multi-writer scenarios that sometimes caused ENOENT failures when accessing a file immediately after creating it.
* Switching a mount to a read-only token with `archil update-token` now makes it read-only instead of failing it.
### macOS
* Disks mounted with a read-only token now mount read-only in the macOS app, so writes fail immediately instead of being accepted and rejected later.
## v0.9.1
### Fixed
* Fixed the default data cache being sized from the host's total RAM instead of the container's memory limit in privileged containers and on cgroup v1 nodes (such as some GKE node pools).
* Fixed renaming or deleting a file in a checked-out directory that also has a hard link outside it turning the whole mount read-only; the operation now fails with an error instead.
* Fixed a stall when listing a directory while `archil invalidate-cache` ran on a file in it on `--conditional` mounts.
* Fixed `archil set-cache-expiry --readdir-expiry` not applying to names learned from a directory listing, so files another client deleted or renamed now disappear after that interval instead of after five minutes.
* Fixed a rare crash or stale file view when a rename raced with other creates, renames, or deletes of the same paths on the same mount.
### Performance
* Preserved data in the kernel page cache when re-opening a file that has not changed since it was last opened, instead of reading it from the server again.
* Sped up writing large directory trees, such as `npm install`, by no longer asking the server about the contents of newly created directories.
## v0.9.0
### Performance
* Improved write throughput in conditional mode by up to 256x when performing small writes.
### Fixed
* Fixed kernel tuning reliability for CSI driver mounts on older kernels.
* Fixed a rare bug in which unmounting or closing a file emits an error.
* Fixed bugs related to stale timestamps and link counts for recently changed file attributes.
* Improved consistency of mount time.
* Various fixes around setuid/setgid during unprivileged fallocate or truncate when mounted in conditional mode.
* Fixed Linux musl (Alpine) client builds.
* Fixed a bug where the file data cache was underutilized, improving read performance for large files.
# Kubernetes CSI driver
Source: https://docs.archil.com/reference/csi-driver
Documentation on mounting using the [Kubernetes Container Storage Interface](https://kubernetes-csi.github.io/docs/) (CSI) driver for Archil disks.
The Archil CSI Driver provides seamless integration between Kubernetes and Archil's distributed disk technology.
It implements the CSI specification to allow Kubernetes workloads to mount and use
Archil disks with both ReadWriteOnce (RWO) and ReadWriteMany (RWX) access modes.
**Requirements:**
* **Helm chart distribution:** The driver is provided as an OCI Helm chart and requires Helm v3.8.0 or newer.
* **DNS access:** Pods must have access to external DNS to mount Archil disks. (Note: some providers, such as Fly.io, don’t enable this by default.)
## Dynamic provisioning
Dynamic provisioning automatically creates and deletes Archil disks when PersistentVolumeClaims are created and deleted. This is the recommended approach for most workloads.
### 1. Install CSI driver with controller
Create a Kubernetes secret with your [API key](/api-reference/introduction):
```bash theme={null}
kubectl create secret generic archil-controlplane-api-key \
--from-literal=api-key=YOUR_API_KEY \
--namespace archil-system
```
Install the CSI driver with the controller and StorageClass enabled:
```bash theme={null}
helm install archil-csi-driver \
oci://registry-1.docker.io/archildata/csi-driver-chart \
--namespace archil-system \
--create-namespace \
--set controller.enabled=true \
--set storageClass.enabled=true \
--set storageClass.region=aws-us-east-1 \
--set storageClass.nodeAuthType=awssts \
--set storageClass.iamRoleArn=arn:aws:iam::123456789012:role/ArchilNodeRole \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/ArchilNodeRole
```
```bash theme={null}
helm install archil-csi-driver \
oci://registry-1.docker.io/archildata/csi-driver-chart \
--namespace archil-system \
--create-namespace \
--set controller.enabled=true \
--set storageClass.enabled=true \
--set storageClass.region=aws-us-east-1 \
--set storageClass.nodeAuthType=token
```
With `nodeAuthType=token`, the controller automatically generates a unique [disk token](/concepts/disk-users#disk-token-authorization) per provisioned disk and stores it in a Kubernetes Secret. No manual token management is required.
### 2. Create a PersistentVolumeClaim
```yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-archil-pvc
spec:
accessModes:
- ReadWriteOnce # Or ReadWriteMany for multi-node access
storageClassName: archil
resources:
requests:
storage: 1Ti # Archil disks are elastic; this value is informational
```
### 3. Use the disk in a pod
```yaml theme={null}
apiVersion: v1
kind: Pod
metadata:
name: archil-test
spec:
containers:
- name: app
image: busybox
command: ['sh', '-c', 'echo "Hello from Archil" > /data/hello.txt && cat /data/hello.txt']
volumeMounts:
- name: archil-storage
mountPath: /data
volumes:
- name: archil-storage
persistentVolumeClaim:
claimName: my-archil-pvc
restartPolicy: Never
```
When the PVC is created, the controller automatically provisions an Archil disk via the controlplane API. When the PVC is deleted (with `reclaimPolicy: Delete`), the disk is cleaned up automatically.
***
## Static provisioning
If you need to use a pre-existing Archil disk, you can create PersistentVolumes manually.
### 1. Install CSI driver
Install the CSI driver using our official Helm chart from [Docker Hub](https://hub.docker.com/r/archildata/csi-driver-chart). The chart deploys the [archildata/csi-driver](https://hub.docker.com/r/archildata/csi-driver) container image.
```bash theme={null}
helm install archil-csi-driver \
oci://registry-1.docker.io/archildata/csi-driver-chart \
--namespace archil-system \
--create-namespace
```
### 2. Configure disk users
Archil natively supports using AWS IAM users or roles to authorize access to the disk. Simply add the ARN of the IAM user or role that will be used to connect to the disk.
* Get your EC2 server's IAM role arn using the following command:
```bash theme={null}
archil utils get-iam-role
```
* Copy the resulting IAM role arn.
* Update your helm release:
```bash theme={null}
helm upgrade archil-csi-driver \
oci://registry-1.docker.io/archildata/csi-driver-chart \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/ArchilCSIRole
```
* Restart the CSI driver daemonset:
```bash theme={null}
kubectl rollout restart daemonset/archil-csi-node -n archil-system
```
This IAM role is used for authentication from the CSI driver to *all* disks. The Archil CSI Driver does not currently support specifying IAM roles on a per-disk basis.
Outside of AWS, Archil can issue a [disk token](/concepts/disk-users#disk-token-authorization) — a per-disk credential used to mount the disk.
* Navigate to the disk's **Details** page in the Archil console
* Click **Generate Token** to create a new disk token
* Copy the disk token
* Create a Kubernetes secret containing the disk token:
```bash theme={null}
# Replace dsk-XXXXXXXXXXXX with your disk ID and YOUR_DISK_TOKEN with the token you just copied.
kubectl create secret generic archil-token-dsk-XXXXXXXXXXXX \
--from-literal=token=YOUR_DISK_TOKEN \
--namespace=archil-system
```
This disk token is **not** the same as the control-plane API key used above for dynamic provisioning. The API key manages disks; the disk token grants mount access to one specific disk.
### 3. Create PersistentVolume
```yaml theme={null}
apiVersion: v1
kind: PersistentVolume
metadata:
name: archil-pv
spec:
capacity:
# Can be any size, Archil disks are infinite
storage: 1000Gi
accessModes:
# Or use ReadWriteOnce for single-node access
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: csi.archil.com
# Your disk ID from Archil console
volumeHandle: dsk-XXXXXXXXXXXX
volumeAttributes:
# Region where your disk exists
region: aws-us-east-1
# Only include this section if using disk token authentication:
# nodePublishSecretRef:
# name: archil-token-dsk-XXXXXXXXXXXX
# namespace: archil-system
```
### 4. Create PersistentVolumeClaim
```yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archil-pvc
namespace: default
spec:
# Must match PV access mode
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1000Gi
# Must match PV name
volumeName: archil-pv
```
### 5. Create a pod
This example creates a simple demo pod to verify connectivity to your Archil disk. You can deploy your own applications using any pod configuration, just mount the PVC as shown below.
```yaml theme={null}
apiVersion: v1
kind: Pod
metadata:
name: archil-test
spec:
containers:
- name: test
image: busybox
command: ['sh', '-c', 'ls -la /data && echo "---" && echo "Archil disk contents listed above"']
volumeMounts:
- name: archil-storage
mountPath: /data
volumes:
- name: archil-storage
persistentVolumeClaim:
# Must match PVC name
claimName: archil-pvc
restartPolicy: Never
```
## Configuration reference
### Storage access modes
The `accessMode` that you use when creating your PersistentVolume and PersistentVolumeClaim will determine whether the
disk is mounted with support for other clients to access it. When using `ReadWriteOnce`, the disk is mounted into single
ownership mode, preventing other clients from accessing the disk. When using `ReadWriteMany`, the disk is mounted with
`--shared`, allowing multiple clients to access the disk simultaneously. For more details, see the
[Shared Disks](/concepts/shared-disks) page.
### Helm chart values
The following table describes all available configuration options in `values.yaml`:
| Parameter | Description | Default | Required |
| - | - | - | - |
| `image.tag` | Container image tag | `0.1.0` | No |
| `image.pullPolicy` | Image pull policy | `IfNotPresent` | No |
| `serviceAccount.annotations` | Service account annotations for AWS IRSA | `{}` | No |
| `extraEnv` | Additional environment variables for the CSI driver container | `[]` | No |
| `resources` | Resource limits and requests for the CSI driver container | `{}` | No |
| `nodeSelector` | Node selector for CSI driver pod placement | `{}` | No |
| `tolerations` | Pod tolerations | `[]` | No |
| `affinity` | Pod affinity rules | `{}` | No |
| `controller.enabled` | Enable the controller plugin for dynamic provisioning | `false` | No |
| `controller.apiKeySecretName` | Name of the K8s Secret containing the API key | `archil-controlplane-api-key` | No |
| `controller.controlplaneURL` | Override controlplane URL (empty = auto-detect from region) | `""` | No |
| `controller.serviceAccount.annotations` | Annotations for the controller service account | `{}` | No |
| `storageClass.enabled` | Create a StorageClass for dynamic provisioning | `false` | No |
| `storageClass.name` | StorageClass name | `archil` | No |
| `storageClass.isDefault` | Set as default StorageClass | `false` | No |
| `storageClass.reclaimPolicy` | PV reclaim policy (`Delete` or `Retain`) | `Delete` | No |
| `storageClass.volumeBindingMode` | When to bind volumes (`Immediate` or `WaitForFirstConsumer`) | `Immediate` | No |
| `storageClass.region` | Archil region for provisioned disks | `""` | Yes (if enabled) |
| `storageClass.nodeAuthType` | Node authentication type (`awssts` or `token`) | `awssts` | No |
| `storageClass.iamRoleArn` | IAM role ARN for node authentication | `""` | Yes (if awssts) |
## Changelog
* Dynamic provisioning via controller plugin
* Automatic disk creation/deletion when PVCs are created/deleted
* Support for both IAM (AWS STS) and disk-token-based node authentication
* New StorageClass with configurable region and auth type
* Initial preview release
# `disk` CLI
Source: https://docs.archil.com/reference/disk-cli
Reference for the Node CLI that creates disks, runs `disk exec`, and manages API keys.
The [`disk`](https://www.npmjs.com/package/disk) npm package ships a CLI under the same name. Run it with `npx disk` from any machine that has Node — no install required, no platform restrictions. Use it to create disks, list and inspect them, manage who can mount them, and run [serverless commands](/compute/serverless-execution) against the data — all from a script, CI job, or your terminal. For a step-by-step walkthrough, see the [Quickstart](/getting-started/quickstart).
```bash theme={null}
npx disk [options]
```
`disk` talks to the Archil control plane over HTTPS. It has no native dependencies and works on any platform Node.js does. This page is a reference for every command it accepts; for a walkthrough, see the [Quickstart](/getting-started/quickstart).
## Authentication
`disk` reads credentials from environment variables or flags:
| | Env var | Flag |
| - | - | - |
| API key | `ARCHIL_API_KEY` | `-k`, `--api-key` |
| Region | `ARCHIL_REGION` | `-r`, `--region` |
API keys are account-level credentials. They are **not** the same as disk tokens — see the [disk users concept page](/concepts/disk-users#disk-token-authorization).
## Disks
### `disk list`
List the disks in the current region.
```bash theme={null}
npx disk list [--limit ] [-o table|json]
```
Outputs a box-drawing table by default; `-o json` emits an array suitable for piping into `jq`.
### `disk get `
Show details for a disk: status, region, mounts, authorized users, connected clients.
```bash theme={null}
npx disk get dsk-abc123 [-o table|json]
```
### `disk create `
Create a new disk. The response includes a one-time **disk token** for mounting — save it; it cannot be retrieved again.
```bash theme={null}
npx disk create my-disk
```
### `disk delete `
Delete a disk. This does not delete the underlying bucket data.
```bash theme={null}
npx disk delete dsk-abc123
```
### `disk exec `
Run a bash command inside a container with the disk mounted at `/mnt/archil`, which is also the working directory — so relative paths address the disk root. Stdout, stderr, and the exit code from the remote command are forwarded back. See [Serverless Execution](/compute/serverless-execution) for the full model.
```bash theme={null}
npx disk exec dsk-abc123 "ls -la && cat config.json"
```
`disk` also accepts the sugar form `disk exec ` for ergonomics:
```bash theme={null}
npx disk dsk-abc123 exec "du -sh logs"
```
The CLI process exits with the remote command's exit code, so it composes naturally with shell pipelines and CI step outcomes.
## API keys
API keys live at the account level. Manage them with the `api-keys` subcommand.
### `disk api-keys list`
```bash theme={null}
npx disk api-keys list [--limit ]
```
### `disk api-keys create `
```bash theme={null}
npx disk api-keys create ci-bot --description "GitHub Actions"
```
The key value is shown once at create time. Save it.
### `disk api-keys delete `
```bash theme={null}
npx disk api-keys delete key-abc123
```
## Global options
```bash theme={null}
npx disk --help
npx disk --version
# Override credentials per-invocation rather than using env vars
npx disk --api-key key-... --region aws-us-east-1 list
# Point at a non-production control plane
npx disk --base-url https://staging.archil.com list
```
# Region availability
Source: https://docs.archil.com/reference/regions
Archil regions and availability
Archil disks are available in multiple regions across AWS and Google Cloud Platform (GCP). When creating a disk, you can select the region that best suits your needs. For optimal performance, we recommend choosing a region closest to where your compute resources are located.
## AWS regions
Archil disks are available in the following AWS regions:
* **US East (N. Virginia)** - `aws-us-east-1`
* **US West (Oregon)** - `aws-us-west-2`
* **EU West (Ireland)** - `aws-eu-west-1`
## Google Cloud Platform regions
Archil disks are available in the following GCP regions:
* **US Central (Iowa)** - `gcp-us-central1`
We are continuously expanding our regional presence. If you need Archil in a region not listed here, please [contact us](https://archil.com/contact).
## Serverless execution availability
[Serverless execution](/compute/serverless-execution) (`disk.exec` and `disk.grep`) is available only in the AWS regions:
| Region | Code | Disk storage | Serverless execution |
| - | - | - | - |
| US East (N. Virginia) | `aws-us-east-1` | ✅ | ✅ |
| US West (Oregon) | `aws-us-west-2` | ✅ | ✅ |
| EU West (Ireland) | `aws-eu-west-1` | ✅ | ✅ |
| US Central (Iowa) | `gcp-us-central1` | ✅ | — |
A disk created in a region without serverless execution works for storage, but calling `exec` returns **`Exec is not enabled`** (and `grep` returns `Grep is not enabled`). If you plan to run commands on a disk, create it in one of the AWS regions above.
# Terraform provider
Source: https://docs.archil.com/reference/terraform
Manage Archil disks, users, and API tokens with Infrastructure as Code
The [Archil Terraform provider](https://registry.terraform.io/providers/archil-data/archil/latest) enables you to manage Archil resources using Infrastructure as Code. It supports creating and managing disks, API tokens, and disk user authorizations.
## Installation
Add the provider to your `required_providers` block:
```hcl theme={null}
terraform {
required_providers {
archil = {
source = "archil-data/archil"
}
}
}
```
Then run `terraform init` to install the provider.
## Provider configuration
```hcl theme={null}
provider "archil" {
api_key = var.archil_api_key
region = "aws-us-east-1"
}
```
### Arguments
| Argument | Description | Required |
| - | - | - |
| `api_key` | API key for authentication. Can also be set via `ARCHIL_API_KEY` environment variable. Accepts the key with or without the `key-` prefix. | Yes |
| `region` | Archil region: `aws-us-east-1`, `aws-us-west-2`, `aws-eu-west-1`, `gcp-us-central1`, or `gcp-us-west1`. Can also be set via `ARCHIL_REGION`. Default: `aws-us-east-1`. See [Region Availability](/reference/regions) for details. | No |
| `endpoint` | Override the API endpoint URL. Can also be set via `ARCHIL_ENDPOINT`. | No |
Store your API key securely. Use a `terraform.tfvars` file (and add it to `.gitignore`) or an environment variable rather than hardcoding the key in your configuration.
## Resources
### archil\_disk
Creates and manages an Archil disk. A [storage mount](/concepts/data-sources) is optional.
```hcl theme={null}
resource "archil_disk" "default" {
name = "my-disk"
}
# Synchronize with your own S3 bucket
resource "archil_disk" "with_obj_store" {
name = "my-disk"
mount {
type = "s3"
bucket_name = "my-s3-bucket"
}
}
```
#### Arguments
| Argument | Description | Required |
| - | - | - |
| `name` | Disk name (1-100 chars, alphanumeric, dashes, underscores). | Yes |
| `mount` | Storage mount configuration block. | No |
#### Mount block arguments
| Argument | Description | Required |
| - | - | - |
| `type` | Mount type: `s3`, `gcs`, `r2`, `s3-compatible`, or `azure-blob`. | Yes |
| `bucket_name` | Bucket name. | When `type = "s3"`, `"gcs"`, `"r2"`, or `"s3-compatible"` |
| `bucket_endpoint` | Storage endpoint URL (auto-populated for S3). | When `type = "r2"` or `"s3-compatible"` |
| `bucket_prefix` | Prefix within the bucket. | No |
| `bucket_region` | Bucket region (e.g., `us-east-1`). Auto-detected when omitted. | No |
| `access_key_id` | Access key ID (sensitive). | When `type = "gcs"`, `"r2"`, or `"s3-compatible"` |
| `secret_access_key` | Secret access key (sensitive). | When `type = "gcs"`, `"r2"`, or `"s3-compatible"` |
| `session_token` | Session token for temporary credentials (sensitive). | No |
| `session_id` | Session identifier for IAM role-based auth. Auto-generated when no access keys are provided; readable after apply as `mount[0].session_id` for use in bucket policies (see [S3 bucket resource policy](#s3-bucket-resource-policy)). | No |
| `container_name` | Azure blob container name. | When `type = "azure-blob"` |
| `endpoint` | Azure blob endpoint URL. | No |
| `storage_account_name` | Azure storage account name. | No |
| `tenant_id` | Azure AD tenant ID. | When `type = "azure-blob"` |
| `client_id` | Azure AD client ID. | When `type = "azure-blob"` |
| `client_secret` | Azure AD client secret (sensitive). | When `type = "azure-blob"` |
All disk attributes require replacement -- changing any attribute will destroy and recreate the disk.
#### Attributes
| Attribute | Description |
| - | - |
| `id` | Disk ID (e.g., `dsk-0123456789abcdef`). |
| `status` | Disk status (e.g., `available`). |
| `organization` | Owning organization. |
| `provider_name` | Cloud provider (e.g., `aws`). |
| `region` | Disk region (e.g., `aws-us-east-1`). |
| `created_at` | Creation timestamp. |
| `data_size` | Total data size in bytes. |
| `token` | Auto-generated mount token (sensitive, only available at creation). Use it to mount the disk without creating a separate disk user. |
| `token_identifier` | Identifier of the auto-generated token user, for managing or removing it. |
#### S3 bucket resource policy
When an `s3` mount has no access keys, Archil is authorized through a [bucket resource policy](/data-sources/s3-object-storage#bucket-resource-policy). A policy attached to your bucket grants access to Archil's IAM role, and an `archilSessionId` condition scopes that grant to this one disk.
```hcl theme={null}
resource "archil_disk" "app" {
name = "my-app-disk"
mount {
type = "s3"
bucket_name = aws_s3_bucket.data.id
}
}
resource "aws_s3_bucket_policy" "archil_access" {
bucket = aws_s3_bucket.data.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Sid = "AllowArchilAccess"
Effect = "Allow"
Principal = { AWS = "arn:aws:iam::904233114299:role/archil-s3.prod.us-east-1" }
Action = ["s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject",
"s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"]
Resource = [aws_s3_bucket.data.arn, "${aws_s3_bucket.data.arn}/*"]
Condition = {
StringEquals = {
"aws:PrincipalTag/archilSessionId" = archil_disk.app.mount[0].session_id
}
}
}]
})
}
```
The `Principal` ARN varies by Archil region, see [S3 Object Storage](/data-sources/s3-object-storage#bucket-resource-policy) for the full table and policy details.
If your bucket's policy is managed outside this configuration, either export the policy with an `output` and apply it separately or set `session_id` explicitly in the mount block.
***
### archil\_api\_token
Creates and manages an Archil API token for programmatic access.
```hcl theme={null}
resource "archil_api_token" "ci" {
name = "ci-token"
description = "Token for CI/CD pipelines"
}
```
#### Arguments
| Argument | Description | Required |
| - | - | - |
| `name` | Token name (1-100 chars). | Yes |
| `description` | Token description (max 500 chars). | No |
#### Attributes
| Attribute | Description |
| - | - |
| `id` | Token ID (hash). |
| `token` | Full token value (sensitive, only available at creation). |
| `token_suffix` | Last 4 characters of the token. |
| `created_at` | Creation timestamp. |
***
### archil\_disk\_user
Adds an authorized user to an Archil disk. Supports token-based and AWS STS authentication. For token users the server generates a cryptographically secure mount token and returns it in the `token` attribute.
```hcl theme={null}
resource "archil_disk_user" "token_user" {
disk_id = archil_disk.default.id
type = "token"
nickname = "ci-pipeline"
}
resource "archil_disk_user" "iam_user" {
disk_id = archil_disk.default.id
type = "awssts"
principal = "arn:aws:iam::123456789012:role/MyRole"
}
```
#### Arguments
| Argument | Description | Required |
| - | - | - |
| `disk_id` | ID of the disk to add the user to. | Yes |
| `type` | User type: `token` or `awssts`. | Yes |
| `principal` | IAM ARN. | When `type = "awssts"` |
| `nickname` | Nickname for the user. | When `type = "token"` |
| `ttl` | Time-to-live as a Go duration string (e.g., `"1h"`, `"30m"`, `"24h"`; max `"8760h"`). Mutually exclusive with `expires_at`. | No |
| `expires_at` | Absolute expiration as an RFC3339 timestamp (e.g., `"2027-01-01T00:00:00Z"`). Mutually exclusive with `ttl`. | No |
| `one_use` | When `true`, the credential works for a single mount authentication only. | No |
| `read_only` | When `true`, mounts authenticated with this credential reject writes with `EROFS`. | No |
#### Attributes
| Attribute | Description |
| - | - |
| `token` | Server-generated mount token (sensitive, only available at creation). |
| `token_suffix` | Last 4 characters of the token. |
| `identifier` | Stable identifier for this user, used for removal. |
| `expires_at` | Expiration time (computed from `ttl` when set). |
| `consumed_at` | When a one-use credential was consumed. |
| `status` | Current lifecycle status of the credential. |
| `created_at` | Creation timestamp. |
## Data sources
### archil\_disk
Look up an existing Archil disk by ID or name.
```hcl theme={null}
# By name
data "archil_disk" "by_name" {
name = "my-disk"
}
# By ID
data "archil_disk" "by_id" {
id = "dsk-0123456789abcdef"
}
```
#### Arguments
Exactly one of `id` or `name` must be set.
| Argument | Description |
| - | - |
| `id` | Disk ID to look up. |
| `name` | Disk name to look up. |
#### Attributes
| Attribute | Description |
| - | - |
| `id` | Disk ID. |
| `name` | Disk name. |
| `status` | Disk status. |
| `organization` | Owning organization. |
| `provider_name` | Cloud provider. |
| `region` | Disk region. |
| `created_at` | Creation timestamp. |
| `data_size` | Total data size in bytes. |
## Full example
This example creates a disk with an S3 mount authorized by static credentials, a disk user, and an API token. As with the provider's `api_key`, pass real credentials in through variables rather than hardcoding them:
```hcl theme={null}
terraform {
required_providers {
archil = {
source = "archil-data/archil"
}
}
}
provider "archil" {
api_key = var.archil_api_key
region = "aws-us-east-1"
}
variable "archil_api_key" {
type = string
sensitive = true
}
resource "archil_disk" "app" {
name = "my-app-disk"
mount {
type = "s3"
bucket_name = "my-app-bucket"
access_key_id = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
}
resource "archil_disk_user" "app" {
disk_id = archil_disk.app.id
type = "token"
nickname = "app-servers"
}
resource "archil_api_token" "ci" {
name = "ci-token"
description = "Token for CI/CD pipelines"
}
data "archil_disk" "app" {
name = archil_disk.app.name
depends_on = [archil_disk.app]
}
output "disk_id" {
value = archil_disk.app.id
}
output "mount_token" {
value = archil_disk_user.app.token
sensitive = true
}
```
Mount the disk with the generated token:
```bash theme={null}
export ARCHIL_MOUNT_TOKEN=$(terraform output -raw mount_token)
sudo --preserve-env=ARCHIL_MOUNT_TOKEN archil mount my-app-disk /mnt/data --region aws-us-east-1
```
## Import
Disks can be imported using their ID:
```bash theme={null}
terraform import archil_disk.default dsk-0123456789abcdef
```
API tokens are imported with the token value:
```bash theme={null}
terraform import archil_api_token.ci key-YOUR-TOKEN-VALUE
```
# Troubleshooting
Source: https://docs.archil.com/reference/troubleshooting
Solutions to common issues with Archil disks
## Problem: Mount fails with "No such file or directory"
```
✗ Unable to mount Archil disk to '/mnt/archil': No such file or directory (os error 2)
```
The mount point must already exist as an empty directory before running `archil mount`. Create it first:
```bash theme={null}
sudo mkdir -p /mnt/archil
sudo archil mount /mnt/archil --region aws-us-east-1
```
If the directory already exists and you still see this error, confirm you are passing the path exactly as it appears on disk (no typos, no trailing characters) and that a parent directory in the path is not itself a stale Archil mount.
## Problem: Mount fails with permission denied
The Archil client cannot authenticate against the disk. This usually means the [disk token](/concepts/disk-users#disk-token-authorization) is wrong or has been regenerated, or the IAM role is not listed as an authorized disk user.
```bash theme={null}
# Verify your IAM identity
archil utils get-iam-role
```
Compare the output ARN to the **Authorized Users** list on the disk's Details page in the [Archil console](https://console.archil.com). If you are using disk token authentication, regenerate the disk token from the console — Archil does not store disk tokens in plaintext, so a lost token cannot be recovered.
When passing a disk token through sudo, use `sudo --preserve-env=ARCHIL_MOUNT_TOKEN` so the variable is available to the Archil process. Using `sudo` alone drops the environment.
## Problem: Mount hangs or times out
The client cannot reach the Archil control plane. Common causes are a wrong region, a network connectivity issue, or DNS resolution failure.
```bash theme={null}
# Confirm the region matches the disk's region shown in the console
sudo archil mount /mnt/data --region aws-us-east-1
# Test connectivity to the Archil control server
curl -v https://mount.green.us-east-1.aws.prod.archil.com:8100 2>&1 | head -20
```
Ensure your security groups or firewall rules allow outbound traffic on ports 8100 and 5000-5002. If you are running inside a VPC without public internet access, verify that DNS resolution is working and that a NAT gateway or VPC endpoint is configured.
## Problem: Mount fails because the client is no longer supported
If the mount fails with `this version of the Archil client is no longer supported by the server`, your client is older
than the minimum version the server, your organization, or the disk requires. Upgrade the client with `archil upgrade`
or rerun the [installer](/mounting/linux), then retry the mount.
## Problem: Files not appearing in S3
This is expected behavior. Archil provides [eventual synchronization consistency](/details/consistency) between the disk and the backing data source. Writes sync to S3 within a few minutes, not instantly.
```bash theme={null}
# Verify the disk is healthy and accepting writes
archil status /mnt/data
```
## Problem: Another client cannot write to files
Only one Archil client can have write ownership (a "delegation") over any given file or directory at a time. When a client mounts without `--shared`, it checks out the entire root, blocking all other clients from writing. Mounting that same disk from a second location will also fail outright, since the first client holds the root delegation — pass `--shared` on every mount to allow concurrent access.
**Solutions:**
```bash theme={null}
# Option 1: Mount both clients in shared mode and partition by directory
sudo archil mount /mnt/data --region aws-us-east-1 --shared
archil checkout /mnt/data/my-dir/
# Option 2: Use subdirectory mounts so each client owns a different subtree
sudo archil mount :/models /mnt/models --region aws-us-east-1 # Client A
sudo archil mount :/data /mnt/data --region aws-us-east-1 # Client B
# Option 3: Force-claim ownership (use with caution)
archil checkout /mnt/data/contested-dir/ --force
```
`--force` immediately revokes another client's delegation. Any unflushed writes from that client will be lost and subsequent `fsync` calls will return `EIO`.
## Problem: Permission denied when accessing files
This is expected. The Archil FUSE client mounts as root, so all files start with root ownership.
```bash theme={null}
# Change ownership to your user
sudo chown -R $USER:$USER /mnt/data
# Or run commands with sudo
sudo ls /mnt/data
```
## Problem: There's a `.archil` directory at the root of my mount
Every Archil disk has a hidden `.archil/` directory at its root. Archil uses it to hold data that still needs to exist on the disk but is no longer reachable through the normal filesystem — most commonly, files that have been unlinked while a client still has them open. POSIX requires those files to keep working until the last handle is closed, so Archil parks them under `.archil/` until they can be reclaimed.
The directory and its contents synchronize back to your backing S3 bucket along with the rest of the disk, so you'll see a `.archil/` prefix there as well. The exact layout is an implementation detail and may change between releases. Treat it as opaque: don't read from, write to, or delete anything inside `.archil/`, either on the mount or in S3.
## Problem: Data not appearing after writing to S3 directly
Objects created directly in S3 (or another backing store) are subject to [eventual synchronization consistency](/details/consistency). It can take a few minutes for new objects to appear on the Archil disk.
To get faster updates, reduce the directory cache expiry:
```bash theme={null}
# Set cache expiry to 0 seconds for near-real-time visibility
archil set-cache-expiry /mnt/data/incoming --readdir-expiry 0
# Or use a short TTL for a performance/freshness balance
archil set-cache-expiry /mnt/data/incoming --readdir-expiry 5
```
Lower cache expiry values improve freshness but increase the number of requests to the Archil server, which may affect read performance.
## Problem: SDK connection fails
The TypeScript SDK cannot connect to the disk. Check the following:
```typescript theme={null}
const client = await ArchilClient.connect({
region: "aws-us-east-1", // Must match the disk's region
diskName: "myorg/mydisk", // Format: org/disk-name or disk ID
authToken: process.env.ARCHIL_DISK_TOKEN, // Disk token; required outside AWS
});
```
1. **Credentials** — Ensure a valid [disk token](/concepts/disk-users#disk-token-authorization) is set in `ARCHIL_DISK_TOKEN`, or that IAM credentials are available (e.g., running on EC2 with an attached role). Note that a disk token is different from a Control Plane API key.
2. **Disk name** -- Use the owner-qualified format `org/disk-name`, not just the disk name alone.
3. **Region** -- The region string must match exactly (e.g., `aws-us-east-1`, not `us-east-1`).
4. **Node.js version** -- The native module requires Node.js 18 or later.
# Uninstall tools
Source: https://docs.archil.com/reference/uninstall
Remove the Archil client, the disk CLI, and the CSI driver.
This page covers removing each Archil tool:
* **The Archil client**, which mounts disks: [Linux](#linux) and [macOS](#macos)
* **The [`disk` npm package](#disk-npm-package)**, which provides the `disk` and `sandbox` CLIs
* **The [CSI driver](#kubernetes)** for Kubernetes
Removing the client works the same way on Linux and macOS:
1. **Unmount every Archil disk.** A clean unmount flushes recently written data to Archil and releases any files you have
checked out. If you remove the client while a disk is still mounted, writes that haven't been flushed yet are lost.
2. **Remove the software.**
3. **Optionally remove local files.** These include saved credentials, so remove them if the machine is being
decommissioned or handed to someone else.
Uninstalling the client doesn't change your disks or the data on them. Disks stay in your account until you delete them
from the [Archil console](https://console.archil.com).
## Linux
The Linux client is distributed as a package named `archil`. It installs two files:
| Path | Purpose |
| - | - |
| `/usr/bin/archil` | The `archil` client |
| `/sbin/mount.archil` | The helper that lets `mount` and `/etc/fstab` mount Archil disks |
List the Archil disks currently mounted on the server:
```bash theme={null}
grep archil /proc/mounts
```
Unmount each one with `archil unmount`, which flushes pending writes before it disconnects:
```bash theme={null}
sudo archil unmount /mnt/data
```
Then confirm that no client processes are still running:
```bash theme={null}
pgrep -a archil
```
If a process is still listed, it's still flushing data. Wait for it to exit, and don't `kill -9` it.
If you [mount disks automatically on startup](/mounting/linux#mount-automatically-on-startup), remove the Archil
entries from `/etc/fstab`. Otherwise, the server tries to mount them at the next boot and fails.
```bash theme={null}
grep -nw archil /etc/fstab # find the entries
sudo nano /etc/fstab # delete them
sudo systemctl daemon-reload # on systemd hosts, drop the generated mount units
```
Also remove any startup scripts, systemd units, or configuration management that runs `archil mount`.
```bash theme={null}
sudo apt remove archil
```
```bash theme={null}
sudo dnf remove archil
```
On older systems, use `sudo yum remove archil`. On systems without either, use `sudo rpm -e archil`.
On Alpine, the installer copies a standalone binary instead of installing a package. Remove it directly:
```bash theme={null}
sudo rm -f /usr/bin/archil
```
The FUSE library that the installer pulled in (`libfuse2`, `fuse-libs`, or `fuse`) is left installed, because other
software may depend on it.
The package manager doesn't remove the files below, even with `apt purge`, because the package didn't create them.
| Path | Contents |
| - | - |
| `/etc/archil/` | Client configuration, including your automatic-upgrade preference |
| `~/.cache/archil/` | Upgrade-check state, created separately for each user who ran `archil`, including `root` |
| `/var/run/archil/` | Control sockets for running mounts. Empty once every disk is unmounted. |
| `/var/log/archil/` | Log files, present only if the client couldn't log to journald |
| Any directory you passed to `--log-dir` | Per-mount log files |
```bash theme={null}
sudo rm -rf /etc/archil /var/run/archil /var/log/archil /root/.cache/archil
rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}/archil" # repeat for each user who ran archil
```
To confirm that the client is gone, run the following command. It should print nothing:
```bash theme={null}
command -v archil mount.archil
```
## macOS
The [macOS client](/mounting/macos) is a menu bar app. The first time it launches, it asks for your administrator
password and installs a mount helper and a `sudoers` rule, so that mounting and unmounting disks doesn't require a
password. Removing the app from `/Applications` doesn't remove these files, so the steps below remove them separately.
| Path | Purpose |
| - | - |
| `/Applications/Archil.app` | The menu bar app and its Finder and file system extensions |
| `/usr/local/bin/archil-mount` | Mount helper |
| `/etc/sudoers.d/archil` | Lets administrators run the mount helper without a password |
| `~/Library/Application Support/Archil/` | Your sign-in session and the tokens for each mounted disk |
Click the Archil icon in the menu bar and choose **Quit**. Quitting unmounts every disk and waits for pending writes
to flush.
To check for any disks that are still mounted, run:
```bash theme={null}
mount | grep archilfs
```
Unmount anything listed with `sudo umount "/Volumes/"`.
Unregister the file system extension, then delete the app:
```bash theme={null}
pluginkit -r /Applications/Archil.app/Contents/Extensions/ArchilMountExtension.appex
sudo rm -rf /Applications/Archil.app
```
Remove the `sudoers` rule first, then the helper it points to:
```bash theme={null}
sudo rm -f /etc/sudoers.d/archil
sudo rm -f /usr/local/bin/archil-mount
```
Don't skip this step. The `sudoers` rule lets administrators run `/usr/local/bin/archil-mount` as root without a
password. Remove the rule even if the helper file is already gone.
These folders hold your Archil sign-in session, the tokens for disks you've mounted, and the app's settings and
caches. Run the following as each macOS user who used Archil:
```bash theme={null}
rm -rf ~/Library/Application\ Support/Archil
rm -rf ~/Library/Group\ Containers/group.com.archil.Archil
rm -rf ~/Library/Containers/com.archil.Archil.Extension
rm -rf ~/Library/Containers/com.archil.Archil.FinderSync
rm -rf ~/Library/Caches/com.archil.Archil ~/Library/Caches/Archil
rm -rf ~/Library/HTTPStorages/com.archil.Archil
defaults delete com.archil.Archil
```
If macOS denies access to the `Containers` or `Group Containers` folders, give Terminal **Full Disk Access** in
**System Settings → Privacy & Security**, or delete the folders from Finder.
The [`disk` CLI](#disk-npm-package) stores its profiles in `~/Library/Application Support/Archil/disk`, so the
first command above removes those too. To keep them, delete only the files directly inside
`~/Library/Application Support/Archil` and leave the `disk` folder in place.
To confirm that the client is gone, open **System Settings → General → Login Items & Extensions → File System
Extensions**. Archil should no longer be listed. If it still appears, log out and log back in.
Deleting local files doesn't revoke the credentials on Archil's side. If the machine was lost or is moving to someone
else, also remove its disk tokens from each disk's users in the [Archil console](https://console.archil.com) or with the
[Remove Disk User API](/api-reference/disk-users/remove-user).
## `disk` npm package
The [`disk` package](/reference/disk-cli) provides two commands, `disk` and `sandbox`. It doesn't install anything
outside npm, so how you remove it depends on how you added it.
If you only run `npx disk`, there's no install to remove. npx keeps a copy in npm's cache, which npm manages
on its own.
```bash theme={null}
npm uninstall -g disk
```
Run this in your project's directory:
```bash theme={null}
npm uninstall disk
```
With other package managers, use `pnpm remove disk`, `yarn remove disk`, or `bun remove disk`.
If you saved credentials with `disk profile create`, they stay on disk after you remove the package. List your
profiles, then delete each one together with its stored API key:
```bash theme={null}
npx disk profile list
npx disk profile delete
```
Or delete the whole configuration directory:
| Platform | Directory |
| - | - |
| Linux | `~/.config/archil/disk`, or `$XDG_CONFIG_HOME/archil/disk` if it's set |
| macOS | `~/Library/Application Support/Archil/disk` |
| Windows | `%APPDATA%\Archil\disk` |
If you set `ARCHIL_DISK_CONFIG_DIR`, profiles are stored in that directory instead.
Deleting a profile only removes the local copy of its API key. To revoke the key itself, delete it in the
[Archil console](https://console.archil.com).
## Kubernetes
The [CSI driver](/reference/csi-driver) runs its own copy of the client inside your cluster, so the server steps above
don't apply. First delete the pods and PersistentVolumeClaims that use Archil volumes. Then uninstall the Helm release:
```bash theme={null}
helm uninstall archil-csi-driver --namespace archil-system
```
# Verify Linux downloads
Source: https://docs.archil.com/reference/verify-linux-downloads
Verify the authenticity and integrity of an Archil Linux release before installing it.
Archil publishes a signed SHA-256 checksum manifest for every signed Linux release. Use it to verify a downloaded DEB, RPM, or musl binary **before** installing or running it.
This procedure verifies that an artifact matches the release signed by Archil. It does not replace your normal review of the release version.
## Normal install or manual verification?
For a normal installation, use the convenience installer:
```bash theme={null}
curl -fsSL https://archil.com/install | sh
```
The convenience installer is the standard installation path. If you want to verify a versioned artifact before executing it, follow the manual OpenPGP signature and checksum procedure below.
## What you need
* `curl`
* `gpg` (GnuPG)
* `sha256sum` (supplied by GNU coreutils or BusyBox)
The commands below use a temporary GnuPG keyring and do not change your normal GnuPG configuration.
## Trust Archil's release signing key
Archil's production Linux release signing-subkey fingerprint is:
```text theme={null}
075082A64956DDB6281A08A827A9B35E790F664A
```
Before trusting a downloaded certificate, confirm that its fingerprint matches the value above. This documentation page is independent from the S3 release bucket.
## Archil Linux release public certificate
The following is Archil's public release certificate. Confirm that its fingerprint matches the value above before using it to verify a release.
```text theme={null}
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGpZOQ8BEADCz/oM30v+wqE0AJBo3zrPG29s5WmAt6iZA+2MC1JMN60EPlfV
TE5rBNvwJ6LbR1OaWNPVrwAvcPAHj2n1+bQI0ACuh7iRrcLh5670qLxluod1vJO8
+//8RzzSV4x+SEU6liYX+ncR0OlT1OyeJovQOmtc+yR2iIKf5tnUOww6DrO/JjYY
QRG5M27eGHl4n3cZUbzfC46Arup2oFI4hMYXcO21EKwzVxAOX1V9193/1hvqSeCy
BMT2F8U/dz7sHDQl3uutkSi71QmVc6y3FrcZGtF8BPXOOKqA/Y1CXCgEymmzIMy9
S8sxX/9Zmw6d1P0sOiz4G6vgGEFXS0d/lf/Qkmv6NewHCDFFbrWm3PnkgBF6weKA
irSSrFxcaPR4mrfXJaSw6ZYznfLDsIFQqd8ixRjQO64jhrNgk3QxEQ5ix/P7krE0
0WoIhZWNoUcM3GuWC5cEFDIop/Y76+XQRvvCzm4xBVnLiRAeQ+emvSmFDPmMZumn
lOJKvCHNX5UhJM9BgpsRdUhYIYsMSSzlf18UfyPFROIPkgGaZAiz9EgtROtKeKNS
xQyrfz9fVVJZNhXQ20GxYQBlv1KiLCoolVCW5+IRxS1kQjd8LvMdB1ANE26KSBo5
O7EozzjbU1CwLerJkX41sBD9tBQWCcmEJfSc9hcPnkBHbxZPK0WYw2SfNwARAQAB
tCtBcmNoaWwgTGludXggUmVsZWFzZXMgPHNlY3VyaXR5QGFyY2hpbC5jb20+iQJz
BBMBCABdFiEEexRMpLwXJLDlcvF7+t6vSq4R2tAFAmpZOQ8bFIAAAAAABAAObWFu
dTIsMi41KzEuMTIsMCwzAhsBBQkB4TOABQsJCAcCAiICBhUKCQgLAgQWAgMBAh4H
AheAAAoJEPrer0quEdrQhA0QALdizU5X0v/2/V2m8NdrIW7sI3/J/w6UdRzuTkGT
u8Q1rK3oVHcQUmETlnqCw3YFq632cZdBtl1CPIbwIx8sn+Hz0d3nFji/9PjK+nLD
FJXzHaD5hiLZ9QPH1U20jU5ogXx2cWUuI5geGGcoQLJOGyt1tjnCRCAaSD+0qb0c
vkXvP5dxgpSY3WRyhH/LOEf0ZEsSjwiQkmpf/Bl7fF6gBjpuKRJHqRN32ui6pDBw
dO24WXO422jBUQvjWVABnErAhsF9z0IF/4WQmy1j5cluhxLGuwKu/RIl6M+U/c2B
BKfDs1lH0vB+mlyg/Wvsl0QCqpldHNT5mMBgXbwDJeSn+WQYTTvur2IXOcxbwv+k
hBVCyiFufpkT88FTLLjKCZA/tiu9UWZEiTOG9C+fJV1zo0yqLTw+XvnVm/kMgixs
vIKams+HWGH641Fd+M3NbFTvhWm30SmEYXAJ7kQUGSvDPKCxHNCrxzxSAdWVWOFw
XiULzcyRPi0+/Hgccr3aFTiN+fRTfmiAVawuyobkSzwhSXsSAFffyegZ70heKd4E
dOc5f88X9yr7yM9u92meNtK6Y4jZ0JYMmPaUJzFFxIDwxiqlch48805cfwvOYiA6
Ru5pDoxvzwWBMUjvQryqGESl8vKd4As21H5FUrPLVxE8MGDdDzphe571hxPL5yKO
pAHuuQINBGpZOTwBEAC0kVgelpyJGSB7/+QRiZuS8pOgGBKZZw2XWaoxqWADky9S
tjTEMQsa4PoGWsmaGyiDoY8T4vvqQbHXYaDctVpWP8cvt1kdg9mVGSougyzr7sbF
1YyMUpINIEpSwFLauhzHz/SatPrM4XPbulkm+AHpUcOsZdm2hy55tYCQkwtDiZB/
0Oi/qbNYHV9f96fpJePgfU0Jg2wC55r2Mhi70SsF77lJV0hGbwNmIgIqBjs/6eDJ
q34VSS2A7TFeIyCRyyXfqCNX7cmMNCuxORgygzAJCCaTWzf2MAgSszU1Vm5mR4We
7l4c0zlZVQ2ajG7R2u1gU9Kc+JqBpnhyxsrihUoNAa4a96sYY+xxmc4zZd1w4GNg
fQ7OxP+Mj09Tn3QCcDWJ7QiBvwq8C5VwWqmJjvZm133xb2U7LUaqHbjZ6xibl4rs
qaywFfn+eOllbFi2MI3Ik/ysVj8V0j9TOPrhq6PGo3rIpQe0qMovVa5DgM1n2uYU
YxwJPMGkSnwFOQ4PkrucMra5Kyl2A5VObRvKljnBtvdUHgpTzTPhP/FNUNAWUZHu
s1v3QZRlCElWlXL9YS9wJ6Omy+5mWxAdv/OrGQxzkiWIwpVdU4e/pXEy+F8u52l+
dJ6APlARCnIb2P1ts6q78zsxTAIDvdIwYvUuIGDYVcdjP0ol0CSdgllErMF/ZwAR
AQABiQSOBBgBCABCFiEEexRMpLwXJLDlcvF7+t6vSq4R2tAFAmpZOTwbFIAAAAAA
BAAObWFudTIsMi41KzEuMTIsMCwzAhsCBQkB4TOAAkAJEPrer0quEdrQwXQgBBkB
CAAdFiEEB1CCpklW3bYoGgioJ6mzXnkPZkoFAmpZOTwACgkQJ6mzXnkPZkphfA//
Yz2BxmKU0zZp5pWVarHLfPxMbcVL0qFMdEvEPfhKA7B6eBAoygwFKjEuSD5jLHKE
E0k0Td6XtvVcaC9NQCskkRFZtDxlSz+eY1fJRrFaCwiwnlxT8fsd89nsvMhwWn0Q
bi7DMvf0aAK9cpG2FeNwcd9i4RdyPwRGCAsDj1UA6bEMoDL/eaT2vImloSlGk0BL
bGsHpj5PJ1kLj4yGq27z3Gh+uTukWbmJuvy8vxlMolpMpXCdBFgyvIGAlXy9q3Hl
GeziPkftVQ8oCW9bmUiZVaYnhLJgoxq8gH0HQI4NA9+JaUg0QV5ylE1EyUk9NDWI
sKkF98ia4hpcUHxs1JjEo4vpJHolFiVxcZbARIDCyoO3wIUkWH8FE7xlhSsFRSx4
+IpAYfCRRdG2Za24y9j4CcM2O0LXni+/8LvNt4D56nfB5T4m33Ds4nFV0Sr55ZbU
0RdJFSmxO0OHGjt3qDTfvYabdtv3xyoIL1XhZ4Yk2xLUXONs5UyphdenAvbXipkW
LYBRQOnINhFI7K3bBBibjKw5Z651AW8x1e+eAgOTZddF/cQwF2bwEoJfDCvF2ctr
3h0fjJD3Awt1oNG2EWKHTFKH82ZYjdfoiQnUYJO/f8iXj+M3WcuZghhKMz5XJLv1
Eu/oRuW9k25lqm/6Og3tB6mFFQgwYEC1PLc/xFiu8mYieRAAi3nlcDnXpv/TAL1a
Pa49wT8KEBWp/rgqzbEjieZq2IZn1XmgcIaR4e4Ei7WmIcZR7Tgzq+0mjodUqCb4
g0VJXrwyG2r9tBhCfn5BYcABfpUC2cLqVQWRBO623njV0GHZk/UXwimk/bTJpKCY
0+JJ+QzQ3HeAxWDZGRCI7TI+CxH/8hGtp+TxtbvGwLkL4RvJBlLXXHTY9IXBiLmT
9Fln2YgwY9AFKLaCD9A6UrPq/QuImgUTLJCPfRbu2oLnjuolP0SiMard/nF20BFz
4W7DVleNo0zm56n+QY6kMvkLLUH9aLPrDi/iIahN1kKztfK5XCYDEzPqTmgYoVlU
p772+AA33toxxj7tavf8yACSNVUicXUA5wP2+xoDi/3vOz1ORrgJpZdSqlU9+fzq
NVYippYkslYf1bGQdLpajQunOZpx37vyDqVliE1N9/djPn0SvTdYd/tfg0E+WAMH
g0L4kPlzkPcswZVrNqxlcA0dsqV61UtFSGXmNfLd5Plg+OxtWoh+iWqnQ3QJ/EsW
iymXJb12hpj4RPvIV2vgllRytvfBjct78D7sTYflEOUKTV0pStg6IhiTm2LxJm60
bOXxYVKzhuDRBM8UP3KlShryLTLntPof7hHsTHsB15DlCsEIMfTTTuKIRoHSf3dU
YNtC4jeY0l42wAa6kB9lPmUVAHI=
=pX2j
-----END PGP PUBLIC KEY BLOCK-----
```
## Choose a release and artifact
Choose the exact version and filename you intend to install. Signed Linux artifacts and their verification metadata are published under `https://s3.amazonaws.com/archil-client/pkg/`.
To select the version currently used by the production install channel, read its `latest` pointer:
```bash theme={null}
curl --fail --location https://s3.amazonaws.com/archil-client/pkg/latest
```
The `latest` pointer is mutable and is **not** authenticated. Use it only to choose a version to download; the signature and checksum steps below verify the artifact. For a reproducible installation, record an explicit version instead of relying on `latest`.
For example, if the release includes `archil_0.8.21-123_amd64.deb`, use `0.8.21-123` as `VERSION`:
```bash theme={null}
VERSION="0.8.21-123"
BASE_URL="https://s3.amazonaws.com/archil-client/pkg"
ARTIFACT="archil_${VERSION}_amd64.deb"
```
Use the exact filename listed in the release. The common artifact names are:
| Platform | Artifact name |
| - | - |
| Debian/Ubuntu AMD64 | `archil__amd64.deb` |
| Debian/Ubuntu ARM64 | `archil__arm64.deb` |
| RPM AMD64 | `archil-.x86_64.rpm` |
| RPM ARM64 | `archil-.aarch64.rpm` |
| musl AMD64 | `archil-linux-musl-amd64-` |
| musl ARM64 | `archil-linux-musl-arm64-` |
For RPMs, use the exact filename from the release. Pre-release versions can have a different RPM-safe version string.
## Download the artifact and verification files
Create a clean working directory and download the package, checksum manifest, detached signature, and public certificate for the **same version**:
```bash theme={null}
mkdir "archil-verify-${VERSION}"
cd "archil-verify-${VERSION}"
curl --fail-early --fail --location --remote-name-all \
"$BASE_URL/$ARTIFACT" \
"$BASE_URL/archil_${VERSION}_SHA256SUMS" \
"$BASE_URL/archil_${VERSION}_SHA256SUMS.sig" \
"$BASE_URL/archil_${VERSION}_linux_release_signing_key.asc"
```
Each signed release includes these verification files:
| File | Purpose |
| - | - |
| `archil__SHA256SUMS` | SHA-256 checksums for the six Linux artifacts. |
| `archil__SHA256SUMS.sig` | ASCII-armored detached OpenPGP signature for the checksum manifest. |
| `archil__linux_release_signing_key.asc` | Public certificate for the Linux release signing key. |
If any download fails, stop. Do not install an artifact when its checksum manifest, signature, or certificate is unavailable, and do not mix files from different versions or release channels. If the verification files for a release are unavailable, contact [support@archil.com](mailto:support@archil.com).
## Verify the public key and checksum signature
Import the downloaded certificate into a temporary keyring. The first check confirms that the certificate contains Archil's expected signing-subkey fingerprint; the second confirms that this exact key signed the checksum manifest.
```bash theme={null}
EXPECTED_FINGERPRINT="075082A64956DDB6281A08A827A9B35E790F664A"
export GNUPGHOME="$(mktemp -d)"
trap 'rm -rf "$GNUPGHOME"' EXIT
chmod 700 "$GNUPGHOME"
gpg --batch --import "archil_${VERSION}_linux_release_signing_key.asc"
if ! gpg --batch --with-colons --fingerprint --list-keys \
| awk -F: -v expected="$EXPECTED_FINGERPRINT" \
'$1 == "fpr" && toupper($10) == toupper(expected) { found = 1 } END { exit !found }'; then
echo "The downloaded certificate does not contain Archil's expected signing fingerprint." >&2
exit 1
fi
if ! gpg --batch --status-fd 1 --verify \
"archil_${VERSION}_SHA256SUMS.sig" \
"archil_${VERSION}_SHA256SUMS" 2>/dev/null \
| awk -v expected="$EXPECTED_FINGERPRINT" \
'$1 == "[GNUPG:]" && $2 == "VALIDSIG" && toupper($3) == toupper(expected) { found = 1 } END { exit !found }'; then
echo "The checksum manifest did not pass Archil's expected signature validation. We recommend not installing the artifact." >&2
exit 1
fi
```
If either check fails, do not install or run the artifact. Delete the downloaded files and obtain them again only from the canonical release location.
## Verify the downloaded artifact
Extract the one expected checksum entry, then verify your downloaded artifact against it:
```bash theme={null}
if ! CHECKSUM_LINE="$(awk -v artifact="$ARTIFACT" \
'$2 == artifact { count++; line = $0 } END { if (count != 1) exit 1; print line }' \
"archil_${VERSION}_SHA256SUMS")"; then
echo "Expected exactly one checksum entry for $ARTIFACT." >&2
exit 1
fi
if ! printf '%s\n' "$CHECKSUM_LINE" | sha256sum -c -; then
echo "Checksum verification failed. Do not install or run $ARTIFACT." >&2
exit 1
fi
```
The command must print:
```text theme={null}
: OK
```
If it prints `FAILED`, reports that the artifact is missing from the manifest, or produces no `OK` line, do not install or run the file.
## Install after verification
Run only the command applicable to your verified artifact:
```bash theme={null}
# Debian/Ubuntu
sudo apt install ./$ARTIFACT
# RPM-based distributions
sudo dnf install ./$ARTIFACT
# musl binary, for example Alpine
sudo install -m 0755 "$ARTIFACT" /usr/local/bin/archil
```
## Clean up
Remove the temporary keyring when you are finished:
```bash theme={null}
rm -rf "$GNUPGHOME"
unset GNUPGHOME EXPECTED_FINGERPRINT
```
## Key rotation and security notices
Before a planned signing-key rotation, Archil publishes the replacement fingerprint and retains the prior fingerprint during a documented overlap. If you suspect a signing key has been compromised, do not install new artifacts until Archil publishes a security notice and replacement verification instructions through its documentation. Report suspected security issues to [security@archil.com](mailto:security@archil.com).
## Scope
This page covers versioned, signed Linux releases only. It does not cover macOS, Windows, package-repository signing, Sigstore or Cosign attestations, or the unversioned `https://archil.com/install` script.
# Python SDK
Source: https://docs.archil.com/sdks/python
Programmatic access to Archil disks from Python scripts, CI, and notebooks
The Archil Python SDK is the [`archil`](https://pypi.org/project/archil/) package on PyPI. It's a pure-Python control-plane client: create disks, list and inspect them, manage who can mount them, run commands against them with [`Disk.exec`](/compute/serverless-execution), read and write their contents through an S3-compatible object API, and manage [persistent sandboxes](#persistent-sandboxes).
```bash theme={null}
pip install archil
```
`archil` talks to the Archil control plane over HTTPS and has no native dependencies. It requires Python 3.10 or later.
Every method works **both synchronously and asynchronously** from a single implementation: `disk.put_object(...)` blocks, while `disk.put_object.aio(...)` returns a coroutine you can `await`. See [Async](#async) below.
Looking for the FUSE mount client? That's the [`archil` CLI](/mounting/linux), which mounts a disk as a real local filesystem. The Python SDK does not mount disks; it talks to the control plane, runs serverless commands, and reads and writes objects over HTTPS.
## Configuration
The recommended pattern is a one-time `configure` call, then the module-level helpers:
```python theme={null}
import archil
archil.configure(api_key="key-...", region="aws-us-east-1")
```
Both options fall back to environment variables (`ARCHIL_API_KEY`, `ARCHIL_REGION`) if omitted, so in most environments you can skip `configure` entirely and let the SDK read the environment.
For multi-tenant scripts that need multiple credentials in one process, instantiate `Archil` directly instead of using the module-level `configure`:
```python theme={null}
from archil import Archil
prod = Archil(api_key=prod_key, region="aws-us-east-1")
staging = Archil(api_key=staging_key, region="aws-us-east-1")
prod_disks = prod.disks.list()
staging_disks = staging.disks.list()
```
The API key is an account-level credential and is **not** the same thing as a disk token. API keys authenticate calls to the control plane (everything in this page); a disk token grants mount access to a single disk. See the [disk users concept page](/concepts/disk-users#disk-token-authorization).
## Managing disks
```python theme={null}
import archil
# Create a disk. The returned disk is available for use immediately. `token`
# here is the disk token — the one-time credential for mounting. Save it; it
# isn't retrievable again.
result = archil.create_disk(name="my-disk")
disk = result.disk
print(f"Created {disk.id}, disk token: {result.token}")
# List and look up disks
all_disks = archil.list_disks()
d = archil.get_disk(disk.id)
```
Per-disk operations are methods on the `Disk` object itself, not top-level functions:
```python theme={null}
from archil import TokenUser
d = archil.get_disk("dsk-abc123")
# Add an additional mount token — save user.token; it's only returned once
user = d.add_user(TokenUser(nickname="ci"))
# Revoke access
d.remove_user("token", user.identifier)
# Delete the disk (this does not delete data in your bucket)
d.delete()
```
## Executing commands
`Disk.exec(command)` runs a shell command inside a container with the file system already mounted, and returns stdout, stderr, exit code, and timing. See the [Serverless Execution concept page](/compute/serverless-execution) for the full picture.
```python theme={null}
d = archil.get_disk("dsk-abc123")
res = d.exec("grep -r ERROR logs")
print(res.stdout, res.stderr, res.exit_code)
print(f"ran in {res.timing.execute_ms}ms (queued {res.timing.queue_ms}ms)")
```
The disk is the working directory inside the container — commands can reference files using relative paths.
Billing is based on `execute_ms` — the wall-clock time your command runs — in 1ms increments, with a 100ms minimum per call. Queue time is not billed. stdout and stderr are each capped at 128 KiB per invocation — pipe larger outputs to a file on the disk instead.
For multi-disk execs (mount several disks at once, optionally pinned to a subdirectory or read-only), call `archil.exec(...)` instead of `Disk.exec`. Each disk is mounted at its own relative path; pass a `Disk`, a disk-id string, or an `ExecMountSpec` for finer control:
```python theme={null}
from archil import ExecMountSpec
res = archil.exec(
disks={
"data": "dsk-abc123", # mount root, read-write
"ref": ExecMountSpec("dsk-def456", read_only=True), # read-only
"logs": ExecMountSpec("dsk-ghi789", subdirectory="2026/01"),
},
command="grep -r ERROR data ref logs",
)
```
See the [bash tool for agents guide](/guides/ai/bash-tool) for an end-to-end example wiring `exec` into an AI agent loop.
## Searching files
`Disk.grep(...)` searches the files on a disk for lines matching a regular expression, fanning the listing and matching out across many ephemeral containers so the search scales across many machines instead of one. Reach for it instead of `exec("grep …")` whenever you just want matching lines. See [Search Files](/compute/search-files) for the full model.
```python theme={null}
d = archil.get_disk("dsk-abc123")
res = d.grep(directory="logs", pattern="ERROR|FATAL", recursive=True)
for m in res.matches:
print(f"{m.file}:{m.line}: {m.text}")
print(f"{len(res.matches)} matches in {res.duration_ms}ms ({res.stopped_reason})")
```
You control cost and latency with three knobs:
* `max_duration_seconds` — wall-clock deadline (default 30, capped at 30).
* `concurrency` — max parallel workers (default 50). More workers scan a large dataset faster, at proportionally more compute.
* `max_results` — short-circuit once this many matches are collected (default 1000).
Always check `stopped_reason` — it tells you whether the search was exhaustive. When it stops early, the returned `matches` are a sample of whichever workers reported first, **not** the lexicographically first N:
| `stopped_reason` | Meaning |
| - | - |
| `completed` | Every file under the directory was scanned successfully. The matches are exhaustive. |
| `max_results` | Stopped after collecting `max_results` matches before scanning everything. |
| `deadline` | Hit `max_duration_seconds` before scanning everything. |
| `incomplete` | The pipeline finished but one or more batches errored (invalid regex, unreadable file). Results may be partial. |
| `list_failed` | Directory listing failed; only partial results, if any, are present. |
## Persistent sandboxes
`archil.create_sandbox()` creates a persistent Linux VM. See
[Persistent sandboxes](/compute/sandboxes/introduction) for the lifecycle, timeouts, networking, and limits. The examples
below use `archil` 0.16.0 or later.
```python theme={null}
import archil
sandbox = archil.create_sandbox(
name="agent-workspace",
base_image="python:3.13",
vcpu_count=2,
mem_size_mib=4096,
max_ttl_seconds=3600,
idle_ttl_seconds=300,
)
result = sandbox.exec("python --version && pwd")
print(result.stdout)
sandbox = sandbox.pause()
sandbox = sandbox.resume()
fork = sandbox.fork(name="agent-task")
sandbox.set_timeout(7200)
sandbox.set_timeout(idle_ttl_seconds=0)
```
`start`, `stop`, `pause`, `resume`, and `refresh` return a new `Sandbox` with the updated state rather than modifying
the one you call them on. They wait for completion by default; pass `wait=False` to return once the operation is
accepted. `set_timeout(timeout, idle_ttl_seconds=...)` changes either timeout, and omitted settings stay unchanged.
A session cannot exceed 24 hours even after a timeout reset; starting or resuming begins a new session.
`fork` pauses a running sandbox while the snapshot is taken and resumes it once the fork is accepted.
`run()` starts a process and returns its handle; `exec()` waits for the command result. Pass an absolute `cwd`
to either method to choose a working directory. `attach()` reconnects by process ID and output cursor.
The older `processes.start()` and `processes.connect()` methods are deprecated compatibility aliases.
```python theme={null}
process = sandbox.run("sleep 600", timeout_seconds=600)
process_id, cursor = process.id, process.cursor
process.disconnect()
resumed = sandbox.attach(process_id, offset=cursor)
resumed.kill()
```
Expose ports, create port tokens, change the network policy, and transfer files with the same methods as the
TypeScript SDK, in snake case:
```python theme={null}
hostname = sandbox.expose_port(8080)
access = sandbox.create_port_token(3000, ttl="1h") # send access.token in X-Archil-Token
sandbox.update_network(
archil.SandboxNetwork(egress=archil.SandboxEgressPolicy(default="deny", allow=["api.github.com"]))
)
sandbox.files.upload_file("./input.tar.gz", "/workspace/input.tar.gz")
sandbox.files.download_file("/workspace/result.json", "./result.json")
```
### Mounting existing disks
```python theme={null}
workspace = archil.create_sandbox(
mounts=[archil.SandboxMountSpec(disk="dsk-0123456789abcdef", path="/mnt/data")]
)
print(workspace.exec("ls /mnt/data").stdout)
workspace = workspace.stop()
workspace = workspace.start(mounts=[])
```
`SandboxMountSpec` accepts `disk` (a disk ID or `Disk` object), `path`, `subdirectory`, `read_only`, `conditional`, and
`queue_ms`. With a single mount, `path` is optional and defaults to `/mnt/archil`.
Omitting `mounts` on `start()` keeps saved mounts; `[]` clears them and a list replaces them. See
[Persistent sandboxes](/compute/sandboxes/introduction#attaching-an-archil-disk) for sharing and forking behavior.
### Private images
```python theme={null}
import os
with archil.Archil() as client:
image = client.images.build(
source="ghcr.io/acme/agent-runtime:v2",
registry_auth=archil.RegistryAuth(username="bot", password=os.environ["REGISTRY_TOKEN"]),
)
agent = client.sandboxes.create(image_id=image.image_id)
```
`images.build()` waits until the image is ready, retrying interrupted builds up to three times. The default timeout
is 30 minutes; pass `timeout=600` to change it. Failed builds raise `ImageBuildError`; timeouts raise `TimeoutError`.
Use `images.get(image_id)` to read the current build state.
Pass `image_id` instead of `base_image` when creating a sandbox from a built image.
See [Sandbox images](/compute/sandboxes/images) for reusing builds and updating tags.
### Waiting for capacity
Sandbox create, start, resume, and fork retry capacity responses (`503` with `no_capacity` or `runtime_retryable`,
and `429`) for up to 10 seconds by default, honoring `Retry-After`. Configure `Archil(max_throttle_wait=seconds)`
or `ARCHIL_MAX_THROTTLE_WAIT` to change that budget; `0` fails immediately and `float("inf")` waits indefinitely.
Forking a running sandbox does not wait for capacity, so its source can be resumed promptly.
## Reading and writing objects
A `Disk` doubles as an S3-compatible bucket: read, write, delete, and list its files by key without mounting it. These methods talk to Archil's S3 endpoint using your same API key — no separate S3 credentials or SigV4 signing on your part.
```python theme={null}
import json
d = archil.get_disk("dsk-abc123")
report = {"generated": "2026-01", "rows": 1234}
# Write — accepts str or bytes. content_type is optional. Returns the etag.
result = d.put_object("reports/2026-01/data.json", json.dumps(report), "application/json")
# Read — returns bytes.
data = d.get_object("reports/2026-01/data.json")
text = data.decode("utf-8")
# Metadata / existence without downloading the body
meta = d.head_object("reports/2026-01/data.json") # None if absent
if d.object_exists("reports/2026-01/data.json"):
...
# Delete (idempotent — deleting a missing key succeeds)
d.delete_object("reports/2026-01/data.json")
```
`put_object` handles any size with one call: small bodies go through a single request, and larger bodies are uploaded as a multipart upload automatically — split into parts, uploaded with bounded concurrency, and assembled, aborting the upload if any part fails so nothing is left half-staged. Tune the switch point and parallelism with keyword options:
```python theme={null}
result = d.put_object(
"backups/2026-01.tar",
big_bytes,
content_type="application/x-tar",
multipart_threshold=5 * 1024 * 1024, # switch to multipart above 5 MiB; default = part_size
part_size=32 * 1024 * 1024, # >= 5 MiB; default 16 MiB
concurrency=8, # parts in flight at once; default 4
)
```
For very large objects the part size grows automatically so the upload never exceeds S3's 10,000-part limit.
`list_objects` auto-paginates by default, returning every matching key. The first argument is a key prefix; a non-recursive listing (the default) returns the immediate level as `objects` plus subdirectory `common_prefixes`:
```python theme={null}
result = d.list_objects("reports/") # one level
all_keys = d.list_objects("reports/", recursive=True) # whole subtree
first_100 = d.list_objects("reports/", limit=100) # cap the total
# Stream pages instead of buffering everything (large listings):
for page in d.list_objects_pages("reports/"):
for obj in page.objects:
print(obj.key, obj.size, obj.last_modified)
# Or drive pagination yourself:
page = d.list_objects("reports/", single_page=True)
if page.is_truncated:
nxt = d.list_objects("reports/", single_page=True, continuation_token=page.next_continuation_token)
```
`delete_objects` removes many keys in one round trip (auto-batched at S3's 1,000-key limit). Unlike `delete_object`, per-key failures are returned rather than raised:
```python theme={null}
result = d.delete_objects(["a.txt", "logs/b.txt", "c.txt"])
for e in result.errors:
print(f"{e.key}: {e.code} {e.message}")
```
`append_object` appends bytes to an existing object (creating it if absent) — handy for log-style writes. Each call may append at most 1 MiB; append in chunks to grow past that:
```python theme={null}
d.append_object("logs/app.log", b"first line\n")
d.append_object("logs/app.log", b"second line\n") # concatenated
```
For manual control over the multipart lifecycle (e.g. uploading parts from separate processes), the raw S3 primitives live in the opt-in `d.multipart` namespace — `create`, `upload_part`, `complete`, `abort`, `list_parts`, `list_uploads`. Most code never needs these; prefer `put_object`, which runs the lifecycle for you.
```python theme={null}
upload = d.multipart.create("big.bin")
p1 = d.multipart.upload_part("big.bin", upload.upload_id, 1, first_chunk)
p2 = d.multipart.upload_part("big.bin", upload.upload_id, 2, second_chunk)
d.multipart.complete("big.bin", upload.upload_id, [p1, p2])
```
## Share URLs
`share` generates a signed, time-limited URL that lets anyone download a file from the disk without credentials:
```python theme={null}
share = d.share("reports/2026-08/summary.csv")
print(share.url) # https://control.…/api/shared/
print(share.expires_in) # 86400 (seconds)
# Custom expiry — any value from 1 to 604800 (7 days)
share = d.share("reports/2026-08/summary.csv", expires_in=3600)
```
The returned URL supports HTTP `Range` requests and streams with a sensible `Content-Type`. After expiry, requests to the URL return `401`.
## Async
Every method on `Archil`, `Disks`, `Disk`, `Tokens`, `Sandboxes`, `Sandbox`, and sandbox processes and files has an `.aio` variant that returns a coroutine. The module-level helpers (`configure`, `create_disk`, `get_disk`, etc.) are synchronous convenience wrappers, so from async code, construct `Archil(...)` directly and use `.aio`:
```python theme={null}
import asyncio
from archil import Archil
async def main():
async with Archil(api_key="key-...", region="aws-us-east-1") as client:
d = await client.disks.get.aio("dsk-abc123")
await d.put_object.aio("a/b.txt", b"hello")
data = await d.get_object.aio("a/b.txt")
async for page in d.list_objects_pages.aio("a/"):
for obj in page.objects:
print(obj.key)
asyncio.run(main())
```
## Managing API keys
API keys are account-level, so these helpers live at the top level rather than on a `Disk`:
```python theme={null}
archil.list_api_keys()
k = archil.create_api_key(name="ci-bot", description="GitHub Actions")
# save k.token — it's only returned once
archil.delete_api_key("key-abc123")
```
## Error handling
All SDK errors extend `ArchilError`, so `except ArchilError` handles control-plane and S3 failures uniformly. Object-API failures raise `ArchilS3Error` with `status` (HTTP status), `code` (the S3 error code, e.g. `"NoSuchKey"`), `request_id`, and the raw body on `raw`:
```python theme={null}
from archil import ArchilError, ArchilS3Error
try:
data = d.get_object("reports/missing.json")
except ArchilS3Error as e:
print(e.status, e.code, e.request_id)
```
`get_object` on a missing key raises a 404 — use `head_object` / `object_exists` to probe without catching.
Transient failures (HTTP 429 and 5xx, plus network errors) are retried automatically with jittered exponential backoff before surfacing; other 4xx are caller errors and aren't retried. The two non-idempotent operations — `complete` (multipart) and `append_object` — are not auto-retried, since a retry after a succeeded-but-unacknowledged call would return a spurious `NoSuchUpload` or duplicate the appended bytes.
## Supported regions
| Region | Provider |
| - | - |
| `aws-us-east-1` | AWS |
| `aws-us-west-2` | AWS |
| `aws-eu-west-1` | AWS |
| `gcp-us-central1` | GCP |
# TypeScript SDK
Source: https://docs.archil.com/sdks/typescript
Programmatic access to Archil disks from TypeScript and Node.js applications
The Archil TypeScript SDK is the [`disk`](https://www.npmjs.com/package/disk) package on npm. It's a pure-JavaScript control-plane client and CLI: create disks, list and inspect them, manage who can mount them, and run commands against them with [`disk.exec`](/compute/serverless-execution).
```bash theme={null}
npm install disk
```
`disk` has no native dependencies and requires Node.js 22.19 or later. For the rare case where you need raw protocol access from Node (inodes, delegations, byte-level reads), there's a separate [`@archildata/native`](#low-level-protocol-access) package — most users will not need it.
| Package | When to use |
| - | - |
| [`disk`](https://www.npmjs.com/package/disk) | Manage disks, run commands with `disk.exec`, manage API keys. The default. |
| [`@archildata/native`](https://www.npmjs.com/package/@archildata/native) | Speak the Archil filesystem protocol directly from Node. Linux/macOS only. Not recommended unless you're building a custom client runtime. |
Looking for the FUSE mount client? That's the [`archil` CLI](/mounting/linux), which mounts a disk as a real local filesystem. The TypeScript SDK does not mount disks; it talks to the control plane and runs serverless commands.
## Configuration
The recommended pattern is a module-namespace import with a one-time `configure` call:
```typescript theme={null}
import * as archil from "disk";
archil.configure({
apiKey: process.env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
```
Both options fall back to environment variables (`ARCHIL_API_KEY`, `ARCHIL_REGION`) if omitted, so in most environments `configure({})` — or skipping `configure` entirely — is enough.
For multi-tenant scripts that need multiple credentials in one process, instantiate `Archil` directly instead:
```typescript theme={null}
import { Archil } from "disk";
const prod = new Archil({ apiKey: prodKey, region: "aws-us-east-1" });
const staging = new Archil({ apiKey: stagingKey, region: "aws-us-east-1" });
```
For endpoints signed by a private certificate authority, pass `tls.ca` (a PEM string, `Buffer`, or array of either) to
`new Archil({ ..., tls: { ca } })`. Like Node's TLS `ca` option, it replaces the default trust list, so include
`getCACertificates()` from `node:tls` if you still need the public CAs.
The API key is an account-level credential and is **not** the same thing as a disk token. API keys authenticate calls to the control plane (everything in this page); a disk token grants mount access to a single disk. See the [disk users concept page](/concepts/disk-users#disk-token-authorization).
## Managing disks
```typescript theme={null}
import * as archil from "disk";
// Create a disk. The returned token is the disk token — save it; it isn't
// retrievable again.
const { disk, token } = await archil.createDisk({
name: "my-disk",
mounts: [{
type: "s3",
bucketName: "my-bucket",
accessKeyId: "AKIA...",
secretAccessKey: "...",
}],
});
// List and look up disks
const all = await archil.listDisks();
const d = await archil.getDisk(disk.id);
```
Per-disk operations are methods on the `Disk` object itself, not top-level functions:
```typescript theme={null}
const d = await archil.getDisk("dsk-abc123");
// Add an additional mount token
const user = await d.createToken("ci");
// save user.token — it's only returned once
// Revoke access
await d.removeUser("token", user.identifier);
// Delete the disk (this does not delete data in your bucket)
await d.delete();
```
## Executing commands
`Disk.exec(command)` runs a bash command inside a container with the file system already mounted, and returns stdout, stderr, exit code, and timing. See the [Serverless Execution concept page](/compute/serverless-execution) for the full picture.
```typescript theme={null}
const d = await archil.getDisk("dsk-abc123");
const { stdout, stderr, exitCode, timing } = await d.exec(
"du -sh logs",
);
console.log(`ran in ${timing.executeMs}ms (queued ${timing.queueMs}ms)`);
```
The disk is the working directory inside the container — commands can reference files using relative paths.
For multi-disk execs (mount several disks at once, optionally pinned to a subdirectory or read-only), call `archil.exec({ disks, command })` instead of `Disk.exec`. See [Mounting multiple disks in one command](/compute/serverless-execution#mounting-multiple-disks-in-one-command).
### `ExecResult`
```typescript theme={null}
interface ExecResult {
exitCode: number;
stdout: string;
stderr: string;
timing: ExecTiming;
}
interface ExecTiming {
/** End-to-end wall-clock measured on the server. */
totalMs: number;
/** Time spent queueing, scheduling, booting a container, and mounting the filesystem. */
queueMs: number;
/** Time the user's command itself ran, measured by the runtime. */
executeMs: number;
}
```
Billing is based on `executeMs` — the wall-clock time your command runs — in 1ms increments, with a 100ms minimum per call. Queue time is not billed. The HTTP response returns after 5 minutes, and stdout and stderr are each capped at 128 KiB per invocation — pipe larger outputs to a file on the disk instead.
### Fan-out
Because each `exec` runs in its own container, `Promise.all` is a map-reduce:
```typescript theme={null}
const files = (await d.exec("ls logs")).stdout.trim().split("\n");
const results = await Promise.all(
files.map((f) => d.exec(`wc -l logs/${f}`)),
);
```
For the common case of *searching* files for a pattern, don't hand-roll this — use [`Disk.grep`](#searching-files), which fans the same work out for you and returns structured matches. See the [bash tool for agents guide](/guides/ai/bash-tool) for an end-to-end example wiring `disk.exec` into an AI agent loop.
## Searching files
`Disk.grep(opts)` searches the files on a disk for lines matching a regular expression, fanning the listing and matching out across many ephemeral containers so the search scales across many machines instead of one. It's the productized version of the fan-out above — reach for it instead of `Promise.all` over `exec("grep …")` whenever you just want matching lines. See [Search Files](/compute/search-files) for the full model.
```typescript theme={null}
const d = await archil.getDisk("dsk-abc123");
const { matches, stoppedReason, durationMs } = await d.grep({
directory: "logs",
pattern: "ERROR|FATAL",
recursive: true,
});
for (const m of matches) {
console.log(`${m.file}:${m.line}: ${m.text}`);
}
console.log(`${matches.length} matches in ${durationMs}ms (${stoppedReason})`);
```
You control cost and latency with three knobs:
* `maxDurationSeconds` — wall-clock deadline (default 30, capped at 30).
* `concurrency` — max parallel workers (default 50). More workers scan a large dataset faster, at proportionally more compute.
* `maxResults` — short-circuit once this many matches are collected (default 1000).
Always check `stoppedReason` — it tells you whether the search was exhaustive. When it stops early, the returned `matches` are a sample of whichever workers reported first, **not** the lexicographically first N:
| `stoppedReason` | Meaning |
| - | - |
| `completed` | Every file under the directory was scanned successfully. The matches are exhaustive. |
| `max_results` | Stopped after collecting `maxResults` matches before scanning everything. |
| `deadline` | Hit `maxDurationSeconds` before scanning everything. |
| `incomplete` | The pipeline finished but one or more batches errored (invalid regex, unreadable file). Results may be partial. |
| `list_failed` | Directory listing failed; only partial results, if any, are present. |
### `GrepResult`
```typescript theme={null}
interface GrepResult {
matches: GrepMatch[];
stoppedReason: GrepStoppedReason; // "completed" | "incomplete" | "max_results" | "deadline" | "list_failed"
filesScanned: number;
containersDispatched: number;
computeSecondsUsed: number; // ≈ billable container-seconds
durationMs: number; // end-to-end wall clock
listingMs: number; // overlaps grepMs, so the two can sum past durationMs
grepMs: number;
}
interface GrepMatch {
file: string; // relative to the disk root
line: number; // 1-based
text: string; // the matching line
}
```
Grep runs on the same container runtime as `exec`, so it bills the same way — on `computeSecondsUsed`, the summed execution time across the containers it dispatched.
## Persistent sandboxes
`Archil.sandboxes` manages persistent Linux VMs. Each sandbox has a dedicated disk, accepts multiple
commands over its lifetime, and can preserve CPU and memory state across pause and resume. See
[Persistent sandboxes](/compute/sandboxes/introduction) for the lifecycle, interactive shells,
services, forks, and limits.
```typescript theme={null}
import { Archil } from "disk";
const client = new Archil({
apiKey: process.env.ARCHIL_API_KEY,
region: "aws-us-east-1",
});
const sandbox = await client.sandboxes.create({
name: "prepared-environment",
baseImage: "node:24-bookworm",
vcpuCount: 4,
memSizeMiB: 8192,
env: { NODE_ENV: "development" },
maxTtlSeconds: 14_400,
maxConcurrentExecs: 32,
});
const execution = await sandbox.run("node --version");
const result = await execution.wait();
console.log(result.stdout);
await sandbox.pause();
await sandbox.resume();
await sandbox.stop();
await sandbox.start();
const fork = await sandbox.fork({ name: "agent-task" });
await fork.stop();
await fork.delete();
```
`maxTtlSeconds` sets the hard timeout from 60 seconds to 24 hours (the default). `idleTtlSeconds` sets
an optional idle timeout from 0 to 24 hours; 0 disables it. Both pause the VM, preserving memory and
processes for `resume()`. Use `sandbox.setTimeout()` to update either setting; a session cannot exceed 24 hours
even after a timeout reset. See
[Setting a time to live](/compute/sandboxes/introduction#setting-a-time-to-live) for examples and reset behavior.
`create`, `pause`, `resume`, `stop`, `start`, and `fork` wait for completion by default. The
server handles the initial wait; if its request budget expires first, the SDK continues polling.
Pass `{ wait: false }` to return as soon as the operation is accepted. A waited `pause()` throws
`SandboxPauseError` if the sandbox does not reach `paused`. `fork()` pauses a running sandbox while
the snapshot is taken and resumes it once the fork is accepted.
```typescript theme={null}
const running = await sandbox.run("sleep 600", {
timeoutSeconds: 600,
});
const processId = running.id;
const cursor = running.cursor;
await running.disconnect();
const resumed = await sandbox.attach(processId, { offset: cursor });
await resumed.kill();
```
Use `client.sandboxes.list()` and `.get(id)` to retrieve sandboxes. A `Sandbox` provides
`refresh`, `fork`, `delete`, `pause`, `resume`, `stop`, `start`, and `setTimeout`; `exposePort`, `listPorts`,
`unexposePort`, and port-token methods for [network services](/compute/sandboxes/introduction#hosting-network-services);
`getNetwork` and `updateNetwork` for the [firewall](/compute/sandboxes/firewall); plus `run` and `attach` for starting and reattaching to sandbox processes.
`exec` runs a command and waits for its result. Pass an absolute `cwd` to `run` or `exec` to choose a working directory.
The older `processes.start` and `processes.connect` methods are deprecated compatibility aliases.
For an interactive command, provide terminal dimensions. The returned `SandboxProcess` accepts
input and terminal resizes while streaming output to `onOutput`:
```typescript theme={null}
const shell = await sandbox.run("/bin/sh", {
terminal: { cols: 120, rows: 40 },
onOutput: ({ data }) => process.stdout.write(data),
});
await shell.sendInput("pwd\n");
await shell.resize({ cols: 160, rows: 50 });
const { exitCode } = await shell.wait();
```
### Mounting existing disks
Pass `mounts` when creating a sandbox to attach existing disks in addition to its internal root disk:
```typescript theme={null}
const workspace = await client.sandboxes.create({
mounts: [{ disk: "dsk-0123456789abcdef", path: "/mnt/data" }],
});
console.log((await workspace.exec("ls /mnt/data")).stdout);
await workspace.stop();
await workspace.start({ mounts: [] });
```
Each mount accepts `disk` (a disk ID or `Disk` object), `path`, `subdirectory`, `readOnly`, `conditional`, and `queueMs`.
With a single mount, `path` is optional and defaults to `/mnt/archil`.
Omitting `mounts` on `start()` keeps saved mounts; `[]` clears them and a list replaces them. See
[Persistent sandboxes](/compute/sandboxes/introduction#attaching-an-archil-disk) for sharing and forking behavior.
### Private images
```typescript theme={null}
const image = await client.images.build({
source: "ghcr.io/acme/agent-runtime:v2",
registryAuth: { username: "bot", password: process.env.REGISTRY_TOKEN! },
});
const agent = await client.sandboxes.create({ imageId: image.imageId });
```
`images.build()` waits until the image is ready, retrying interrupted builds up to three times. The default timeout
is 30 minutes; set it with `images.build(request, { timeoutSeconds: 600 })`. Failed builds raise `ImageBuildError`;
timeouts raise `ArchilError` with code `IMAGE_BUILD_TIMEOUT`. Use `images.get(imageId)` to read the current build state.
Pass `imageId` instead of `baseImage` when creating a sandbox from a built image.
See [Sandbox images](/compute/sandboxes/images) for reusing builds and updating tags.
## Reading and writing objects
A `Disk` doubles as an S3-compatible bucket: read, write, delete, and list its files by key without mounting it. These methods talk to Archil's S3 endpoint using your same API key — no separate S3 credentials or SigV4 signing on your part.
```typescript theme={null}
const d = await archil.getDisk("dsk-abc123");
const report = { generated: "2026-01", rows: 1234 };
// Write — body is a string, Uint8Array/Buffer, or ArrayBuffer. contentType is
// optional (defaults to application/octet-stream). Returns the etag.
const { etag } = await d.putObject(
"reports/2026-01/data.json",
JSON.stringify(report),
"application/json",
);
// Read — returns a Uint8Array.
const bytes = await d.getObject("reports/2026-01/data.json");
const text = new TextDecoder().decode(bytes);
// Metadata / existence without downloading the body
const meta = await d.headObject("reports/2026-01/data.json"); // null if absent
if (await d.objectExists("reports/2026-01/data.json")) {
// ...
}
// Delete (idempotent — deleting a missing key resolves successfully)
await d.deleteObject("reports/2026-01/data.json");
```
`putObject` handles any size with one call: small bodies go through a single request, and larger bodies are uploaded as a multipart upload automatically — split into parts, uploaded with bounded concurrency, and assembled, aborting the upload if any part fails so nothing is left half-staged. You don't pick a different method for big files. Tune the switch point and parallelism with options:
```typescript theme={null}
const { etag } = await d.putObject("backups/2026-01.tar", bigBytes, {
contentType: "application/x-tar",
multipartThreshold: 5 * 1024 * 1024, // switch to multipart above 5 MiB; default = partSize
partSize: 32 * 1024 * 1024, // ≥ 5 MiB; default 16 MiB
concurrency: 8, // parts in flight at once; default 4
});
```
For very large objects the part size grows automatically so the upload never exceeds S3's 10,000-part limit.
`listObjects` auto-paginates by default, returning every matching key. The first argument is a key prefix; a non-recursive listing (the default) returns the immediate level as `objects` plus subdirectory `commonPrefixes`:
```typescript theme={null}
const oneLevel = await d.listObjects("reports/"); // one level
const subtree = await d.listObjects("reports/", { recursive: true }); // whole subtree
const first100 = await d.listObjects("reports/", { limit: 100 }); // cap the total
// Stream pages instead of buffering everything (large listings):
for await (const page of d.listObjectsPages("reports/")) {
for (const obj of page.objects) {
console.log(obj.key, obj.size, obj.lastModified);
}
}
// Or drive pagination yourself:
const page = await d.listObjects("reports/", { singlePage: true });
if (page.isTruncated) {
const next = await d.listObjects("reports/", {
singlePage: true,
continuationToken: page.nextContinuationToken,
});
}
```
`deleteObjects` removes many keys in one round trip (auto-batched at S3's 1,000-key limit). Unlike `deleteObject`, per-key failures are returned rather than thrown:
```typescript theme={null}
const { deleted, errors } = await d.deleteObjects(["a.txt", "logs/b.txt", "c.txt"]);
for (const e of errors) console.warn(`${e.key}: ${e.code} ${e.message}`);
```
`appendObject` appends bytes to an existing object (creating it if absent) — handy for log-style writes. Each call may append at most 1 MiB; append in chunks to grow past that:
```typescript theme={null}
await d.appendObject("logs/app.log", "first line\n");
await d.appendObject("logs/app.log", "second line\n"); // concatenated
```
For manual control over the multipart lifecycle (e.g. uploading parts from separate processes), the raw S3 primitives live in the opt-in `d.multipart` namespace — `create`, `uploadPart`, `complete`, `abort`, `listParts`, `listUploads`. Most code never needs these; prefer `putObject`, which runs the lifecycle for you.
```typescript theme={null}
const { uploadId } = await d.multipart.create("big.bin");
const p1 = await d.multipart.uploadPart("big.bin", uploadId, 1, firstChunk);
const p2 = await d.multipart.uploadPart("big.bin", uploadId, 2, secondChunk);
await d.multipart.complete("big.bin", uploadId, [p1, p2]);
```
Transient failures (HTTP 429 and 5xx, plus network errors) are retried automatically with jittered exponential backoff before surfacing; other 4xx are caller errors and aren't retried. The two non-idempotent operations — `completeMultipartUpload` and `appendObject` — are not auto-retried, since a retry after a succeeded-but-unacknowledged call would return a spurious `NoSuchUpload` or duplicate the appended bytes.
Object-API failures throw `ArchilS3Error` (a subclass of `ArchilError`) with `status` (HTTP status), `code` (the S3 error code, e.g. `"NoSuchKey"`), `requestId`, and the raw XML body on `raw`. `getObject` on a missing key throws a 404 — use `headObject` / `objectExists` to probe without catching.
```typescript theme={null}
import { ArchilS3Error } from "disk";
try {
await d.getObject("reports/missing.json");
} catch (e) {
if (e instanceof ArchilS3Error) {
console.log(e.status, e.code, e.requestId);
}
}
```
## Share URLs
`share` generates a signed, time-limited URL that lets anyone download a file from the disk without credentials:
```typescript theme={null}
const share = await d.share("reports/2026-08/summary.csv");
console.log(share.url); // https://control.…/api/shared/
console.log(share.expiresIn); // 86400 (seconds)
// Custom expiry — any value from 1 to 604800 (7 days)
const hourShare = await d.share("reports/2026-08/summary.csv", { expiresIn: 3600 });
```
The returned URL supports HTTP `Range` requests and streams with a sensible `Content-Type`. After expiry, requests to the URL return `401`.
## Managing API keys
API keys are account-level, so these helpers live at the top level rather than on a `Disk`:
```typescript theme={null}
await archil.listApiKeys();
const k = await archil.createApiKey({ name: "ci-bot", description: "GitHub Actions" });
// save k.token — it's only returned once
await archil.deleteApiKey("key-abc123");
```
## CLI
`disk` ships a CLI under the same name. See the [`disk` CLI reference](/reference/disk-cli) for the full command list.
```bash theme={null}
npx disk create my-disk
npx disk list
npx disk dsk-abc123 exec "ls -la"
```
## Low-level protocol access
For the small number of integrations that need to speak Archil's filesystem protocol directly from Node — inodes, delegations, byte-level reads, paginated directory enumeration — install [`@archildata/native`](https://www.npmjs.com/package/@archildata/native) alongside `disk`:
```bash theme={null}
npm install disk @archildata/native
```
Then call `Disk.mount()`, which lazy-loads the native client:
```typescript theme={null}
import { getDisk } from "disk";
const d = await getDisk("dsk-abc123");
const client = await d.mount({ authToken: process.env.ARCHIL_DISK_TOKEN });
// `client` is an ArchilClient from @archildata/native — see that package's
// README on npm for the full inode-level API.
await client.close();
```
`@archildata/native` ships prebuilt binaries for **Linux** (x64, arm64, glibc) and **macOS** (arm64). On other platforms, `mount()` throws; the rest of `disk` still works.
If you're tempted to reach for `@archildata/native` to "run a bash command on a disk," use [`disk.exec`](#executing-commands) instead — it gives you a real shell with the filesystem mounted in a container, with no native dependencies on the caller side.
## Need a bash executor inside Node?
The legacy [`@archildata/just-bash`](/mounting/just-bash) package implements a JavaScript bash interpreter that talks to a disk through `@archildata/native`. It's still published, but for almost every use case `disk.exec` is the better answer — it runs your command in a real container with the filesystem mounted, returns stdout/stderr/exit code, and has no native dependencies on the caller.