​
Overview

Archil transforms S3 buckets into high-performance local storage with centralized caching and data management. Archil is a fully-managed, durable, high-speed caching layer that sits between your EC2 instance and your existing S3 bucket. You don’t need to deploy any infrastructure or provision capacity to use Archil. Archil’s cache automatically expands with the working set size of your application, and you’re only billed for what you use.

Your instance connects to Archil using encrypted NFSv3, and Archil translates file-based operations into S3 API calls against your bucket. Archil automatically stages writes in the durable cache provide low-latency, and asynchronously completes those writes in S3.

​
Core Components

Centralized Cache

Archil maintains a centralized, durable, shared cache for frequently accessed data and recently written data. This cache automatically expands with the working set size of your application, and Archil intelligently caches data based on your access patterns. The Archil cache is designed to provide sub-millisecond latency for cached reads and writes. Because the Archil cache is shared across all clients, you can achie low-latency access to reference data from multiple instances simultaneously.

POSIX File Semantics

Archil’s durable caching layer allows it to efficiently perform file-based operations, and asynchronously persist writes to S3. Archil’s POSIX-compliant file system interface is strongly consistent for all connected file clients, and supports all standard POSIX file operations (including renames, appends, file locks, and symbolic links).

​

Your EC2 instances connect to Archil using NFSv3. Archil’s managed, high-speed caching layer runs in our cloud accounts, and synchronizes to the specified S3 bucket in your AWS, GCS, or Cloudflare account. You don’t need to deploy any infrastructure or provision capacity.

​
How It Works

1

File System Mount

You can mount a file system using the Archil mount helper. This performs IAM authentication on the mounting instance, establishes an encrypted NFSv3 connection between your EC2 instance and Archil’s caching layer.

2

Read Operations

When reading a file, Archil attempts to serve data from the centralized cache. If the data is not present in the cache, Archil fetches the backing data from S3 and updates the cache. For each read request, Archil may intelligently cache additional data based on your access patterns. Archil also caches file metadata, including directory contents, to minimize the number of S3 API calls that you have to pay for.

3

Write Operations

When performing mutating operations on the file system (including writes, renames, and directory changes), Archil first stages this data on its high-speed caching layer to provide strong consistency to other file clients. Archil will then asynchronously persist these changes to the backing S3 bucket, generally within 5 minutes. If Archil detects multiple write operations to the same backing S3 object, it will batch these operations into a single upload to minimize requests and reduce costs.

Archil’s high-speed cache automatically expands to the working set size of your application. By default, Archil will keep data that has been read or written in the cache for up to 1 hour.

​
Security

Archil automatically encrypts all data in transit and at rest by tunneling your instance’s NFSv3 traffic through a TLS connection.

Archil uses AWS IAM to authorize access to your file system. You can specify which IAM principals have access to the file system in the Archil web console. On mount, the Archil client generates a pre-signed AWS STS GetCallerIdentity request, which is used to authenticate the client to the file system.

​
Data Durability

Archil is designed to provide 99.999% (5 nines) durability for newly written data before it is synchronized to S3, and 99.99% (4 nines) of file system availability.

Archil automatically stores data redundantly within a single AWS Availability Zone. We are actively working on support for file systems which redundantly storage data across multiple Availability Zones for the highest data durability. If you need multiple Availability Zone durability, please let us know at [email protected].

​
Write Consistency

The Archil cache provides strongly consistent, read-after-write atomic POSIX operations for all connected file system clients.

Archil is generally able to synchronize changes between the file system view and S3 within minutes. Archil allows you to concurrently access your data from both the file system interface and S3.

For best results, we don’t recommend concurrently editing individual files from both interfaces. For example, if you write to the middle of a file using Archil and simultaneously call PutObject on the same object via the S3 API, Archil does not guarantee which edit will be reflected in S3.

Similarly, if you perform operations which are not atomic in the S3 API (such as a directory rename), you may observe partial results of the operations in your S3 bucket while Archil synchronizes the changes. You may experience inconsistent results if you add or remove files from the directory using the S3 API during this synchronization period.

We recommend segmenting your usage of the file system for writes. For example, structure your application so that PutObject calls which ingest data go to one prefix (for example, ‘input/’), while newly created files on Archil go to a different directory (for example, ‘output/’). This way, your application never needs to simultaneously edit the same file from both interfaces.

​
Comparison to Other Solutions

There are many different solutions which provide file-like interfaces on top of S3 buckets. However, only Archil provides an interface which is fully POSIX-compatible, stores data in the native format in S3, does not require capacity provisioning, and provides shared cache benefits to multiple instances.

​
FUSE Adapters

Existing FUSE adapters, such as S3FS and Mountpoint for Amazon S3, are not designed to support full POSIX semantics. For example, Mountpoint for Amazon S3 does not support file locks, random writes, renames, or symbolic links. In order to use these tools, you need to instrument your file applications to understand whether or not they use these unsupported operations, which can be difficult and error-prone. In addition, these tools do not provide shared caching for reference data sets to multiple clients, so large clusters may need to re-download the same data from S3 repeatedly. In comparison, Archil is designed to support all standard POSIX file operations, which means that it’s out-of-the-box compatible with all file applications.

​
Block-format Storage

Other file system adapters for S3, such as JuiceFS or ObjectiveFS, require that you store data in your S3 bucket in a proprietary block format. This means that you cannot use your existing data sets in S3 with these tools, and these buckets aren’t compatible with any existing S3 workflows that you use. Archil stores data in S3 in its native format, so you can simultaneously use and share the data from both a file system and S3. You can also use Archil to connect to existing data sets in S3.

​
Enterprise Storage

Some solutions, like Alluxio or FSx for Lustre, require that you either deploy and manage your own infrastructure or that you provision capacity. It’s often difficult to predict the specific storage needs of an application, especially if the application can have unexpected or bursty needs. Archil is a fully managed, serverless solution that automatically scales with your application. You don’t need to provision capacity or deploy infrastructure, and you’re only billed for what you use.