Data Sources

Archil disks support optional, bidirectional synchronization to a data source. This is useful when you have an existing data set stored in cloud object storage, like Amazon S3, that you want to expose to agents as a native Linux file system. If, instead, you need to simultaneously use strongly consistent data from both the file system and an S3 API for a net-new data set, we instead recommend that you use Archil’s built-in support for the S3 API to access your data.

​

Supported data sources

Archil supports a wide variety of data sources out of the box. Each data source has specific configuration requirements and credential setup procedures. For detailed configuration instructions for each supported data source, see the dedicated pages:

• S3-compatible Object Storage - including Amazon S3, Google Cloud Storage, Cloudflare R2, and other providers

​

Consistency

Archil is strongly, read-after-write consistent for all clients connected through the file system interface or users of the Archil S3 API. For performance and efficiency reasons, writes to an Archil disk are replayed asynchrously to the backing data source. For example, if you create a new file hello.txt on an Archil disk using touch hello.txt, other clients connected to the same Archil disk will immediately see the new file. Clients of the Archil S3 API will also see the new file immediately. A backing S3 bucket configured as a data source will see the new file with eventual consistency, usually within 10s of seconds. To reduce the number of operations on backing object storage, Archil automatically combines multiple writes to individual files into a single PutObject call ot the data source. The specific amount of time that it takes Archil to replay a write into a backing data source depends on the specific workload and the compleixty of the operations. Before writes appear in the backing object storage, they are stored durably on Archil’s SSD storage servers. Archil redundantly stores data across multiple servers, in multiple Availability Zones, so the failure of any individual node (or an entire Availability Zone) to does not result in data loss. See Data durability for details. Archil does not offer a way for users to force a synchronization to the backing data source. If you find yourself needing to force synchronizing to the backing bucket, we instead recommend moving S3 API clients to the Archil S3 API, which is strongly consistent with changes in the file system. Synchronization from a data source is bidirectional. To reduce read latencies (often by up to 30x), Archil caches active data from a data source on our high-performance SSD storage servers. As a result, newly created object on the backing storage system can take minutes to appear in the corresponding directory on disk.

​

Concurrent writes

Archil supports concurrently modifying the disk from both the file system and from the origin data source — as long as these writes occur to different locations on the diks. If you attempt to write to the same file or directory which is being actively modified via the file system, it may result in undefined behavior including data loss or corruption. If you need to concurrently modify the same directory from the file system and the S3 API, we recommend using Archil’s built-in S3 API which supports concurrent modification.

​

Atomicity

Some data sources, such as Amazon S3, do not have atomic operations that mirror each of the possible POSIX file operations (for example, directory renames). When executing these kinds of operations against an Archil disk, you may notice partial results for the operations when reading directly from the underlying data source. Reads from the Archil disk itself are always strongly consistent.

​

Availability

If the configured data source becomes unreachable while a disk is mounted, writes are not lost: Archil buffers them durably in the disk’s SSD storage cache and flushes them to the data source once it is reachable again. This SSD cache for unacknowledged writes has a fixed size to maintain the durability of the disk, and if you exceed this size, the disk will stop accepting write operations. Reads served from cache are unaffected; a read that misses the cache and requires the data source blocks until it can be satisfied.

​

POSIX metadata

When Archil synchronizes a disk to a data source, file contents are written as ordinary objects at the matching key. Archil will internally track the POSIX metadata for each file, including its mode bits, ownership information, and any extended attributes. These are not visible by reading the bucket directly. Symbolic links are stored as a text file that contains just the path that the link points to. Hard links are stored as a full copy of the linked data.

​

Deleting a disk

Deleting an Archil disk does not delete the data in its data source. An Archil disk is a synchronized view over the bucket you own — when you delete the disk, Archil stops synchronizing and discards its cached copy, but the objects already written to your data source remain in your bucket, under your control.

​

Deleting a data source

If you delete the data source while an Archil disk remains connected to it, the associated disk will stop accepting read, write, or mount operations. This process is eventually consistent, and some operations may continue to function after the deletion ofr a data source for a short period of time.