Data Sources

Archil disks use a bucket managed by Archil by default. With bring your own bucket (BYOB), your bucket stores the data and Archil synchronizes with it bidirectionally. Active data is temporarily held in Archil’s storage layer in both modes. Connecting a data source is useful when you have an existing data set stored in cloud object storage, like Amazon S3, that you want to expose to agents as a native Linux file system. If, instead, you need to simultaneously use strongly consistent data from both the file system and an S3 API for a net-new data set, we instead recommend that you use Archil’s built-in support for the S3 API to access your data.

​

Supported data sources

Archil supports a wide variety of data sources out of the box. Each data source has specific configuration requirements and credential setup procedures. For detailed configuration instructions for each supported data source, see the dedicated pages:

• S3-compatible Object Storage - including Amazon S3, Google Cloud Storage, Cloudflare R2, and other providers

​

Consistency

Once a write is committed to Archil, reads that reach the disk server, including the Archil S3 API, see it immediately. Mounted clients can continue to serve cached data until their caches refresh. A local write must first be flushed with fsync or a background flush before other clients can read it from the server. See Consistency. Writes synchronize asynchronously to the backing bucket. Reading that bucket directly can therefore show an older version even after a write is committed to Archil. To reduce the number of operations on backing object storage, Archil automatically combines multiple writes to individual files into a single PutObject call ot the data source. The specific amount of time that it takes Archil to replay a write into a backing data source depends on the specific workload and the compleixty of the operations. Before writes appear in the backing object storage, they are stored durably on Archil’s SSD storage servers. Archil redundantly stores data across multiple servers, in multiple Availability Zones, so the failure of any individual node (or an entire Availability Zone) to does not result in data loss. See Data durability for details. Archil does not offer a way for users to force a synchronization to the backing data source. If you find yourself needing to force synchronizing to the backing bucket, we instead recommend moving S3 API clients to the Archil S3 API, which is strongly consistent with changes in the file system. Synchronization from a data source is bidirectional. To reduce read latencies (often by up to 30x), Archil caches active data from a data source on our high-performance SSD storage servers. As a result, newly created object on the backing storage system can take minutes to appear in the corresponding directory on disk.

​

Concurrent writes

Archil supports concurrently modifying the disk from both the file system and from the origin data source — as long as these writes occur to different locations on the diks. If you attempt to write to the same file or directory which is being actively modified via the file system, it may result in undefined behavior including data loss or corruption. If you need to concurrently modify the same directory from the file system and the S3 API, we recommend using Archil’s built-in S3 API which supports concurrent modification.

​

Atomicity

Some data sources, such as Amazon S3, do not have atomic operations that mirror each of the possible POSIX file operations (for example, directory renames). When executing these kinds of operations against an Archil disk, you may notice partial results for the operations when reading directly from the underlying data source. Reads that reach the Archil disk server are strongly consistent; mounted clients may serve a cached view.

​

Availability

If the configured data source becomes unreachable while a disk is mounted, writes are not lost: Archil buffers them durably in the disk’s SSD storage cache and flushes them to the data source once it is reachable again. This SSD cache for unacknowledged writes has a fixed size to maintain the durability of the disk, and if you exceed this size, the disk will stop accepting write operations. Reads served from cache are unaffected; a read that misses the cache and requires the data source blocks until it can be satisfied.

​

POSIX metadata

When Archil synchronizes a disk to a data source, file contents are written as ordinary objects at the matching key. Archil will internally track the POSIX metadata for each file, including its mode bits, ownership information, and any extended attributes. These are not visible by reading the bucket directly. Symbolic links are stored as a text file that contains just the path that the link points to. Hard links are stored as a full copy of the linked data.

​

Deleting a disk

Deleting an Archil disk does not delete the data in its data source. An Archil disk is a synchronized view over the bucket you own — when you delete the disk, Archil stops synchronizing and discards its cached copy, but the objects already written to your data source remain in your bucket, under your control.

​

Deleting a data source

If you delete the data source while an Archil disk remains connected to it, the associated disk will stop accepting read, write, or mount operations. This process is eventually consistent, and some operations may continue to function after the deletion ofr a data source for a short period of time.