Skip to main content
Each Archil disk is configured to synchronize data from one or more data sources. When you create a new Archil disk, you configure which data sources you want to attach to your disk as well as the location on disk where the data source should be available. For example, you may have an Amazon S3 bucket labeled “model-data”, which you want to make available in the “/models” directory of your Archil disk. When you create a new Archil disk, you can simply “Add a Data Source”, select “Amazon S3”, and input “model-data” as the bucket name and “/models” as the disk path. You can now access any data from the “model-data” bucket at “/models” in your Archil disk, and any changes to that data on-disk will be synchronized back to the Amazon S3 bucket. When you connect to non-public data sources, you’ll need to configure credentials which allow Archil to access the data source. Archil disks require permissions to read and list objects (for read-only access), and write and delete objects (for read-write access).

​
Supported data sources

Archil supports a wide variety of data sources out of the box. Each data source has specific configuration requirements and credential setup procedures. For detailed configuration instructions for each supported data source, see the dedicated pages:
  • S3 Object Storage - Configure Amazon S3, Google Cloud Storage, Cloudflare R2, and other S3-compatible providers

​
Synchronization details

Archil is designed to provide eventually consistent, high-performance synchronization between the disk and data sources. Archil is always strongly read-after-write consistent for clients connected directly to the Archil disk, read more here. Archil automatically attempts to combine multiple writes into a single PutObject call to the data source. After an application stops writing to a file, it can take 10s of seconds for the data source to reflect the new data while Archil combines the writes. Because Archil automatically caches active data from S3 onto high-performance SSD storage, it can take minutes for a newly written object to appear in the corresponding directory on disk.

​
Concurrent writes

While a data source is configured as part of an Archil disk, Archil supports concurrent writes to the disk and the underlying data source. However, these writes must occur to different paths in the data source than those which are being actively written to from the disk. If you attempt to write to same file or directory which is being actively changed from the disk, it may result in undefined behavior including data loss or corruption.

​
Atomicity

Some data sources, such as Amazon S3, do not have atomic operations that mirror each of the possible POSIX file operations (for example, directory renames). When executing these kinds of operations against an Archil disk, you may notice partial results for the operations when reading directly from the underlying data source. Reads from the Archil disk itself are always strongly consistent.