Sharing Disks

Archil supports sharing disks between multiple connected users simultaneously. This easily allows you to support workflows where multiple agents are collaborating on the same dataset, or if multiple servers need to have live access to the same data set. Archil delivers the local-like disk performance by matching the semantics of xfs, ext4, or zfs and enabling connected clients to execute file system operations locally. This allows Archil to exceed the performance of comparable NFS or 9p systems, in which file system operations must always be performed on the remote server. Archil requires that clients hold a write-lock, called a “delegation”, in order to operate in this higher-performance mode. In the fullness of time, Archil will dynamically assign and manage delegations for optimal performance. Today, users must manually manage these delegations. Archil supports multiple mount modes, which change how delegations work.

Exclusive mode - By default (archil mount), mounting an Archil disk puts it in exclusive mode. This locks the entire file system for the machine which mounted. That machine will receive local-like performance that matches an EBS volume, but other machines will not be able to write to the disk until it’s unmounted. Other machines can still mount the disk with archil mount --read-only to read it.
Shared mode - If you mount the disk in shared mode (archil mount --shared), no lock is taken out on the disk. By default, the entire disk will appear read-only to the client until it manually requests a delegation (archil checkout <path>). With a delegation, the client will have exclusive write access to the specified <path> and receive local-like performance.
Conditional mode - If you mount the disk in conditional mode (archil mount --conditional), you enable the disk to execute operations remotely, like an NFS or 9p file system. This means that all files (which are not write-locked by another client) are immediately writeable, but the disk performance will be reduced to that of a remote file system. Users in conditional mode can continue to use archil checkout to get exclusive write access and higher performance to subdirectories on the disk.

We intend to make Conditional mode the default for new disks soon. The remainder of this article will describe how Shared mode and delegations work, followed by how Conditional mode works.

Choosing a mode

	Exclusive	Shared	Conditional
Mount command	`archil mount`	`archil mount --shared`	`archil mount --conditional`
Clients that can mount at once	One, plus any number of read-only clients	Many	Many
Where the client can write	Anywhere	Paths it has checked out	Anywhere another client hasn’t checked out
Needs `archil checkout` to write	No	Yes, though some operations check out automatically	No
Write performance	Local disk	Local disk	Remote file system; local disk on paths it has checked out

Selecting a mode on serverless execution

By default, disks in serverless execution are mounted in Shared mode (archil mount --shared) to enable multiple sandboxes to easily access the same data. You can change the disk into Exclusive mode by first executing archil checkout /mnt/archil as part of your serverless execution command. You can select Conditional mode by using the more verbose serverless execution API, like below:

await archil.exec({
  disks: {
    data: {
      disk: "dsk-def456",
      conditional: true,
    },
  },
  command: "ls /mnt/archil/data",
});

In this example, the disk will be mounted to /mnt/archil/data instead of /mnt/archil.

Automatic delegations in shared mode

When mounting in Shared mode, some operations automatically grant a write-lock on the file, to reduce the number of times that you need to call archil checkout. These operations are:

Creating a new file inside of a directory that does not have a delegation
Opening a file that is not covered by a delegation for writing (delegation automatically released on close)
Creating a new directory

These operations will automatically grant a delegation to the client who ran the operation. If you want other clients to have write-access to these resources, you will need to manually call archil checkin on the resource after creation. These rules lead to a common pitfall with Archil in which it’s possible to create a file or directory, but a removal operation will fail with EROFS: Read-only file system. This is because of the automatic delegation granting on file/folder creation. Deleting a file or folder is a directory-level operation, and requires a delegation on the parent directory of the file being removed.

These rules apply only to clients in Shared mode. Clients mounting in Conditional mode will never have delegations automatically issued to them.

Understanding delegations

Each delegation is an exclusive write-lock on a part of an Archil disk for a single client. When you call archil checkout <path>, all files and folders underneath that path will be included as part of the write delegation. This will prevent other write operations, including S3 PutObject operations, from completing against those paths. For example, consider a file system with the following structure:

/
├── group/
│   └── models/
│       ├── model1.txt
│       └── model2.txt
└── data/
    └── data.txt

If one client calls archil checkout group/ to get exclusive write-access to the “group” directory, it will also have exclusive write access to the group/models/ directory and its contents. Other clients will be able to continue to read these files and folders but will not be able to write to them, or call PutObject. Unless it’s mounted in conditional mode, this client would not be able to edit any other part of the disk. Other clients are simultaneously able to call PutObject to add files to the data/ directory or get exclusive write access over that directory by calling archil checkout data/. When your client is operating in conditional mode, write operations are executed on the server-side, and multiple clients can simultaneously read and write to any part of the file system without a delegation.

Releasing delegations

Delegations represent an exclusive write-lock on a part of the Archil disk. As a result, they need to be manually released by clients to signal that they are no longer performing writes to that portion of the disk. You can do this by either unmounting the disk from the server which requested the delegation or by manually calling archil checkin <path>.

Forcing ownership

In some cases, a client may crash or poweroff without cleanly unmounting or calling archil checkin to release a delegation. If this happens, you will not automatically be able to write to the location where the client’s delegation was held. Instead, you will need to force release the delegation. This will cause any in-progress writes from the original client to be lost. This mirrors “force detaching” a block device, such as an EBS or Hyperdisk disk. You can force checkout a delegation by calling archil checkout <path> --force, and you can force mount a disk into exclusive mode by calling archil mount --force. You can also remove delegations using the Archil API, if you don’t have the disk mounted locally:

Typescript

const disk = await archil.getDisk("dsk-abcdef")

for (const delegation of await disk.listDelegations()) {
  // If the client which issued the delegation is no longer
  // connected, revoke the delegation.
  if (delegation.isOrphaned) { 
    await disk.revokeDelegation(delegation)
  }
}

Conditional mode

When mounting in Conditional mode, a client can write to any file or directory on the disk without calling archil checkout, as long as no other client holds a delegation on it. This makes Conditional mode a good fit when several clients need to write to the same disk, such as multiple agents working in a shared workspace. Creating, renaming, and deleting files and directories are executed on the server before the operation returns. Writes to file contents are buffered on the client and sent to the server in the background, like on any Archil mount. Closing the file or calling fsync waits for them to reach the disk. Clients in conditional mode don’t lock files for each other. If two clients write to the same part of a file, the last write to reach the disk wins. If several clients need to update the same file, coordinate them in your application, or use archil checkout to give one client exclusive access to it. A client in conditional mode can still call archil checkout <path> to get exclusive write access and higher performance on part of the disk. While a client holds a delegation, other clients in conditional mode can’t write to that path. If a client in conditional mode still has unsent writes to that path when another client checks it out, those writes are lost and its mount becomes read-only; see Troubleshooting.

Getting started

Compute

Concepts

Data Sources

Details

Administration

Protocols

Reference

Legal

Choosing a mode

Selecting a mode on serverless execution

Automatic delegations in shared mode

Understanding delegations

Releasing delegations

Forcing ownership

Conditional mode

​Choosing a mode

​Selecting a mode on serverless execution

​Automatic delegations in shared mode

​Understanding delegations

​Releasing delegations

​Forcing ownership

​Conditional mode

Choosing a mode

Selecting a mode on serverless execution

Automatic delegations in shared mode

Understanding delegations

Releasing delegations

Forcing ownership

Conditional mode