AWS Lambda MicroVMs

An AWS Lambda MicroVM is a Firecracker VM that Lambda builds from your Dockerfile, snapshots, and restores whenever you start one. When a MicroVM terminates, its files go with it. Mount an Archil disk in the MicroVM and anything written there stays on the disk, where the next MicroVM can mount it and your app can read it with the Archil SDK. Lambda MicroVMs support FUSE when the image is built with full OS capabilities, so the disk mounts as a regular directory. Mount it from the MicroVM’s run lifecycle hook and unmount it from the terminate hook.

Prerequisites

An Archil disk in an AWS region that also runs Lambda MicroVMs: aws-us-east-1, aws-us-west-2, aws-eu-west-1, or aws-us-east-2 (in preview).
AWS CLI 2.35.10 or later, which adds the aws lambda-microvms commands, and jq.
An S3 bucket in the same region to hold the image’s build artifact.

Set your Archil API key and region, and the matching AWS region for the AWS CLI:

export ARCHIL_API_KEY=key-...
export ARCHIL_REGION=aws-us-west-2
export AWS_REGION=us-west-2

Create the disk with the disk CLI. The output includes the disk ID and a one-time mount token, which grants access to this disk only:

npx disk create microvm-data

Export both for the steps below:

export ARCHIL_DISK_ID=dsk-...
export ARCHIL_MOUNT_TOKEN=adt_...

Build the image

The image needs the archil client and a process that serves the MicroVM’s lifecycle hooks. Lambda MicroVMs run on ARM64, and the installer picks the matching build.

Dockerfile

FROM public.ecr.aws/lambda/microvms:al2023-minimal
RUN dnf install -y python3 && dnf clean all
RUN curl -fsSL https://archil.com/install | sh
WORKDIR /app
COPY server.py .
CMD ["python3", "-u", "server.py"]

server.py serves the hooks on port 9000 and a small app on port 8080. The run hook mounts the disk using the disk ID, region, and token passed when the MicroVM starts. The terminate hook unmounts it. The app appends a line to a file on the disk for every request, and refuses requests while the disk isn’t mounted, so it never writes to the MicroVM’s local disk by mistake:

server.py

import json
import os
import subprocess
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread

MOUNTPOINT = "/mnt/archil"
HOOK_PREFIX = "/aws/lambda-microvms/runtime/v1/"
microvm_id = None


def archil(*args, env=None):
    # Stay inside the 30-second hook timeout set on the image.
    subprocess.run(["archil", *args], check=True, env=env, timeout=25)


class Hooks(BaseHTTPRequestHandler):
    def do_POST(self):
        global microvm_id
        body = self.rfile.read(int(self.headers.get("Content-Length") or 0))
        hook = self.path.removeprefix(HOOK_PREFIX)
        try:
            if hook == "run":
                request = json.loads(body)
                microvm_id = request["microvmId"]
                config = json.loads(request["runHookPayload"])
                os.makedirs(MOUNTPOINT, exist_ok=True)
                archil("mount", config["disk"], MOUNTPOINT, "--region", config["region"],
                       env={**os.environ, "ARCHIL_MOUNT_TOKEN": config["token"]})
            elif hook == "terminate":
                archil("unmount", MOUNTPOINT)
            self.send_response(200)
        except Exception as error:
            print(f"{hook} hook failed: {error}")
            self.send_response(500)
        self.end_headers()


class App(BaseHTTPRequestHandler):
    def do_GET(self):
        if not os.path.ismount(MOUNTPOINT):
            self.send_error(503, "Archil disk is not mounted")
            return
        with open(f"{MOUNTPOINT}/visits.log", "a+") as log:
            log.write(f"Hello from {microvm_id}\n")
            log.flush()
            os.fsync(log.fileno())
            log.seek(0)
            content = log.read().encode()
        self.send_response(200)
        self.send_header("Content-Length", str(len(content)))
        self.end_headers()
        self.wfile.write(content)


Thread(target=ThreadingHTTPServer(("0.0.0.0", 9000), Hooks).serve_forever, daemon=True).start()
ThreadingHTTPServer(("0.0.0.0", 8080), App).serve_forever()

Zip both files and upload the zip to your bucket:

export BUCKET=<your-bucket>
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
zip artifact.zip Dockerfile server.py
aws s3 cp artifact.zip "s3://$BUCKET/microvm/artifact.zip"

Lambda needs two IAM roles: one to build the image, which reads the artifact from S3, and one for the running MicroVM, which writes your app’s output to CloudWatch Logs. Both trust lambda.amazonaws.com:

TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]}'
aws iam create-role --role-name ArchilMicrovmBuild --assume-role-policy-document "$TRUST"
aws iam put-role-policy --role-name ArchilMicrovmBuild --policy-name read-artifact --policy-document \
  "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::$BUCKET/microvm/*\"}]}"
aws iam create-role --role-name ArchilMicrovmExec --assume-role-policy-document "$TRUST"
aws iam put-role-policy --role-name ArchilMicrovmExec --policy-name write-logs --policy-document \
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"*"}]}'

Now create the image. --additional-os-capabilities ALL gives the MicroVM the /dev/fuse device and the capabilities that archil mount needs. The --hooks option enables the run and terminate hooks on port 9000, plus the ready hook that tells Lambda when the image is ready to snapshot. Set the hook timeouts explicitly: the defaults are only a few seconds, and Lambda terminates a MicroVM whose run hook doesn’t finish in time:

export IMAGE_ARN=$(aws lambda-microvms create-microvm-image \
  --name archil-microvm \
  --base-image-arn "arn:aws:lambda:$AWS_REGION:aws:microvm-image:al2023-1" \
  --build-role-arn "arn:aws:iam::$ACCOUNT:role/ArchilMicrovmBuild" \
  --code-artifact "uri=s3://$BUCKET/microvm/artifact.zip" \
  --additional-os-capabilities ALL \
  --cpu-configurations architecture=ARM_64 \
  --hooks 'port=9000,microvmHooks={run=ENABLED,runTimeoutInSeconds=30,terminate=ENABLED,terminateTimeoutInSeconds=30},microvmImageHooks={ready=ENABLED}' \
  --query imageArn --output text)

The build takes a few minutes. Wait until the first version reports SUCCESSFUL before you start a MicroVM:

aws lambda-microvms get-microvm-image-version --image-identifier "$IMAGE_ARN" --image-version 1.0 --query state --output text

Start a MicroVM with the disk

Pass the disk ID, region, and token in --run-hook-payload. Lambda delivers the payload to the run hook as each MicroVM starts:

PAYLOAD=$(jq -nc --arg disk "$ARCHIL_DISK_ID" --arg region "$ARCHIL_REGION" --arg token "$ARCHIL_MOUNT_TOKEN" \
  '{disk: $disk, region: $region, token: $token}')
MICROVM_ID=$(aws lambda-microvms run-microvm \
  --image-identifier "$IMAGE_ARN" \
  --execution-role-arn "arn:aws:iam::$ACCOUNT:role/ArchilMicrovmExec" \
  --run-hook-payload "$PAYLOAD" \
  --query microvmId --output text)

The MicroVM stays PENDING until its run hook returns, so wait for it to reach RUNNING. If the mount fails, the hook returns an error and Lambda terminates the MicroVM; get-microvm shows why in stateReason, and your app’s output is in CloudWatch Logs under /aws/lambda-microvms/archil-microvm.

until [ "$(aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" --query state --output text)" != PENDING ]; do
  sleep 1
done
aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" --query '[state,stateReason]' --output text

Request the app on port 8080 with an auth token for the MicroVM:

ENDPOINT=$(aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" --query endpoint --output text)
AUTH=$(aws lambda-microvms create-microvm-auth-token --microvm-identifier "$MICROVM_ID" \
  --expiration-in-minutes 60 --allowed-ports '[{"port":8080}]' --query 'authToken."X-aws-proxy-auth"' --output text)
curl -H "X-aws-proxy-auth: $AUTH" "https://$ENDPOINT/"

Hello from microvm-cefc2a0c-f93b-36ca-ba38-babe9515a690

Terminate the MicroVM. Its terminate hook unmounts the disk, which sends every buffered write to the disk and releases it:

aws lambda-microvms terminate-microvm --microvm-identifier "$MICROVM_ID"

Start a second MicroVM with the same commands and request the app again. It finds the line the first MicroVM wrote:

Hello from microvm-cefc2a0c-f93b-36ca-ba38-babe9515a690
Hello from microvm-bd32b1eb-bba4-3db4-bae1-277799e7943d

If you would rather not pass the token at launch, store it in AWS Secrets Manager, pass the secret’s ARN in the payload instead, and read the secret in the run hook. Grant the execution role secretsmanager:GetSecretValue on that secret.

How the disk behaves in a MicroVM

Mount in the run hook, not while the image builds. Lambda restores every MicroVM from one snapshot of the image, so a disk mounted before the snapshot would come back in every MicroVM as the same Archil client. Mounting in the run hook gives each MicroVM its own. One MicroVM holds the disk at a time. A plain mount is exclusive. To mount the same disk in several MicroVMs at once, add --conditional to the archil mount command. Every MicroVM can then read and write anywhere on the disk; see Sharing disks. Suspend and resume need no special handling. When a MicroVM suspends, the mount is suspended with it. On resume, the client reconnects on its own and finishes any request that was in flight. Unmount in the terminate hook. Unmounting sends the client’s buffered writes to the disk and releases it for the next MicroVM. If a MicroVM ever goes away with the disk still mounted, see Release a stale lock.

Clean up

Delete the image, its build artifact, the roles, and the disk once you no longer need them:

aws lambda-microvms delete-microvm-image --image-identifier "$IMAGE_ARN"
aws s3 rm "s3://$BUCKET/microvm/artifact.zip"
aws iam delete-role-policy --role-name ArchilMicrovmBuild --policy-name read-artifact
aws iam delete-role --role-name ArchilMicrovmBuild
aws iam delete-role-policy --role-name ArchilMicrovmExec --policy-name write-logs
aws iam delete-role --role-name ArchilMicrovmExec
npx disk delete "$ARCHIL_DISK_ID"

Next steps

Python SDK: read, write, and search the disk without mounting it.
Sharing disks: mount one disk in many MicroVMs at once.
archil CLI reference: every archil mount option.

Agents

Analytics

Training

Apps & databases

Use Archil on other platforms

AWS Lambda MicroVMs

Prerequisites

Build the image

Start a MicroVM with the disk

How the disk behaves in a MicroVM

Clean up

Next steps

​Prerequisites

​Build the image

​Start a MicroVM with the disk

​How the disk behaves in a MicroVM

​Clean up

​Next steps

Prerequisites

Build the image

Start a MicroVM with the disk

How the disk behaves in a MicroVM

Clean up

Next steps