Transparent S3 Integration
Write to normal filesystem paths and logs automatically sync to S3
Instance Isolation
Each instance creates its own directory for proper ownership separation
Infinite Capacity
Never run out of capacity because your logs are going to S3
Standard Logging Libraries
Use your existing logging setup - Python logging, winston, etc.
Create and mount an Archil disk
First, create an Archil disk for your logging infrastructure. Follow the Quickstart Guide to create a disk, then mount it in shared mode to allow multiple instances to write logs simultaneously.Mount the logging disk
Create instance-specific directories
Each application instance should create its own directory within the mounted filesystem. We recommend doing this as part of a startup script or on application startup:Application configuration
Configure your application loggers to write to the mounted Archil filesystem. Your logs will automatically flow to S3 without any S3 API calls.Log organization and management
With Archil, you don’t need traditional log rotation because logs are automatically offloaded to S3, where there’s infinite capacity. The standardized logging approach implements automatic minute-based rotation across all languages, creating new log files every minute with the consistent naming patternservice_YYYYMMDDTHH_MM_00Z.log
for natural chronological organization while maintaining optimal query performance.
S3 lifecycle management
One of the key advantages of using Archil for logging is that your logs automatically flow to S3, giving you access to all of S3’s built-in cost optimization features. You can leverage S3’s intelligent tiering and lifecycle policies to automatically move your log data to cheaper storage classes over time without any application changes. Configure lifecycle policies on your S3 bucket to automatically transition logs to cheaper storage classes:Querying logs with Amazon Athena
Once your logs are flowing to S3 through Archil, you can use Amazon Athena to run SQL queries directly against your log data without needing to load it into a database.Setting up Athena for log analysis
Create a table in Athena that leverages the standardized filename format for efficient time-based partitioning. The standardizedservice_YYYYMMDDTHH_MM_00Z.log
format enables Athena to automatically partition data by time:
- Efficient time-range queries: Athena can skip entire partitions when filtering by time
- Automatic partition discovery: No need to manually add partitions as new log files are created
- Cost optimization: Only scan the data you need for time-based analysis
- Scalability: Performance remains consistent as log volume grows