Scaling Challenges
When workflows scale to thousands of daily runs, typical self-hosted configurations start to creak. UI dashboards load slowly, API requests timeout, and background tasks run into execution delays. Monitoring high-volume instances requires a deep understanding of database indexes, queue architectures, and network tuning.
The Database Bottleneck
By default, n8n logs details of every successful and failed execution. At 10,000 executions per day, your database will record millions of executions a year. If you use n8n's default SQLite configuration, SQLite's single-write-lock architecture will quickly block requests, causing UI freezes and lost payloads.
First Step to Scale: Migration to PostgreSQL is mandatory. Ensure you create indexes on the execution database tables to speed up query requests from external monitors.
Implementing Queue Mode
For workloads beyond 10k runs, you should transition n8n from "regular mode" to "Queue Mode". This divides work between a main controller instance, a Redis message queue, and multiple worker instances:
- Main Node: Handles webhook triggers and routes dashboard access.
- Redis: Holds the execution queue.
- Worker Nodes: Pull execution jobs from Redis and run them concurrently.
This allows you to scale workers horizontally as your automation volume grows, preventing single-server CPU limits from crashing your service.
How AutoNod Optimizes Ingest
Generic monitors poll execution logs using heavy SQL operations (like SELECT * FROM execution), which degrades database performance on high-volume servers. AutoNod resolves this by using cursors and incremental paging. We fetch only the newest executions using indexed timestamps. This keeps database CPU utilization below 2% even on servers running over 100,000 workflows daily.
Instance Tuning Checklist
Ensure these environment variables are set in your high-volume n8n instances to prune redundant data:
# Prune execution logs automatically
EXECUTIONS_DATA_PRUNE=true
# Keep successful executions for only 1 day
EXECUTIONS_DATA_MAX_AGE=24
# Keep failed executions for up to 7 days
EXECUTIONS_DATA_MAX_AGE_FAILED=168
# Do not save successful execution details for simple workflows
EXECUTIONS_PROCESS_LINK_SUCCESS=false