Skip to main content

Sidekiq Config Overview

Sidekiq Config Variables​

┌──────────────────────────────────────────────────────────────┐
│                  config/deploy.yml                           │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ env:                                                   │  │
│  │   clear:                                               │  │
│  │     WEB_CONCURRENCY: "2"       ─────┐                  │  │
│  │     RAILS_MAX_THREADS: "5"     ─────┼─┐                │  │
│  │     SIDEKIQ_CONCURRENCY: "10"  ─────┼─┼─┐              │  │
│  └────────────────────────────────┼────┼─┼─┼──────────────┘  │
└───────────────────────────────────┼─┼──┼─┼─┼─────────────────┘
                                    │ │  │ │ │
                    Used by all ────┘ │  │ │ │
                    config files      │  │ │ │
                                      ▼  ▼ ▼ ▼
┌────────────────────────────────────────────────────────────────┐
│         config/sidekiq_connection.yml                          │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ production:                                              │  │
│  │   server_pool_size: <%= SIDEKIQ_CONCURRENCY + 2 %>       │  │
│  │   # Result: 10 + 2 = 12                                  │  │
│  │                                                          │  │
│  │   client_pool_size: <%= WEB_CONCURRENCY * MAX_THREADS %> │  |
│  │   # Result: 2 * 5 = 10                                   │  │
│  └──────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────┐
│              config/sidekiq.yml                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ production:                                            │  │
│  │   :concurrency: <%= SIDEKIQ_CONCURRENCY %>             │  │
│  │   # Result: 10                                         │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

Calculating thread count​

Total vCPUs available: 2

  • qa
  • production

Reserve for system: 0.5 vCPU Available for app: 1.5 vCPU

Priority split:

  • Web (70%): ~1 vCPU worth of processes
  • Sidekiq (30%): ~0.5 vCPU worth of processes

Recommendation:

  • WEB_CONCURRENCY: 2 (one process per vCPU, for process isolation)
  • RAILS_MAX_THREADS: 3-5 (I/O-bound = more threads OK)
  • SIDEKIQ_CONCURRENCY: 2-3 (enough for background processing)

Monitoring and Tuning​

How to Monitor and Tune

  1. Monitor CPU Usage:
# SSH into your EC2 instance
ssh -i ~/.ssh/jodapp.qa.deploy ubuntu@<bastion> -L 2222:<qa-private-ip>:22

# Check CPU usage
top

# Look for:
# - %CPU for puma processes
# - %CPU for sidekiq process
# - If consistently > 80%, reduce concurrency
# - If consistently < 30%, can increase concurrency
  1. Monitor Sidekiq Queue Depth:
# From Rails console
bin/rails console

# Check queue size
Sidekiq::Queue.new("default").size

# If queue is growing and CPU is low → increase concurrency
# If queue is low and CPU is high → decrease concurrency
  1. Application Performance Monitoring: If you set up Sidekiq Web UI:
  • Processing time: Should be mostly I/O wait, not CPU
  • Queue latency: Time between enqueue and processing
    • < 30 seconds = OK
    • > 1 minute = Need more concurrency