Sidekiq Scheduler
We have two containers running:
- jodapp-api
- sidekiq
Look at the containers
sshintoqaorprod
ssh jodapp.pro.api
- Look at the docker containers running
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0919403f9177 ghcr.io/jod-app/jodapp-api:5161dec4dc8d35bcddc1d833c510f943f3baec86 "/rails/bin/docker-e…" 21 hours ago Up 21 hours 80/tcp jodapp-api-sidekiq-5161dec4dc8d35bcddc1d833c510f943f3baec86
f4eca2c45edd ghcr.io/jod-app/jodapp-api:5161dec4dc8d35bcddc1d833c510f943f3baec86 "/rails/bin/docker-e…" 21 hours ago Up 21 hours 80/tcp jodapp-api-web-5161dec4dc8d35bcddc1d833c510f943f3baec86
1796f5208e3c basecamp/kamal-proxy:v0.9.0 "kamal-proxy run" 2 months ago Up 2 months 0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp kamal-proxy
Look at the NAMES column
NAMES
jodapp-api-sidekiq-5161dec4dc8d35bcddc1d833c510f943f3baec86
jodapp-api-web-5161dec4dc8d35bcddc1d833c510f943f3baec86
kamal-proxy
jodapp-api-sidekiq
- this container executes
bundle exec sidekiqdefined indeploy/deploy.ymlbyservers: sidekiq: cmd: bundle exec sidekiq
jodapp-api-web
- this container executes
rails serverdefined inDockerfilebyCMD ["./bin/thrust", "./bin/rails", "server"]
Where does sidekiq_scheduler run?
Runs in the jodapp-api-sidekiq container.
How does it run in jodapp-api-sidekiq?
┌─────────────────────────────────────────────┐
│ Container 1: jodapp-api │
│ CMD: ./bin/thrust ./bin/rails server │
│ │
│ Rails boots → All initializers load │
│ │
│ ┌────────────────────────────────┐ │
│ │ sidekiq.rb initializer │ │
│ │ │ │
│ │ Sidekiq.configure_client ✓ │ │
│ │ - Sets up Redis connection │ │
│ │ - Pool size: 5 │ │
│ │ │ │
│ │ Sidekiq.configure_server ✗ │ │
│ │ - SKIPPED! Not a server! │ │
│ └────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────┐ │
│ │ sidekiq_scheduler.rb │ │
│ │ │ │
│ │ Sidekiq.configure_server ✗ │ │
│ │ - SKIPPED! Not a server! │ │
│ └────────────────────────────────┘ │
│ │
│ Role: Sidekiq CLIENT │
│ - Can enqueue jobs via MyJob.perform_async │
│ - Cannot process jobs │
│ - Scheduler does NOT run here │
└─────────────────────────────────────────────┘
┌───────────────────────────────────────┐
│ Container 2: sidekiq │
│ CMD: bundle exec sidekiq │
│ │
│ Rails boots → All initializers load │
│ │
│ ┌────────────────────────────────┐ │
│ │ sidekiq.rb initializer │ │
│ │ │ │
│ │ Sidekiq.configure_client ✓ │ │
│ │ - Sets up Redis connection │ │
│ │ │ │
│ │ Sidekiq.configure_server ✓ │ │
│ │ - RUNS! This IS a server! │ │
│ │ - Pool size: 13 │ │
│ │ - Error handlers configured │ │
│ └────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────┐ │
│ │ sidekiq_scheduler.rb │ │
│ │ │ │
│ │ Sidekiq.configure_server ✓ │ │
│ │ - RUNS! This IS a server! │ │
│ │ - Loads sidekiq_scheduler.yml│ │
│ │ - Starts scheduler thread │ │
│ └────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────┐ │
│ │ Sidekiq Scheduler Thread │ │
│ │ ↓ │ │
│ │ Checks schedule every second │ │
│ │ ↓ │ │
│ │ "Is it time to run heartbeat?"│ │
│ │ ↓ │ │
│ │ YES → Enqueue HeartbeatJob │ │
│ └────────────────────────────────┘ │
│ │
│ Role: Sidekiq SERVER │
│ - Processes jobs from queues │
│ - Scheduler thread runs HERE │
│ - Scheduler enqueues jobs to Redi │
└───────────────────────────────────────┘
The Foundation: Two Roles for Sidekiq
Sidekiq operates in two distinct modes:
- Sidekiq Client: Code that enqueues jobs (puts jobs into Redis)
- Sidekiq Server: Code that processes jobs (pulls jobs from Redis and executes them)
┌──────────────────────────────────────────────────────┐
│ Box 1: jodapp-api Container │
│ ┌────────────────────────────────────────────────┐ │
│ │ Rails Server (Puma/Thruster) │ │
│ │ - Handles HTTP requests │ │
│ │ - Sidekiq CLIENT only │ │
│ │ - Can call: MyJob.perform_async(args) │ │
│ │ - Scheduler: DOES NOT RUN │ │
│ └────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘
↓ enqueues jobs to Redis
↓
┌──────────────────────────────────────────────────────┐
│ Redis (Queue Storage) │
│ - Stores job queues │
│ - Stores scheduler configuration │
└──────────────────────────────────────────────────────┘
↑ stores jobs ↓ fetches jobs
↑ ↓
┌──────────────────────────────────────────────────────┐
│ Box 2: sidekiq Container │
│ ┌────────────────────────────────────────────────┐ │
│ │ Sidekiq Server Process │ │
│ │ │ │
│ │ Main Thread: │ │
│ │ - Fetches jobs from Redis queues │ │
│ │ - Executes job classes (e.g., HeartbeatJob) │ │
│ │ - Sidekiq SERVER │ │
│ │ │ │
│ │ Scheduler Thread: ← Lives INSIDE this process │ │
│ │ - Wakes up every ~5 seconds │ │
│ │ - Checks: "Is it time to run heartbeat?" │ │
│ │ - If YES: HeartbeatJob.perform_async() │ │
│ │ (enqueues to Redis, main thread picks it up)│ │
│ └────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘
Recurring Job Example
We will not use ActiveJob since it's just much simpler to directly use Sidekiq.
1. Create a class that includes Sidekiq::Job
# Example recurring job
class HeartbeatJob
include Sidekiq::Job
# Job Configuration
# - https://github.com/sidekiq/sidekiq/wiki/Error-Handling#configuration)
# - https://github.com/sidekiq/sidekiq/wiki/Advanced-Options#jobs
sidekiq_options(
queue: :default, # use a named queue for this Job, default default
retry: 3, # enable retries for this Job, default true. Alternatively, you can specify the max number of times a job is retried (ie. retry: 3)
dead: true, # whether a failing job should go to the Dead queue if it fails retries, default: true
backtrace: 10, # whether to save any error backtrace in the retry payload to display in web UI, can be true, false or an integer number of lines to save, default false. Be careful, backtraces are big and can take up a lot of space in Redis if you have a large number of retries. You should be using an error service like Honeybadger.
# pool: use the given Redis connection pool to push this type of job to a given shard.
tags: [ 'example' ] # add an Array of tags to each job. You can filter by tag within the Web UI.
)
# Defined `args: ['args 1', 2]` in sidekiq_scheduler
# Because `include_metadata: true`, we need to add `metadata` to the args
#
# Output in the sidekiq process (i.e. bundle exec sidekiq):
# INFO 2025-10-01T16:06:15.995Z pid=23511 tid=cw7 jid=5022e2ff1a64ad6c707d72d2 class=HeartbeatJob: start
# Heartbeat at 2025-10-01 16:06:16 UTC
# args passed from scheduler.yml: args 1, 2
# include_metadata: true: {"scheduled_at" => 1759334775.992}
#
def perform(arg1, number, metadata)
puts "Heartbeat at #{Time.current}"
puts "args: ['args 1', 2]: #{arg1}, #{number}"
puts "include_metadata: true: #{metadata}"
end
end
2. Add to config/sidekiq_scheduler.yml
# Provided as an example
heartbeat:
# one of cron: | every: | in:
every: 30s # Runs every 30s starting from the time sidekiq was last restarted
# By default the job name will be taken as worker class name.
# If you want to have a different job name and class name, provide the 'class' option
class: HeartbeatJob
queue: low
args: ['args 1', 2]
description: "Example usage of scheduler"
# Enable the `metadata` argument which will pass a Hash containing the schedule metadata
# as the last argument of the `perform` method. `false` by default.
include_metadata: true
# Enable / disable a job. All jobs are enabled by default.
enabled: false
# Deconstructs a hash defined as the `args` to keyword arguments.
#
# `false` by default.
#
# Example
#
# my_job:
# cron: '0 0 * * * *'
# class: MyJob
# args: { foo: 'bar', hello: 'world' }
# keyword_argument: true
#
# class MyJob < ActiveJob::Base
# def perform(foo:, hello:)
# # ...
# end
# end
keyword_argument: true
Testing Locally
You MUST be able to run jobs locally. It's much easier to test locally vs in QA.
# 1. Open your terminal
# 2. Start Redis, assuming you installed it locally
redis-server
# 3. Open a new terminal window
# 4. cd into the project folder in your local
cd path/to/local/repo/jodapp-api
# 5. Start sidekiq process (equivalent othe jodapp-api-sidekiq docker container)
bundle exec sidekiq
- Ensure you added to rails credentials development
sidekiq_web:
password: your-password
username: your-username
This is read in in the sidekiq block in routes.rb
- In your browser, navigate to
localhost:3000/sidekiq - put in the username and password
- Navigate to "Recurring Jobs"
Testing in QA
- Recurring jobs are defined in
config/sidekiq_scheduler.rb - Loading the recurring jobs are defined in
config/initializers/sidekiq_scheduler.rb
# Don't load scheduler if in Development and QA
return unless Rails.configuration.sidekiq_connection.dig(:is_scheduler_enabled)
To test your recurring jobs in QA, you have to update config/sidekiq_connection.yml
qa:
# ...
is_scheduler_enabled: true # <--- update to true
warning
Recurring jobs generally should not run in QA.
Ensure you turn update it back to false after testing.