This page explains how RQ’s rate limiting works under the hood and why it is built the way it is. It’s aimed at contributors; see the user guide for the public API and supported behavior.

The strategy implemented today is concurrency-based: it caps how many jobs sharing a key can be queued or executing at the same time, across all workers and queues using the same Redis database.

Terminology

• slot — one unit of capacity; a key with concurrency=N has N slots.
• allowed — jobs currently holding a slot (queued or executing).
• rate_limited — jobs waiting for a slot; also the JobStatus they carry.
• promote — move the oldest waiting job into allowed and push it onto its queue.
• rate limit registry — the per-key bookkeeper (RateLimitRegistry) that maintains the allowed and rate_limited sets and performs promotion.

The Mental Model

A step-by-step example — three jobs sharing a rate limit that allows two at a time:

rate_limit = RateLimit(key='reports', concurrency=2)

job_a = queue.enqueue(generate_report, rate_limit=rate_limit)  # slot free → queued
job_b = queue.enqueue(generate_report, rate_limit=rate_limit)  # slot free → queued
job_c = queue.enqueue(generate_report, rate_limit=rate_limit)  # no slots left → rate_limited

Contents of the queue and the rate limit registry (its allowed and rate_limited sets) after each event:

The key subtlety: a rate_limited job exists only as a job hash plus an entry in the rate_limited sorted set. It is not on any queue — workers cannot see it until promotion pushes it onto its origin queue.

Enqueueing always goes through the rate limit registry: every rate-limited job is saved as rate_limited and added to the waiting set first, then promotion runs — when capacity is free, the job promoted is usually the one just parked. This single admission path keeps waiting jobs FIFO (nothing jumps ahead of existing waiters), puts the capacity decision in exactly one place and makes plain enqueue(), scheduled jobs and resolved dependents behave identically.

Data Model in Redis

Everything lives in rq/rate_limit.py (RateLimit, RateLimitRegistry). Each rate limit key has one registry, stored as:

• rq:rl:{key} — config hash, stores concurrency.
• rq:rl:{key}:allowed — sorted set of jobs holding a slot, scored by acquire time.
• rq:rl:{key}:rate_limited — sorted set of waiting jobs, scored by enqueue time, so ZPOPMIN promotes oldest-first.
• rq:rl-keys — set of all known rate limit keys, so maintenance can sweep every registry without scanning the keyspace.

Jobs persist rate_limit_key and rate_limit_concurrency on their hash; Job.has_rate_limit is true when both are set.

The Two Operations and Why They’re Lua

• acquire_and_enqueue — while ZCARD(allowed) < concurrency, pop the oldest waiting job, add it to allowed, push it onto its origin queue and mark it queued. Stale entries are dropped. Returns the promoted ids.
• release_and_enqueue — remove a job from allowed, then run the same acquire logic. The release script is the acquire script with one ZREM prepended — a single shared body, so the two can’t drift.

Each operation runs as a single Lua script, so the capacity check and the promotion execute atomically in Redis and cannot interleave across concurrent workers. The promoted job is only known inside the script, so it builds the job and queue keys from the Job and Queue key prefixes passed as arguments. Enqueueing a rate-limited job is rejected when its job class or queue class overrides those prefixes.

The scheduler and the ready-jobs path queue the acquire script on their own pipelines, so promotion runs in the same transaction as the job’s writes. enqueue() rejects a caller pipeline, because saving the returned job on that pipeline afterwards would overwrite the promoted status.

Interactions Worth Knowing

• Retries and repeats — a job holds its slot only while queued or started (rate_limit.release_slot). An immediate retry or repeat keeps its slot; a scheduled one releases it and re-acquires one when due.
• Cancel and delete — both remove the job from the rate limit sets and fill free slots with waiting jobs, but a job with a live execution keeps its slot until the execution ends. Deleting through a caller pipeline keeps the slot until the execution ends or cleanup finds no live execution; cancelling through one is rejected.

Rate Limit Registry Cleanup

Rate limit state can go stale: a worker can crash after taking a slot and before releasing it, and deferred promotions leave freed capacity while jobs are still waiting. RateLimitRegistry.cleanup() reconciles this. It runs as part of clean_registries, the periodic registry maintenance performed by workers, and:

1. Releases allowed entries whose job is missing or no longer queued or started, filling free slots with waiting jobs after each release.
2. Promotes waiting jobs into any remaining free slots.
3. Deletes the registry once both sets are empty.

Known Sharp Edges

• Per-key concurrency drift — concurrency is stored once per key and the last registrant wins, but enqueue-time acquire uses the per-job value while release reads the stored config. Two jobs registering different values for the same key can over-admit. Since one call fills every free slot, a job carrying a higher value can admit several extra jobs at once.
• Returned job status can briefly lag — enqueue() returns an in-memory job whose status can be stale if a concurrent release promoted it in a narrow window; refresh() corrects it.