When to Reach for a Message Queue vs a Job Queue

D
Dick Edidiong Bassey
·

A job queue (Laravel Queues, Celery, Sidekiq) is for deferring work from the web request to a background worker. Send email. Process image. Generate report. The semantics: a producer creates a job, a worker consumes it exactly once.

A message queue (Apache Kafka, RabbitMQ, AWS SQS) is for decoupled event propagation. Multiple consumers can read the same message. Consumers can be added or removed without changing the producer. The semantics: a producer publishes an event, any number of consumers can react.

The wrong choice: using Kafka for a simple "send confirmation email after signup" flow. The operational overhead is not justified. Use a job queue.

The other wrong choice: using a job queue for "notify six downstream services when a payment is completed." The job queue serialises the notifications. A message queue parallelises them and lets each downstream service scale independently.

— Dick Bassey | DevDick | 2024