- Move work to a background job when the user does not need to wait for it: emails, imports, reports, image processing and calls to slow outside services.
- A queue stores each job until a worker process picks it up, so a slow or failing service no longer slows down or breaks the page.
- Retries, a failed-jobs list and alerts are part of the setup, not extras; a job that fails silently is worse than a slow page.
- Most queues can deliver a job more than once, so every job should be safe to run twice.
- Scheduled tasks need the same care: one cron entry, no overlapping runs and a check that tells you when a task stops running.
When does a web app need background jobs?
A web app needs background jobs when a request would otherwise make the user wait for work they do not need to watch, or when that work depends on another service that can be slow or down. Sending emails, importing spreadsheets, generating PDFs and reports, resizing images, syncing with a CRM and charging cards all fit. The job goes onto a queue, the page answers at once, and a separate worker does the work and retries it if something fails.
Small apps often start without a queue and add one after the first incident: a checkout that hangs while an email is sent, or an import that times out halfway. Planning it from the start costs little. It is a standard part of the Laravel development work we do, and the same ideas apply to Node.js and other back ends.
Which work should move off the page?
- Anything that talks to an outside service: email and SMS providers, payment webhooks you need to process, CRM and accounting syncs, AI APIs.
- Anything slow: file imports, exports, PDF generation, image and video processing, search index updates.
- Anything that fans out: one action that triggers many messages, such as notifying every member of a team.
- Anything that should retry: work that must happen eventually even if a service is down for a few minutes.
Keep inside the request only what the user needs to see straight away, such as saving the order and showing the confirmation page.
How does a queue actually work?
Your code creates a job, a small record that says what to do and with which data, and pushes it onto a queue. A worker, a separate long-running process, takes jobs from the queue one at a time and runs them. Laravel, for example, gives one queueing API across several back ends such as Amazon SQS, Redis or a relational database, so you can start simple and switch later without rewriting your jobs.
Because workers are long-lived processes, they keep running old code until they are restarted. Laravel's documentation says to restart workers during every deployment, and to run them under a process monitor such as Supervisor, which restarts them automatically if they stop. Forgetting this is a common reason why a bug fix seems not to work.
Database, Redis or Amazon SQS: which queue back end?
| Back end | Good for | Watch out for |
|---|---|---|
| Your database | Small apps; nothing new to run or pay for | Adds load to the database you already depend on |
| Redis | Many jobs, quick pick-up, a dashboard such as Laravel Horizon for throughput, runtime and failures | Another service to host, secure and back up |
| Amazon SQS | Managed queue on AWS; messages kept for 4 days by default and up to 14 days | Standard queues can deliver a message more than once and out of order |
For most small business apps, a database queue is a fine start and Redis or SQS comes in once volume grows or the database is busy.
What happens when a job fails?
Jobs fail for ordinary reasons: a provider times out, a file is malformed, an API key has expired. Plan three things.
- Retries with a delay. Set how many attempts a job gets and how long to wait between them. Laravel supports a growing delay, for example 1 second, then 5 seconds, then 10 seconds, so a struggling service gets room to recover.
- A place for jobs that gave up. Laravel records them in a failed jobs table and lets you retry one job or all of them with a command. On SQS the equivalent is a dead-letter queue, which receives a message once it has been received more times than the maxReceiveCount you set.
- An alert. Someone should hear about failed jobs the same day, with the error attached, instead of finding out from a customer.
Why must every job be safe to run twice?
Queues favour not losing work over never repeating it. Amazon SQS standard queues guarantee at-least-once delivery, which means more than one copy of a message might be delivered. Timeouts can cause repeats too: Laravel warns that if a worker's timeout is longer than the queue's retry_after setting, jobs may be processed twice.
So write jobs that check before they act. Mark an invoice as sent before sending it again, store the ID of the payment you created, and use the payment provider's idempotency keys, which let you safely repeat a request without creating a second charge. One more trap: a job dispatched inside a database transaction can start before the transaction is saved and miss the new data. Laravel's after_commit option waits until the transaction is committed.
What about scheduled tasks?
Daily reports, reminder emails and clean-up tasks run on a schedule. Laravel's scheduler needs only a single cron entry on the server and keeps the schedule in your code, where it is reviewed like everything else. Two settings prevent common problems: withoutOverlapping stops a slow task from starting again while the last run is still going, and onOneServer makes sure a task runs once when you have several servers.
The quiet risk is a task that simply stops. Our guide to monitoring a production web app explains heartbeat checks that alert you when a scheduled job has not reported in on time.
What should you watch once queues are live?
- Queue length and the age of the oldest job: the clearest sign that workers cannot keep up.
- Failed jobs: a count per day and the most common errors.
- Worker health: workers running, restarted on each deploy and not using more and more memory.
- Slow jobs: give long imports their own queue so they never block short, urgent jobs.
If your automations run in n8n rather than your own code, it has its own approach; see self-hosting n8n safely. For servers, process monitoring and alerts around your app, our cloud and DevOps team sets this up as part of the deployment.
Frequently asked questions
Can we use cron jobs instead of a queue?
For work that runs at fixed times, such as a nightly report, a scheduled task is enough. A queue is for work triggered by users or events, like an order or an upload, where you want it done soon but not inside the request. Most apps end up with both: the scheduler adds jobs to the queue, and workers process them.
Do background jobs work on shared hosting?
Partly. Many shared hosts allow a cron entry, so a database queue processed every minute can work for light use. What shared hosting usually does not allow is a long-running worker process kept alive by a process monitor. Once jobs matter to the business, a small server or a managed container service is the safer home.
How many queue workers do we need?
Start with one or two and watch two numbers: how many jobs are waiting and how old the oldest one is. If the oldest job keeps getting older at busy times, add workers or split slow jobs onto their own queue so they do not hold up quick ones such as password reset emails.
How do users know when a background job has finished?
Show the job as a status the user can see, such as "Your export is being prepared", and then tell them when it is done with an in-app notice, an email with a download link, or a live update on the page. Never leave people guessing whether their import worked.
Should every email go through a queue?
In most apps, yes. Sending through an email provider can take a moment and can fail for reasons outside your control, and neither should slow down a sign-up or a checkout. Queue the email, retry it if the provider is briefly unavailable, and log failures so support can see them.
Last updated