Current Infrastructure: Two EC2s behind a Load Balancer with a MySQL server
1. Problem Description
In our current infrastructure, Laravel queue workers are deployed across multiple EC2 servers behind a load balancer. These workers consume jobs from a shared queue backend (MySQL). This configuration introduces the risk of duplicate job processing.
Specifically:
- A job is dispatched and enters the queue.
- Multiple servers (e.g., Server 1 and Server 2) are subscribed to the same queue.
- These servers may retrieve the same job concurrently.
- Without proper synchronization, the job may be processed multiple times.
2. Why ShouldBeUnique is Insufficient
Laravel’s ShouldBeUnique job middleware is inadequate to prevent duplicate processing in this distributed environment because:
- It only prevents duplicate job dispatching, not concurrent execution.
- It relies on Laravel’s cache driver for locking. If the cache driver is not shared across servers (e.g., using a local file or array cache), locking will not function correctly in preventing concurrent execution.
3. Solution: Using WithoutOverlapping for Distributed Locking
The recommended solution is to use the WithoutOverlapping middleware.
- WithoutOverlapping leverages Laravel’s cache store to implement a distributed lock.
- When the CACHE_DRIVER is configured to use a shared cache (in our case, MySQL DB), WithoutOverlapping effectively prevents concurrent execution of jobs with the same lock key across multiple servers.
4. Current Setup Evaluation
Our current configuration, which utilizes WithoutOverlapping and unique lock keys, is the correct architectural approach for our distributed job processing needs.
- We are generating unique lock keys based on the $products chunk using unique LockKey().
- We are using the WithoutOverlapping middleware to prevent simultaneous runs of the same job instance with the same lock key.
- We are setting releaseAfter(600) to automatically release the lock, which is crucial for handling failures or timeouts in long-running jobs.
This setup ensures that if a job with the same $products chunk is already running, any other queued instance with the same lock key will wait or be discarded, depending on Laravel’s queue retry configuration.
Important Considerations for WithoutOverlapping
- WithoutOverlapping prevents parallel execution of jobs with the same lock key. It does not prevent jobs from being queued.
- If different chunks generate the same lock key (e.g., due to a hash collision), they will block each other. While unlikely, this possibility should be considered.
5. Best Practices
- Continue using WithoutOverlapping with unique keys generated per chunk.
- Ensure that the CACHE_DRIVER is set to Redis (or another centralized cache) and that this cache is shared across all EC2 instances. (Currently, we are using MySQL Database)
- Configure releaseAfter() with a timeout value sufficient to accommodate the longest expected job execution time.
- Implement logging for job start and completion events to monitor and verify that overlaps are effectively prevented.
6. Optional Enhancements
- Implement retry logic to handle jobs skipped due to overlapping.
- Incorporate job monitoring and metrics to detect unexpected race conditions.
- Consider using a centralized job monitoring tool such as Laravel Horizon or Telescope.
7. Conclusion
ShouldBeUnique is used to avoid duplicate job dispatching, whereas WithoutOverlapping is the recommended mechanism to prevent the concurrent execution of identical jobs.
So, for us in a multi-server Laravel queue environment, WithoutOverlapping made more sense to use rather than ShouldBeUnique