Fault-Tolerant Distributed Task Queue
A decoupled background processing engine built with Express and Redis, featuring atomic BLMOVE queue transitions, automated retries, and DLQ isolation.
Architecting a bulletproof background engine where no task is left behind, even when the system crashes.
Most web applications struggle with high-latency operations like web scraping, report generation, or bulk data processing. If these tasks are handled inside the main API request-response cycle, the server's event loop becomes starved and unresponsive, leading to cascading client timeouts and degraded user experience.
Furthermore, basic queue implementations often lack robust fault tolerance—if a worker node crashes or loses network connectivity mid-task, in-flight data is lost forever, creating a massive reliability gap in production environments.
This project solves that by decoupling task ingestion from background execution through a Fault-Tolerant Distributed Producer-Consumer Engine powered by Redis atomic primitives and event-driven worker processes.
Technical Approach & Execution Flow
The system is structured as a decoupled producer-consumer pipeline with strict separation of concerns across ingestion, state orchestration, and execution:
1. Ingestion (Producer Layer): An Express.js API acts as the entrypoint. When a client submits high-latency work, the API avoids synchronous processing. Instead, it wraps the payload in a structured metadata "envelope"—generating a unique UUID, timestamp, priority level, and initial attempt counters—before pushing the envelope into a Redis-backed queue. The client immediately receives a 202 Accepted response with the task identifier.
2. Orchestration & State Broker (Redis): Rather than using naive RPOP/LPUSH combinations that leave tasks vulnerable to data loss during network hiccups or mid-execution crashes, the engine uses the atomic BLMOVE primitive. Tasks transition seamlessly from the tasks:pending list to an active tasks:in-progress list in a single atomic database operation.
3. Processing (Event-Driven Workers): Autonomous worker instances continuously poll and consume tasks from the processing list. Each worker executes modular task handlers—such as distributed web scraping via Axios—under controlled timeouts and isolated memory boundaries.
4. Completion & State Hash: Upon successful execution, the worker acknowledges the job, removes it from the tasks:in-progress list, and persists the execution summary and metadata into a Redis Hash (task:<id>), enabling instant traceability and status polling.
In traditional queue architectures, pulling a message often involves a GET followed by a DELETE or MOVE. If the worker process crashes between these two commands, the task vanishes from existence.
By leveraging Redis's atomic BLMOVE (Blocking Left Move), the item is popped from the pending queue and pushed to the in-progress list within a single uninterruptible operation. The task is never in an unmanaged state—if a node fails, the record remains in Redis for automated recovery.
Challenges & Failure Modes
Engineering a distributed system requires planning for when things break, not if:
The Zombie Task: If a worker node crashes or gets killed abruptly by an OOM killer mid-execution, the task remains indefinitely trapped in the tasks:in-progress list without any active process working on it. I solved this by implementing a recovery pattern where state is preserved and orphaned tasks are safely recovered and rescheduled.
Transient Network Errors: Web scraping and external I/O tasks are prone to network timeouts and rate limits. To prevent premature job death, I implemented a 3-Stage Retry Policy. If an execution fails due to a transient exception, the worker increments the attempt counter in the metadata envelope and pushes the job back to the queue.
Poison Pills & Dead Letter Queue (DLQ): Some tasks fail deterministically due to malformed payloads or invalid schemas. Without guardrails, these tasks repeatedly cycle through workers, exhausting compute resources and blocking other traffic. Once a job exceeds its maximum retry threshold (3 attempts), the system automatically routes it to a Dead Letter Queue (DLQ) to isolate persistent failures for manual debugging and post-mortem inspection.
Key Takeaways
Atomicity is Non-Negotiable: Moving task coordination from application-level multi-step logic to atomic database primitives (BLMOVE) is far superior for preventing race conditions and data loss under heavy concurrent loads.
Observability is a Requirement: Building a dedicated CLI monitor reinforced that you cannot manage what you cannot see. Real-time metrics on queue depth, worker throughput, and task states are essential for understanding distributed system health.
Infrastructure Matters: Containerizing the stack with Docker proved that environment parity and horizontal scaling (spinning up 5 workers with one command) are the backbone of modern SDE workflows:
docker compose up --scale worker=5