Distributed Job Scheduler
A custom database-backed task queue with non-blocking transactional dequeues, decentralized lease-based crash recovery, and a lock-free telemetry API.
Project Links
Technology Stack
TypeScript
Node.js
Express
PostgreSQL
Prisma
JestElevator Pitch
A high-performance, fault-tolerant background task execution engine designed to manage job distribution, worker crash recovery, and task prioritization across a horizontally scaling cluster of worker nodes, built from scratch without a heavy external message broker.
🛠️ The Challenge
- —Strict Transactional Safety (ACID): Guaranteeing a job is never double-allocated to workers.
- —Worker Fault Tolerance: Automatically detecting worker crashes mid-job and rescheduling the task.
- —Low Operational Overhead: Packaged entirely within containerized configurations, running seamlessly with minimal dependencies.
🏗️ Architectural Core
Worker Cluster (Horizontally Scaling)
1. Concurrency Control (FOR UPDATE SKIP LOCKED)
- •To support scaling workers horizontally, the database polling engine implements non-blocking row-level database locking.
- •Concurrent worker nodes skip locked rows and claim the next available job in line without race conditions.
- •This isolates the claiming step inside a single atomic transaction, preventing worker thread contention or database write blocking.
WITH next_job AS (
SELECT id FROM "Job"
WHERE (status = 'PENDING' AND "availableAt" <= NOW())
OR (status = 'RUNNING' AND "lockedAt" < NOW() - INTERVAL '30 seconds')
ORDER BY "priority" DESC, "createdAt" ASC
LIMIT 1
FOR UPDATE SKIP LOCKED
)
UPDATE "Job"
SET status = 'RUNNING', "lockedBy" = $1, "lockedAt" = NOW(), "attempts" = attempts + 1
FROM next_job WHERE "Job".id = next_job.id
RETURNING *;2. Heartbeat Leases & Resilient Crash Recovery
- •**The Lease**: While a worker runs a task, it continuously updates a `lockedAt` timestamp every 5 seconds (heartbeat loop).
- •**Failover Recovery**: Other workers poll for running tasks whose `lockedAt` is older than 30 seconds. If found, the failed job is immediately reclaimed and rescheduled.
- •**Exponential Backoff**: Reschedules failed tasks using progressive backoffs (`[5s, 20s, 80s, 240s]`) to avoid overloading downstream systems.
3. Graceful Shutdown
- •Worker processes trap operating system interrupts (`SIGINT`/`SIGTERM`) inside the process runtime.
- •Halts the acquisition loop, allows active jobs to finish executing, clears heartbeat timers, and terminates PostgreSQL client connections cleanly to prevent orphaned locks.
4. Dynamic Execution Registry (Strategy Pattern)
- •Utilizes a Strategy Pattern to register type-safe handlers for specific job types (e.g., `send_email`, `image_processing`).
- •Avoids conditional branching blocks in the worker's polling loop, allowing developers to cleanly scale and register new job tasks.
📊 Key Highlights & Observability
Observability Pipeline
Integrated pino-http logging to generate structured JSON log attributes in production (ideal for Grafana Loki or Datadog log parsing) and colorized output in development.
Telemetry stats endpoint (GET /job/stats)
Exposed telemetry showing real-time queue depths (jobs grouped by PENDING, RUNNING, COMPLETED, FAILED status) and the number of active workers calculated via heartbeat timestamps.
Production Deployment Flow
Shifted from prototyping initialization (prisma db push) to a production-grade automated database migration workflow (prisma migrate deploy) integrated directly into the Docker Compose setup.
Test Suite (Jest)
Built ESM-compliant unit tests using supertest and ts-jest's dynamic module mocking (jest.unstable_mockModule) to test routes, Zod schema validations, and stats calculation algorithms in isolation.