Projects
Distributed Systems
Distributed Job Scheduling Platform
A fault-tolerant scheduler for immediate, delayed, and cron-based execution.
10,000
Jobs tested
100
Workers
0
Failures
At-least-once
Delivery
Immediate, delayed, cron
Execution
Problem
A scheduler that works on one process fails when jobs, retries, delayed execution, and crash recovery need to survive multiple workers and deployment units.
Java 21Spring Boot 3KafkaRedisPostgreSQLDockerPrometheusOpenTelemetry
GitHub Repository Client01
Spring Boot API02
PostgreSQL03
Kafka04
Worker Pool05
Redis Locks06
Requirements
- Support immediate, delayed, and cron-based execution without coupling execution to API requests.
- Coordinate multiple workers without duplicate ownership of due jobs.
- Recover from worker crashes while preserving at-least-once delivery semantics.
Architecture
- Spring Boot API accepts scheduling requests and writes canonical job state to PostgreSQL.
- Indexed due-job scans lease work in batches and publish execution events through Kafka.
- Worker pools consume queue partitions while Redis locks guard critical coordination paths.
Design Decisions
- Used PostgreSQL FOR UPDATE SKIP LOCKED for concurrent leasing without blocking worker pools.
- Used the outbox pattern so database state and Kafka publication can be reconciled reliably.
- Kept retry metadata in PostgreSQL so backoff, DLQs, and crash recovery remain inspectable.
Scalability
- Batch leasing reduces database round trips under high due-job volume.
- Kafka fan-out lets API traffic and worker execution scale independently.
- Verified 10,000 jobs across 100 simulated workers with zero failures in the recorded load test.
Reliability
- Lease recovery reclaims expired work after worker crashes.
- Dead letter queues preserve failed jobs after retry exhaustion.
- Exponential backoff reduces repeated pressure from failing downstream tasks.
Challenges
- Preventing duplicate processing while keeping worker throughput high.
- Making delayed and cron jobs efficient without treating Kafka as the source of truth.
- Making failures observable across scheduler, broker, and worker boundaries.
Tradeoffs
- At-least-once delivery accepts possible duplicate execution, so worker idempotency matters.
- PostgreSQL leasing is operationally simpler than a bespoke coordinator but needs careful indexing.
- Redis locks add coordination speed while requiring explicit expiry and recovery behavior.
Results
- Completed a 10,000-job load test across 100 simulated workers with zero failures.
- Supported immediate, delayed, and cron-based execution paths.
- Added observability around retries, lease recovery, DLQs, and worker execution.
Lessons Learned
A scheduler is primarily a consistency problem, not only a timer problem.
Operational visibility has to be designed into queues, leases, and retry loops from the beginning.
At-least-once semantics are practical when duplicate prevention and recovery paths are explicit.