π Job System
A work-stealing thread pool that spreads physics, animation, streaming and other jobs across CPU cores, with three priority levels, completion handles and a blocking barrier.
Overview#
The Job System distributes heavy work (physics, AI, animation, asset loading) across CPU cores. Each job is a lightweight unit of work executed by a worker thread.
Approach: thread pool with work-stealing β each worker has local deques; when they are empty, it steals from another worker. This provides load balancing without central coordination.
Implemented today:
- Three priority levels: Critical / Normal / Background
- Per-worker deques per priority level, plus global queues per priority
JobHandlecompletion tracking with slot index + versionJobBarrierfor waiting on a group of jobsscheduleParallelForchunking
Not implemented yet: fibers. A job runs to completion on an OS thread; it cannot suspend halfway without blocking that thread.
API#
namespace Caffeine::Threading {
enum class JobPriority : u8 {
Critical = 0, // rendering, physics
Normal = 1, // gameplay, animation, events
Background = 2 // asset loading, streaming, compression
};
constexpr u32 PRIORITY_COUNT = 3;
// Base interface for every job. Jobs must be cheap to move; avoid
// allocating inside execute().
struct IJob {
virtual ~IJob() = default;
virtual void execute() = 0;
virtual JobPriority priority() const { return JobPriority::Normal; }
};
// Type-safe job that owns its data (no separate allocation for the payload).
template<typename DataT>
struct JobWithData : IJob {
DataT data;
std::function<void(DataT&)> func;
JobPriority prio = JobPriority::Normal;
void execute() override { func(data); }
JobPriority priority() const override { return prio; }
};
// Handle used to track and synchronize one job.
// index + version avoid the ABA problem without a lock.
class JobHandle {
public:
JobHandle() = default;
explicit JobHandle(u32 index, u32 version, std::atomic<u32>* completionFlag);
void wait() const; // blocks until the job completes (yields the CPU)
bool isComplete() const; // non-blocking check
explicit operator bool() const { return isComplete(); }
};
// Barrier: wait() blocks until the pending count reaches zero.
class JobBarrier {
public:
explicit JobBarrier(u32 targetCount = 0);
void add(); // increments the pending count
void release(); // decrements it; the JobSystem calls this when a job completes
void wait(); // blocks until count == 0
private:
std::atomic<u32> m_count{0};
std::mutex m_mutex;
std::condition_variable m_cv;
};
class JobSystem {
public:
explicit JobSystem(u32 workerCount = 0); // 0 = auto (hardware_concurrency - 1, at least 1)
~JobSystem();
JobSystem(const JobSystem&) = delete;
JobSystem& operator=(const JobSystem&) = delete;
JobHandle schedule(std::unique_ptr<IJob> job,
JobBarrier* barrier = nullptr,
JobPriority prio = JobPriority::Normal);
// Convenience: builds a JobWithData<DataT> automatically.
template<typename DataT, typename FuncT>
JobHandle scheduleData(DataT data,
FuncT&& func,
JobBarrier* barrier = nullptr,
JobPriority prio = JobPriority::Normal);
// Parallel-for over [0, count), split into one chunk per worker.
template<typename FuncT>
JobHandle scheduleParallelFor(u32 count,
FuncT&& func,
JobBarrier* barrier = nullptr,
JobPriority prio = JobPriority::Normal);
void waitAll(); // blocks until no jobs are pending
u32 workerCount() const;
struct Stats {
u32 activeWorkers;
u32 pendingJobs;
u64 completedJobsTotal;
f64 avgJobMs;
};
Stats stats() const;
};
} // namespace Caffeine::Threading
scheduleDatastores the callable instd::function<void(DataT&)>, so the lambda receives a mutableDataT&. Capture shared state by reference or by pointer; the job owns only itsDataTpayload and the callable.
Usage Examples#
// ββ Physics distributed in chunks βββββββββββββββββββββββββββββ
Caffeine::Threading::JobSystem jobs;
constexpr Caffeine::u32 CHUNK_SIZE = 256;
const Caffeine::u32 chunkCount = (particleCount + CHUNK_SIZE - 1) / CHUNK_SIZE;
Caffeine::Threading::JobBarrier barrier(chunkCount);
for (Caffeine::u32 i = 0; i < particleCount; i += CHUNK_SIZE) {
const Caffeine::u32 count = std::min(CHUNK_SIZE, particleCount - i);
jobs.scheduleData(
ChunkJob{i, count, dt},
[&particles](ChunkJob& j) {
for (Caffeine::u32 k = j.start; k < j.start + j.count; ++k)
particles[k].integrate(j.dt);
},
&barrier,
Caffeine::Threading::JobPriority::Critical
);
}
barrier.wait(); // all chunks ready before rendering
// ββ Simplified parallel-for βββββββββββββββββββββββββββββββββββ
jobs.scheduleParallelFor(entityCount,
[&](Caffeine::u32 i) { animators[i].advance(dt); },
nullptr,
Caffeine::Threading::JobPriority::Normal
);
// ββ Background asset loading ββββββββββββββββββββββββββββββββββ
auto handle = jobs.scheduleData(
AssetLoadJob{"textures/hero.caf"},
[](AssetLoadJob& j) { loadAsset(j.path); },
nullptr,
Caffeine::Threading::JobPriority::Background
);
// The main thread keeps running; poll when the asset is needed:
if (handle.isComplete()) { /* use the asset */ }
When the whole frame must wait for outstanding work, call jobs.waitAll().
JobHandle::wait() blocks only on that one job.
Internal Architecture#
JobSystem- Three global queues, one per priority, used by
schedule() - One
WorkStealDequeper worker per priority (3 Γ workerCount) for worker-local work tryExecuteOne()checks local queues in priority order, then global queues, then steals from other workers (also in priority order)- Each deque is a
std::dequeguarded by astd::mutex
- Three global queues, one per priority, used by
- Completion tracking
- A fixed ring of 4096 completion slots
JobHandlestores a slot index, a version and a pointer to its atomic flag- Reusing a slot increments the version, which prevents stale handles from observing a newer job's completion
JobBarrier- Atomic count plus mutex/condition variable
JobSystemcallsrelease()after a scheduled job finishes
Work stealing: a worker pops from the front of its own deque; when empty it steals from the back of another worker's deque, which reduces contention.
Priority is best-effort: workers always look at Critical before Normal before Background, but a running Background job is never preempted.
ECS Integration#
World::forEachParallel builds a JobBarrier, splits each matching archetype
into batches of 1000 entities and schedules the batches on the supplied
JobSystem:
Caffeine::ECS::ComponentQuery query;
query.with<Caffeine::ECS::RigidBody2D,
Caffeine::ECS::Collider2D,
Caffeine::ECS::Position2D>();
world.forEachParallel(query, jobs,
[](Caffeine::ECS::Entity entity,
Caffeine::ECS::RigidBody2D& rb,
Caffeine::ECS::Collider2D& col,
Caffeine::ECS::Position2D& pos) {
integrateRigidBody(rb, pos);
});
The callback receives the Entity first, followed by the requested components.
forEachParallel waits on its internal barrier before returning.
Tests and Limits#
tests/test_jobsystem.cpp covers:
- Priority ordering
IJob/JobWithDataexecution and defaultsJobHandlecompletion and bool conversionJobBarrieradd/release/wait- Worker count selection, scheduling,
scheduleData,scheduleParallelFor waitAll()andstats()- Stress runs with 1K jobs, 10K jobs and a 100K-element parallel-for
There is no fiber scheduler, no job cancellation and no work-stealing guarantee across priorities. These are roadmap items, not current API.
Design Decisions#
| Decision | Rationale |
|---|---|
workerCount = n-1 by default (at least 1) |
The main thread does not compete with CPU workers |
| Per-worker deques + work stealing | Better throughput than a single central queue for heterogeneous workloads |
index + version in the handle |
Avoids the ABA problem without a mutex |
| Three priority levels | Rendering/physics do not queue behind asset loading |
| Best-effort priority, no preemption | Keeps workers simple and avoids priority inversion from long jobs |
| Completion slots instead of per-job allocations | Handles are cheap and stable while jobs execute |
Dependencies#
- Upstream:
Caffeine::Core::Types, standard<thread>,<atomic>,<mutex>,<condition_variable>,<deque> - Downstream: ECS β
forEachParallel, Physics 2D, Animation, Asset Manager
π Related Topics#
| Topic | Description |
|---|---|
| Concurrency & Runtime | Job System in the frame pipeline |
| Assets & Pipeline | Async loading uses the Job System |