# Designing a Durable Scheduler

> Why the scheduler in ARGUS treats projects — not agent loops — as the unit that must survive restarts, with the restore mechanics and the tradeoffs.

作者：[达芬奇实习生](https://www.j0z.cn/about/)
原文：https://www.j0z.cn/articles/designing-a-durable-scheduler/
发布：2026-09-10
更新：2026-09-18
主题：ARGUS / Scheduler / Systems

## The unit that must survive

Most agent frameworks make the conversation, or the agent loop, the primary object. That works until a process dies. Once it restarts, everything you were counting on — the half-finished task, the leases, the ordering — is gone. In ARGUS 里我们做了一个不同的决定：持久的单位是 project，而不是 conversation。

A project has an identity, a queue of tasks, and a status. An agent is a worker that drains the queue. If the worker disappears, the project remains, and a new worker can resume exactly where the old one stopped. That reframing is the whole design.

**Definition**

When this article says “durable”, it means a task that was accepted before a restart is still accepted after it, with the same idempotency key and the same effect.

## The restore path

On boot, the scheduler loads every project that is not terminal, re-leases its in-flight tasks, and re-enqueues anything whose lease lapsed. The critical invariant is that a lease is a *claim*, not a *reservation*: if the holder never renews it, the work becomes eligible for someone else.

**scheduler/restore.ts**

```typescript
export async function restore(projects: Project[]): Promise<void> {
  for (const project of projects) {
    if (isTerminal(project.status)) continue;
    const lapsed = await claimStore.findLapsedLeases(project.id, now());
    for (const task of lapsed) {
      await queue.enqueue({ taskId: task.id, key: task.idempotencyKey });
    }
    await scheduler.resume(project.id);
  }
}
```

The enqueue is keyed by `task.idempotencyKey`, so a restore that runs twice is safe. The second run finds no lapsed leases and enqueues nothing.

**The trap**

The first implementation compared wall-clock lease expiry across two nodes with unsynchronised clocks. Under clock skew a healthy lease looked lapsed and the same task ran twice. Lease renewal is now a monotonic counter, not a timestamp.

## Measured before and after

We ran a restore against a mixed backlog after killing the worker mid-drain. These are internal harness numbers, not production SLAs.

| scenario                 | re-drained | duplicates          | wall time |
| ------------------------ | ---------- | ------------------- | --------- |
| before (timestamp lease) | 100%       | observed under skew | 4.1s      |
| after (counter lease)    | 100%       | none observed       | 3.9s      |

*Restore behaviour, 500 mixed backlog (internal harness)*

## How the pieces fit

A coarse view of the runtime, with the queue deliberately kept out of the source-of-truth role.

**Scheduler control plane**

```text
  API ──▶ Store (projects, tasks, claims)   ← source of truth
              │
              ▼
        Scheduler ──▶ Queue (Redis) ──▶ Workers
              ▲                              │
              └────── lease renew / heartbeat
```

*Projects are the durable unit; the queue is a transient optimisation.*

## Timeline

**How this took shape**

- 2026-01 — project-first model proposed
- 2026-04 — first lapsed-lease restore
- 2026-06 — clock-skew duplication found in test
- 2026-09 — counter-based leases shipped

The queue can be flushed and rebuilt from the store. That property is why we can treat Redis as disposable and keep confidence in the system.

## What I'd keep, what I'd change

Durable projects are the right primitive and I'd keep them without hesitation. The lease model took two rewrites to get right; a counter from the start would have saved weeks of chasing a bug that only appeared under load and skew. The counterargument — that a single per-process timer is enough — holds only while you run one node. The moment you run two, the lease has to be a shared, monotonic fact.
