HR-ATS-Portal/docs/architecture/adr/0004-background-job-queue.md

# ADR 0004 — Postgres-Backed Durable Job Queue (procrastinate), Not Celery + Redis

| | |
|---|---|
| **Status** | **Accepted** — 2026-07-29 |
| **Scope** | Queue technology, queue topology, retry and timeout policy, idempotency and locking conventions, failure visibility, periodic tasks, connection budget, deploy semantics, monitoring |
| **Owner** | Talha Ahmed owns queue configuration, locks, retry policy and the worker deployment. Ahmed Mujtaba writes task function bodies and the intake triage UI that surfaces failures — with a Talha review checkpoint |
| **Consistent with** | `_decisions.md` Part 1 → *Queue technology*, *Which processes are separate*, *Deployment topology* |
| **Related ADRs** | 0002 (the queue lives in the primary database — that is the whole point), 0001 (the `worker` process is the queue consumer), 0003 (scanning, parsing, derivative generation and retention purges are queue jobs) |

---

## Context

**No background processing exists today.** The repository has no backend at all (`_repo-findings.md` §B): no broker, no worker, no scheduler, no cron definition, no `Dockerfile`. The prototype's "processing statuses" are literals in a generated array (`js/data.js:284-300`) — an inbox that is already pre-resolved and joined to candidates and jobs, with no unresolved or failed state that cannot become a candidate. So the entire asynchronous layer is greenfield, and it has to be designed around one requirement above all others.

### The requirement that decides this ADR

**No inbound document may ever be silently lost** (BRD §6.3), and the intake chain is `raw_intake → attachment → scan → parse → candidate → application → score`. Every step after the first is asynchronous: scanning takes seconds, OCR takes multiple seconds, model calls take longer and fail intermittently.

That produces a specific correctness question: **when a `raw_intake` row is inserted, how do we guarantee that its processing job exists — no more, no less?**

With a queue in a *different* datastore, this is the classic dual-write problem, and both failure directions are real:

| Failure | Cause | Consequence here |
|---|---|---|
| Job enqueued, row rolled back | Broker write succeeded, then the transaction aborted | A worker picks up a job referencing a row that does not exist; noise at best, a crash loop at worst |
| Row committed, job never enqueued | Broker write failed or the process died between `COMMIT` and `publish` | **A document is silently lost.** It sits in the database in `received` forever, and nobody notices until a candidate asks why nobody called them back |

The second one is a direct violation of BRD §6.3, and it is not a hypothetical — it is the standard outcome of Celery's `task.delay()` being called inside or immediately after a Django transaction under any kind of process instability.

### Other forces

| Force | Figure / evidence | Implication |
|---|---|---|
| Volume is modest | 200–600 documents/day at peak; 20k–60k applications/year (**assumptions**) | Broker throughput is nowhere near a constraint. Correctness and operability decide |
| Two developers, no ops staff | `_repo-findings.md` §I | Every additional durable component is a backup, a monitor, a restore drill and an on-call surface |
| One database already exists and is managed with PITR | ADR 0002 | A queue inside it inherits all of that for free |
| BRD §8.3 requires retrievable async job status | | Jobs must be addressable rows, not fire-and-forget messages |
| Some work must be serialised per entity | Duplicate detection per candidate; rescoring per requisition version | Naive concurrent workers produce duplicate `candidate_duplicate_pair` rows and racing `ats_result` inserts |
| Redis is already provisioned | For cache, rate limiting and sessions (ADR 0001) | The tempting "we already have Redis, use it as a broker" argument must be answered, not ignored |

---

## Options considered

### Option A — `procrastinate`: Postgres-backed durable queue with `LISTEN`/`NOTIFY` + `SKIP LOCKED`

**Pros**
- **Transactional enqueue.** `INSERT raw_intake` and `INSERT procrastinate_jobs` commit or roll back together. Both failure modes in the table above become *impossible* — not mitigated, impossible — with no outbox table to build and maintain.
- Zero new durable infrastructure. One backup story, one PITR story, one thing to restore.
- Jobs are rows, so job status is a `SELECT`. BRD §8.3's "retrievable status" needs no extra machinery, and debugging is `psql`, which both developers already need.
- **Queueing locks** are declarative: `queueing_lock` serialises per-candidate dedupe and per-requisition rescore without hand-rolled advisory locks.
- Ships Django integration: migrations, admin views, retry with backoff, periodic (cron-style) tasks. That last one removes the need for a separate scheduler component entirely.
- `LISTEN`/`NOTIFY` means latency is push-based (sub-second pickup), not poll-interval-bound.
- `SKIP LOCKED` fetching is the well-understood, correct Postgres queue pattern.

**Cons (stated)**
- **Much smaller community than Celery.** When the junior developer is stuck, there are fewer search results and fewer Stack Overflow answers. This is a genuine cost on a two-person team.
- Queue load lands on the primary database: extra writes, index churn and dead tuples on `procrastinate_jobs`, which shares autovacuum and IO with recruiter queries.
- Throughput ceiling is lower than a dedicated broker (thousands of jobs/hour comfortably, not tens of thousands per second).
- No native fan-in primitives (no chords/canvas equivalent), so "score 200 applications then run one aggregate" must be composed manually.
- Fewer third-party integrations (monitoring exporters, dashboards) than Celery's ecosystem.
- One `LISTEN` connection held open per worker process — a small but permanent connection cost.

### Option B — Celery + Redis (the familiar default)

**Pros — real ones**
- The most widely known Python task queue by a large margin. Documentation, tutorials, Stack Overflow coverage and AI-assistant familiarity are all far better, which directly helps a junior developer working alone.
- Mature, well-understood operations: Flower for inspection, extensive monitoring integrations, battle-tested at scale far above ours.
- Rich composition primitives: chains, groups, chords, canvas — genuinely useful for fan-out/fan-in batch rescoring.
- Redis is already in the topology, so on the surface it appears to cost nothing.
- Very high throughput headroom; we would never think about it again.

**Cons**
- **Non-transactional enqueue in the one flow that must never lose data.** This is the disqualifier. `task.delay()` inside a transaction publishes to Redis immediately, before `COMMIT` — so a rollback leaves a job for a nonexistent row, and a crash after `COMMIT` but before publish loses the document permanently.
- Redis as a broker becomes a *store of record* for pending work, which means it now needs durability configuration, backups and a restore story. A Redis flush or eviction silently discards queued work — and eviction policy misconfiguration is a common, quiet failure.
- Job status is not a first-class queryable row without adding a result backend, which is another store to configure and expire.
- More moving parts for two developers: broker durability, result backend, beat scheduler (with its own single-instance-lock problem).

**The honest counter-argument, and why it does not win:** you can fix the correctness problem with `transaction.on_commit(lambda: task.delay(...))`. That closes the rollback direction but **not** the lost-document direction — a process death between `COMMIT` and the callback loses the job with no trace. Fixing that properly requires a transactional outbox table plus a relay, which means building, testing and monitoring the exact machinery Option A gives us for free, while still keeping Redis in the durable path. We would end up with more components and a weaker guarantee.

### Option C — Celery + Redis + a transactional outbox

**Pros**
- Restores the correctness guarantee while keeping Celery's ecosystem, canvas primitives and familiarity.
- The right answer at high throughput, where a database-backed queue would genuinely be the bottleneck.

**Cons**
- Two queues in the system: the outbox table and Redis. Relay lag, relay failure, at-least-once redelivery from the relay, and outbox table cleanup are all now ours to own.
- Strictly more code and more failure modes than Option A for an identical guarantee, at a volume where Celery's throughput advantage is worth nothing.
- The relay is a third process, or a periodic task that itself needs a scheduler.

**Verdict:** this is the option to adopt *if and only if* a throughput trigger below actually fires. It is recorded here as the pre-planned migration target, not as a rejected idea.

### Option D — RQ, Django-Q2, or `django-tasks`

**Pros:** simpler than Celery; Django-Q2 and `django-tasks` can use the database as the broker, which would recover transactional enqueue.
**Cons:** RQ is Redis-based (same correctness problem as B) and weaker on periodic tasks and retry policy. Django-Q2 is a thinner project than procrastinate with weaker locking primitives — and per-key serialisation is a requirement here, not a nicety. `django-tasks` is promising but immature for a system that needs retry policy, periodic tasks and locks on day one.

### Option E — Azure Service Bus / SQS (managed cloud queue)

**Pros:** fully managed durability, dead-letter queues, no capacity planning, scales far beyond our needs, and fits the Azure assumption from ADR 0001.
**Cons:** loses transactional enqueue (the dual-write problem returns in full), adds a cloud dependency to local development, and makes job status a two-system join. Dead-letter queues are a *worse* fit than our requirement, which is that a failure must surface **in the intake triage UI as a domain state** — not sit in an operator-only queue nobody opens. No benefit at this scale.

### Option F — A hand-rolled `SKIP LOCKED` queue

**Pros:** exactly the features we need, no dependency, full understanding of every line, transactional enqueue by construction.
**Cons:** we would reimplement retry with backoff, timeouts, periodic scheduling, queueing locks, graceful shutdown and admin visibility — perhaps 1,000+ lines of the most bug-prone code in any system, owned by a two-person team. Procrastinate is precisely this, already tested. Building it would be the wrong use of the senior developer's only scarce resource.

### Option G — Cron-driven polling scripts, or in-process threads

**Pros:** zero infrastructure; cron is understood by everyone.
**Cons:** no retry semantics, no per-job status (violating BRD §8.3), poll-interval latency on interactive AI, no locking so overlapping runs double-process, and in-process threads put OCR CPU on the request path and lose all in-flight work on deploy. Ruled out in ADR 0001.

---

## Decision

**A Postgres-backed durable queue via `procrastinate`, running in the `worker` process from the same image and codebase as `web`. Redis is provisioned for cache, rate limiting and session storage ONLY — never as a broker, never as a store of record for pending work.**

```mermaid
sequenceDiagram
  autonumber
  participant P as Mail poller [worker]
  participant DB as PostgreSQL [one database]
  participant W as Parse worker
  participant B as Object storage
  P->>DB: BEGIN
  P->>DB: INSERT staging.raw_intake
  P->>DB: INSERT staging.raw_intake_attachment with object_store_key
  P->>DB: INSERT procrastinate_jobs queue=ingest
  P->>DB: COMMIT
  Note over DB: Row and job commit together —<br/>neither can exist without the other
  DB-->>W: NOTIFY procrastinate_any_queue
  W->>DB: fetch job FOR UPDATE SKIP LOCKED
  W->>B: read bytes from intake-quarantine
  W->>DB: BEGIN, set virus_scan_status, enqueue parse job, COMMIT
```

### 1. Queue topology

| Queue | Work | Trust | Phase 1 concurrency | Phase 2 process |
|---|---|---|---|---|
| `mail` | Microsoft Graph polling, message → `raw_intake` | trusted | 1 (serialised by lock) | `worker-default` |
| `ingest` | Checksum, magic-byte validation follow-up, malware scan, quarantine → promote | boundary | 2 | `worker-default` |
| `parse` | PDF/DOCX/OCR text and layout extraction, derivative generation | **untrusted input** | 2 | **`worker-untrusted`** — restricted OS user, no outbound network, CPU/wall timeout, memory cap |
| `score` | ATS scoring, batch rescoring on config or requisition version change | trusted | 2 | `worker-default` |
| `ai` | Interactive model invocations, assistant retrieval, explanation generation | trusted | 2 | `worker-default` |
| `maintenance` | Retention purge, audit partition create/export, hash-chain verification, orphan-blob reconciliation, search index rebuild | trusted | 1 | `worker-default` |

Phase 1 runs **one** worker process consuming all six queues at concurrency 4. Phase 2 splits by queue into two processes from the same image — a security split (untrusted parsing isolation), and the mechanism by which ADR 0001's starvation trigger T5 is answered.

### 2. Retry and timeout policy — declared per task class, never defaulted

| Task class | Max attempts | Backoff | Wall timeout | On exhaustion |
|---|---|---|---|---|
| Mail poll | 5 | exponential, 30 s → 15 min | 5 min | Alert; channel marked `degraded`; next scheduled run retries |
| Malware scan | 5 | exponential, 10 s → 5 min | 2 min | `virus_scan_status = scan_failed`; attachment **blocked from parse**; visible in triage |
| Document parse | 3 | exponential, 1 min → 10 min | **90 s per document** (hard CPU + wall) | `intake_parse_attempt.state = failed` with the reason; **surfaced in the intake triage UI**; a recruiter can request a manual re-parse |
| Score (single) | 3 | exponential, 30 s → 5 min | 60 s | `ats_result` not written; application flagged `scoring_failed`; never silently defaults to a score |
| Batch rescore (per application) | 3 | exponential | 60 s | Per-application failure recorded; the batch completes with a partial-failure summary |
| AI invocation | 2 | 5 s, 20 s | 45 s | Circuit breaker opens (BRD NFR-7); UI degrades gracefully with AI absent — never a fabricated result |
| Retention purge | 3 | exponential, 1 h | 30 min | **Halt the run**, alert, and record partial `retention_action` rows. Never continue past an unexplained failure |
| Audit partition export / verify | 5 | exponential, 1 h | 30 min | Alert; the closed partition remains exportable on the next run |

**There is no dead-letter queue.** Exhaustion writes a **domain** terminal state that a human sees in the product — because BRD §6.3's requirement is not "the message is retained somewhere", it is "nothing is silently lost". An operator-only DLQ that nobody opens satisfies the letter and fails the intent.

### 3. Conventions every task must follow

| # | Convention | Rationale |
|---|---|---|
| 1 | **Delivery is at-least-once. Every task body is idempotent, keyed on the domain row** — e.g. parse keyed on `(raw_intake_attachment_id, parse_attempt_no)`, scoring keyed on `input_fingerprint` so an unchanged input is a no-op | A `SIGTERM` mid-job, a timeout, or a retry must not double-write. `input_fingerprint` already exists to make "has anything changed" an index lookup |
| 2 | **Enqueue only inside the transaction that creates the referenced row** — never after, never via `on_commit` | `on_commit` reintroduces the lost-job window this ADR exists to close |
| 3 | **`queueing_lock` on per-entity serial work** — `dedupe:candidate:{id}`, `rescore:requisition_version:{id}`, `mailpoll:channel:{id}` | Concurrent dedupe produces duplicate pair rows; concurrent rescore produces racing `ats_result` inserts |
| 4 | **Tasks take identifiers, never objects or model instances** | Payloads are `jsonb`; a stale serialised object is a subtle correctness bug |
| 5 | **Tasks re-check authorization and state on entry** — a job enqueued 10 minutes ago may reference a soft-deleted candidate or a superseded version | Time-of-enqueue and time-of-execution are different worlds |
| 6 | **AI tasks write their `AiRun` row before the result is usable**, and pass the *human* actor into `iam.can()` | The AI governance boundary is not relaxed just because the code runs in a worker |
| 7 | **No task writes to another module's tables** — tasks call the owning module's `service.py` facade | ADR 0001 boundary rules apply identically in the worker |

### 4. Periodic tasks (procrastinate's scheduler, not cron)

| Schedule | Task |
|---|---|
| every 2 min | Graph mail poll per active channel |
| every 15 min | Circuit-breaker health probe; requeue `scheduled` retries |
| hourly | Search-index refresh sweep for rows whose triggers deferred work |
| nightly | Retention purge; orphan-blob reconciliation; `retention_due_on` recompute; audit hash-chain verification |
| daily | Export the closed audit partition to the immutable archive (ADR 0003) |
| monthly | Create next month's audit partitions (and drop/detach at 13 months) |
| Phase 3, weekly | Fairness / disparate-impact evaluation |

Using the queue's own scheduler removes a component: no cron container, no Celery beat, and no "two beat instances double-fired the purge" incident, because periodic dispatch is itself a locked job row.

### 5. Connection budget — stated explicitly rather than discovered in production

| Consumer | Connections (Phase 1) |
|---|---|
| `web`: 2 uvicorn workers × 4 threads, persistent connections | 8 |
| `worker`: concurrency 4 | 4 |
| `worker`: `LISTEN` connection (1 per worker process) | 1 |
| Periodic scheduler | 1 |
| Migrations / admin / ad-hoc `psql` | ~5 (transient) |
| **Total steady-state** | **~14** |

Against several hundred available on an 8 GB managed instance (**verify the exact `max_connections` for the chosen tier**), Phase 1 headroom is ample. The number matters because it grows multiplicatively with web replicas and worker processes, and each Postgres backend costs memory. **PgBouncer in transaction mode is the pre-planned response** when peak connections exceed ~40% of `max_connections` — noted here so the growth path is decided before it is urgent. Procrastinate's `LISTEN` connection must bypass a transaction-mode pooler (session-mode port or direct connection), which is exactly the kind of detail that causes an afternoon of confusion if it is not written down in advance.

### 6. Deploy and shutdown semantics

- Worker receives `SIGTERM` on deploy, stops fetching, and is given a **90-second** grace period to finish in-flight jobs. This is why the parse wall timeout is 90 s and not longer.
- Anything killed mid-flight remains a durable row and is retried — hence convention 1. There is no "in-flight work lost on deploy" failure mode, which was the decisive flaw of in-process threads (Option G).
- A migration that changes a task's payload shape must tolerate old-shape payloads for one release, because jobs enqueued before the deploy will execute after it. This is an explicit code-review checklist item.

### 7. Monitoring — the four signals that matter

| Signal | Alert threshold |
|---|---|
| Oldest pending job age, **per queue** | `ai` >30 s; `ingest`/`parse` >10 min; `maintenance` >2 h |
| Queue depth per queue | `parse` >500 pending, or any queue growing monotonically for 30 min |
| Failure rate | >5% of attempts failing over 15 min, or **any** exhausted-attempt job on `maintenance` |
| `procrastinate_jobs` table health | Dead-tuple ratio >20%, or table size >2 GB after the completed-job sweep |

Completed jobs are pruned by a `maintenance` task after 30 days, keeping the table small and its indexes dense.

---

## Justification

**Transactional enqueue is the deciding factor, and everything else is secondary.** The platform's single most important reliability promise is that no inbound document is silently lost. Putting the queue in the same database as the intake row makes the dual-write bug class *structurally impossible* rather than mitigated — no outbox, no relay, no `on_commit` race, no reasoning about process death windows. Every other option either accepts that risk (B, D, E, G) or reintroduces the guarantee by building more machinery than procrastinate already provides (C, F).

**One fewer durable component is worth a great deal at two developers with no ops staff.** Redis as a *cache* can be flushed at any time with no consequence beyond a latency blip. Redis as a *broker* is a store of record: it needs durability settings, an eviction policy that cannot be wrong, a backup, and a restore procedure. Keeping pending work in the one database we already back up with PITR removes an entire operational surface.

**Declarative per-key locking is not a luxury here.** Duplicate detection and rescoring are exactly the operations where naive concurrency produces silent data corruption — duplicate `candidate_duplicate_pair` rows, racing `ats_result` inserts. Getting that right with advisory locks by hand is achievable; getting it right declaratively, reviewed once by Talha, is better.

**Jobs-as-rows collapses three requirements into one mechanism.** BRD §8.3's retrievable job status, BRD §6.3's no-silent-loss, and the intake triage UI's need to show failures are all served by the same table plus a domain terminal state. With a broker they would be three separate mechanisms.

**The cost we are choosing to pay, named honestly: community size.** Celery has vastly more documentation and answers, and the junior developer *will* hit a wall that a search engine solves for Celery and does not for procrastinate. The mitigation is a division of labour rather than a wish: **Talha owns queue configuration, retry policy, locks and deployment; Ahmed writes task function bodies**, which are ordinary Python functions with a decorator and require no queue expertise. If that mitigation fails in practice — if the junior is repeatedly blocked on queue mechanics rather than task logic — that is a signal worth acting on, and it is listed as a revisit trigger.

**Judgement call, stated:** we are trading ecosystem familiarity and raw throughput for a correctness guarantee and one less durable component. At 200–600 documents/day the throughput we are giving up is unusable, and the correctness we are buying is the product's central reliability promise. At 50× the volume this trade would invert, which is precisely what triggers T1 and T2 below encode.

---

## Consequences

### Positive

- The dual-write bug class cannot occur in the intake path. Not "is unlikely to" — cannot.
- One durable component, one backup, one PITR, one restore drill for both domain data and pending work.
- Job status is a `SELECT`, so BRD §8.3 is satisfied with no result backend and debugging is `psql`.
- No separate scheduler component and no double-fire risk on the nightly retention purge.
- Per-entity serialisation is declarative and reviewed once.
- Failures land as domain states in the product, so "nothing silently lost" is verifiable by a recruiter looking at a screen, not by an engineer reading a DLQ.
- The Phase 2 untrusted-parsing isolation is a queue-routing change plus a second revision — no new technology.
- Task bodies are plain functions, which keeps the junior's workstream about domain logic.

### Negative — costs we are accepting

| Cost | Accepted because |
|---|---|
| **Smaller community; fewer answers when the junior is stuck** | Explicit ownership split (Talha: configuration; Ahmed: task bodies); listed as revisit trigger T6 |
| **Queue write load, index churn and dead tuples on the primary database** | Volume is ~600 jobs/day peak, orders of magnitude below concern; completed-job pruning and table-health monitoring are specified up front |
| **Lower throughput ceiling than a dedicated broker** | Thousands of jobs/hour vs hundreds of jobs/day required. Numeric migration trigger defined |
| **No native fan-in (chords/canvas)** | Batch rescore is currently "N independent jobs plus a summary row", which needs no fan-in. Named as trigger T3 if a real fan-in requirement appears |
| **At-least-once delivery pushes idempotency onto every task author** | Unavoidable in any durable queue; made concrete by convention 1 and `input_fingerprint`, and enforced in review |
| **One `LISTEN` connection per worker, and a pooler-compatibility wrinkle** | Documented in the connection budget before it bites |
| **90-second deploy grace period couples release cadence to the parse timeout** | Both numbers chosen together, deliberately |
| **Payload-shape compatibility across one release is now a review concern** | Explicit checklist item; the alternative (draining the queue before every deploy) is worse |

---

## Risks

| # | Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|---|
| R1 | A developer calls `.defer()` outside the creating transaction (or wraps it in `on_commit`), silently reintroducing the lost-job window | **Medium — this is the most likely way this design is undermined** | High | Convention 2 is a documented review rule; the intake service facade is the only place allowed to enqueue intake jobs, so the surface is small and reviewable; a service-level test asserts that a rolled-back intake leaves zero jobs |
| R2 | Batch rescoring floods the queue and starves interactive `ai` tasks | Medium | Medium | Separate queues from day one; per-queue oldest-pending alerts; ADR 0001 trigger T5 splits worker processes by queue when it actually fires |
| R3 | A poison document retries three times, each burning 90 s of CPU | Medium | Low–Medium | Hard wall timeout, capped attempts, terminal `failed` state with the reason, and a manual re-parse action rather than an infinite loop |
| R4 | `procrastinate_jobs` bloat degrades recruiter query latency | Low | Medium | Completed-job pruning after 30 days; dead-tuple and table-size alerts; the table is narrow and the working set small |
| R5 | Queue polling or `LISTEN` interacts badly with a connection pooler added later | Medium (once PgBouncer arrives) | Medium | Documented now: the `LISTEN` connection needs session-mode or a direct connection. Verified when PgBouncer is introduced, not after an outage |
| R6 | The retention purge fails halfway, leaving some subjects pseudonymised and others not | Low–Medium | **High — a compliance failure** | Purge halts on failure rather than continuing; per-subject `retention_action` rows make the boundary auditable; a weekly verifier cross-checks purged blob keys (ADR 0003 R4) |
| R7 | `procrastinate` becomes unmaintained or lags a Postgres major version | Low | Medium | Trigger T6; the migration target (Option C) is pre-planned, and because tasks are plain functions calling module facades, the port is mechanical rather than architectural |
| R8 | A payload-shape change deploys while old-shape jobs are pending, crash-looping the worker | Medium | Low–Medium | One-release backward compatibility rule; capped attempts mean a crash loop terminates in a visible `failed` state rather than running forever |
| R9 | Someone "temporarily" uses Redis as a broker because it is already provisioned | Low–Medium | High | Stated prohibition in this ADR; Redis is configured with an eviction policy appropriate to a cache, which makes it visibly unsuitable as a broker |

---

## Revisit conditions

Reviewed quarterly by Talha alongside the ADR 0001 split triggers.

| # | Trigger | Threshold | Expected response |
|---|---|---|---|
| T1 | Sustained throughput | **>20 jobs/second sustained** | Migrate to Option C: Celery + Redis **with a transactional outbox** — the guarantee is non-negotiable and must be preserved |
| T2 | Worker fleet size | **More than 4 worker processes** | As T1; also re-evaluate `LISTEN` connection cost and pooling |
| T3 | Composition requirements | A genuine fan-in requirement appears (chords/canvas) that manual composition makes error-prone | Evaluate Celery canvas against building one aggregate-completion task; prefer the latter if it is a single case |
| T4 | Interactive latency | p95 time-to-start on the `ai` queue **>10 s** while web p95 <300 ms, after worker vertical scaling is exhausted | Split worker processes by queue (ADR 0001 T5) **before** changing queue technology |
| T5 | Database impact | Queue-attributable database CPU **>10%**, or `procrastinate_jobs` **>50 M rows** / **>2 GB** after pruning, or dead-tuple ratio persistently >20% | Tune pruning and autovacuum first; then reconsider T1 |
| T6 | Maintainer / team risk | No `procrastinate` release for **>12 months**, or it lacks support for the Postgres major version we are on; **or** the junior developer is blocked on queue mechanics (not task logic) more than twice in a quarter | For the first two, migrate per T1. For the third, first strengthen internal documentation and the task-body template — a familiarity gap is cheaper to fix than a migration |
| T7 | Connection pressure | Peak database connections **>40% of `max_connections`** | Introduce PgBouncer in transaction mode, with a session-mode path for `LISTEN` |
| T8 | Scheduling requirements | A need for sub-minute precision scheduling, or timezone-aware business-calendar scheduling beyond simple intervals | Evaluate a dedicated scheduler; do **not** reach for cron, which loses locking and status |

**Explicitly never a trigger:** "Celery is what everyone uses"; a desire to use Redis because it is already provisioned; or preference for a managed cloud queue in the absence of a numeric problem. Any change here must preserve transactional enqueue or replace it with an outbox — reverting to a non-transactional enqueue in the intake path is out of scope for any future ADR that does not also solve BRD §6.3 another way.