# 04 — Integrations and Asynchronous Processing ## Status / Scope of this document Design document, implementation-ready. Covers assignment §13 (careers email), §14 (careers website), §15 (CV and AI processing pipeline), §16 (background processing) and §17 (file storage), plus the external-provider abstraction that keeps email, AI, storage and malware scanning swappable. Binding inputs: `_repo-findings.md` (verified repository evidence) and `_decisions.md` (binding architecture and database decisions). Where this document names a table, it uses the physical names from `_decisions.md` Part 2 (`raw_intake`, `raw_intake_attachment`, `intake_parse_attempt`, `intake_resolution`, `candidate_document`, `ats_result`). What is **not** in scope here: the relational schema in detail (owned by the data-model document), authorization internals (`identity`/`iam.can()`), the scoring algorithm itself, and outbound job-board publishing beyond its integration contract — `_decisions.md` puts `integrations_outbound` in Phase 4. Two facts constrain everything below and are stated once: 1. **The repository contains no integration code of any kind.** No `.env` or `.env.example`, no HTTP client, no email library, no queue, no storage client, no server-side code at all (findings §B). `grep` for `fetch(`, `XMLHttpRequest`, `axios`, `WebSocket` and `EventSource` across `js/` and `index.html` returns nothing (findings §C). Every provider named in this document is therefore a **recommendation with a labelled assumption**, not a discovered fact. The prototype's `inbox` array (`js/data.js:284-300`) is generated in-browser by a seeded PRNG (`js/data.js:8-10`); it is a mock, not a mail integration. 2. **There is no meeting transcript and no recruitment document beyond the BRD this project produced** (findings §B). Requirements come from the assignment prompt and the BRD. Confidence labels used throughout: **High** (follows from repository evidence or a binding decision), **Medium** (standard practice, low risk of being wrong), **Low** (depends on an unanswered question, named inline). ### Vocabulary pinned once — actor and decision enums Two closed sets appear in guards, CHECK constraints and audit rows throughout this document. They are written from **one** source so that a service-layer guard and the database constraint it mirrors can never compare against a value the other side cannot hold. Both are `text` plus a `CHECK` (not Postgres enum types) per the `_decisions.md` enum-strategy decision. | Set | Column(s) | Permitted values | Notes | |---|---|---|---| | Actor kind | `actor_kind` on `audit.audit_event` and every `*_history` table; `created_by_actor_kind` on `outbound_message` | `user`, `system`, `integration`, `ai_agent` | `_decisions.md` audit decision; Part 2 §9.5 and §28.1. **`user` is the only "a real person did this" value** — it is the only one that carries an `actor_user_id`. There is **no `human` value.** Every "AI must never auto-reject" guard is therefore written `<> 'user'`, never `!= 'human'`. `ai_agent` is the chatbot acting `on_behalf_of_user_id`; `integration` is a service principal such as the careers-form endpoint; `system` is a timer or an unattributed trigger write (`actor_unknown = true`). | | Decision mode | `intake_resolution.decision_mode` | `human`, `automatic` | A **different** set with a genuinely different meaning — it records whether a triage decision was taken by a person or by the auto-create policy (§4.2 step 16). `human` here is correct and must not be renamed to `user`; it labels a *mode*, not an actor identity. | Reading these as one set is the mistake to avoid: `decision_mode = 'human'` and `actor_kind = 'user'` both mean "a person did it", in two different columns, deliberately. --- ## 1. Provider evidence and the assumption register ### 1.1 What the repository says about providers: nothing | Question | Repository evidence | Verdict | |---|---|---| | Which mail provider hosts the careers mailbox? | No `.env`, no `.env.example`, no mail client, no OAuth config, no SDK (findings §B). `.gitignore` anticipates `.env` files but none exist and its Python section postdates `devserver.py`, which findings §H explicitly warns must not be read as stack intent. | **Unknown from the repository.** Must be confirmed with Utopia Brands IT. | | Which AI provider? | No AI code. The prototype's `aiScore` is `int(52,98)` (`js/data.js:123`) and `js/aiassistant.js` renders static markup. | **Unknown.** `_decisions.md` flags this as BRD OQ-1, unresolved, and the single decision most likely to change the design. | | Which object store? | No storage client, no bucket name, no key convention. Attachments do not exist; the prototype has no file handling. | **Unknown.** | | Which malware scanner? | Nothing. | **Unknown.** | I will not manufacture evidence for a provider. What follows is the recommendation and the reasoning, with the assumption labelled at every point where it is load-bearing. ### 1.2 Assumption register Every assumption in this document, in one place, so a reviewer can attack them individually. | ID | Assumption | Basis | If wrong | |---|---|---|---| | **A1** | Utopia Brands runs Microsoft 365 for corporate mail, and the careers mailbox is an Exchange Online mailbox in the corporate tenant. | BRD §8.1 names Outlook as inbound channel #1; `_decisions.md` already assumes M365 for Entra ID SSO and recommends Azure for that reason (line 233). | Fall back to the IMAP/SMTP design in §2.12. Google Workspace would need a Gmail API adapter — same port, ~4-6 days. | | **A2** | A **dedicated shared mailbox** (e.g. `careers@utopiabrands.com`) exists or can be created, distinct from any individual's mailbox. | Corporate norm for a careers address; required for a service-principal access policy. | If it is a distribution list, there is no mailbox to read. A distribution list forwarding into a shared mailbox is acceptable; a list with no mailbox is a blocker. | | **A3** | Corporate IT will grant an Entra ID app registration with **application** (not delegated) Graph mail permissions, admin-consented, restricted to that one mailbox. | Standard M365 admin operation. | Delegated-only access forces a resource-owner flow with a service account password — worse security and a refresh-token expiry problem. `_decisions.md` already flags this as a Phase 1 critical-path risk (line 269) and says to start the request in Phase 0. | | **A4** | Volume: **200-600 documents/day at peak**, 20k-60k applications/year. | Carried forward from `_decisions.md` (line 126) so sizing is consistent across the package. | Bulk job-board feeds would raise this 1-2 orders of magnitude (`_decisions.md` risk, line 462) and both queue and storage sizing must be re-derived. | | **A5** | AI model access is a **contracted API provider under a data-processing agreement**, called only from the worker plus one streaming endpoint. | Binding assumption in `_decisions.md` (line 138), pending BRD OQ-1. | Self-hosting adds GPU infrastructure and an MLOps burden two developers cannot absorb; the phase ranges break. | | **A6** | The careers website is a **separate application** (marketing-owned CMS or Next.js site) that does not and will not share the ATS database. | Assignment §14 forbids shared-database access; no careers site exists in this repository. | If the careers site is later built inside this repository as a public Django app, §3.3 still applies — the public surface calls `/api/v1/` like any other client and gets its own credentials and its own rate-limit bucket. | | **A7** | Cloud is **Azure** (Container Apps, PostgreSQL Flexible Server, Blob Storage, Key Vault, Entra ID). | `_decisions.md` line 233, labelled an assumption there too. | Drives the storage decision in §6 and is the one place this document must diverge from a literal reading of the task; see §6.2 and §9.1. | --- ## 2. Careers email — Microsoft 365 and Microsoft Graph ### 2.1 Recommendation and why **Recommended: Microsoft 365 / Exchange Online, accessed through Microsoft Graph v1.0 with an Entra ID application registration, client-credentials flow, restricted to a single mailbox by an Exchange `ApplicationAccessPolicy`.** Confidence: **Medium** (the design is sound; the premise A1/A2/A3 is unconfirmed). Reasoning, decisively: - **Delta queries plus change notifications remove the polling/latency tradeoff.** Graph gives a server-maintained cursor (`@odata.deltaLink`) and a push subscription. IMAP gives `UID`/`MODSEQ` and `IDLE`, which work but require the client to maintain state that Graph maintains for us and re-derive it after every reconnect. - **Least privilege is actually achievable.** With IMAP the credential is a mailbox password or an OAuth token for the whole mailbox; there is no way to say "this app may only touch this one mailbox". Graph application permissions are tenant-wide by default — which is worse — but `New-ApplicationAccessPolicy` narrows them back to a mail-enabled security group containing exactly the careers mailbox. That combination is the only configuration in this space that is genuinely least-privilege, and it is why Graph wins. - **One tenant for identity, mail and hosting.** A7 puts Entra ID SSO, the Graph app registration and the container platform in the same tenant, which is the largest free reduction in integration risk available (`_decisions.md` line 235). - **Python SDK exists and mail volume is I/O-light at A4 volumes** (`_decisions.md` line 91). **Rejected:** Gmail API (wrong tenant under A1 — revisit only if A1 is false). A third-party inbound-mail SaaS such as Mailgun Routes or SendGrid Inbound Parse (candidate CVs would transit a processor outside controlled infrastructure; BRD §7.4 forbids it, and it also means the careers address stops being a real mailbox a recruiter can open). Reading the mailbox from a recruiter's Outlook desktop client via a plugin (no server-side durability, no audit). IMAP/SMTP as the Phase 1 default — designed in §2.12 as the fallback, not the default, because of the least-privilege point above. ### 2.2 App registration and least-privilege scopes | Item | Value | Rationale | |---|---|---| | Registration | Single-tenant app, name `utopia-ats-mail-`. One registration **per environment** (staging registration points at a separate test mailbox, per `_decisions.md` line 233). | An environment boundary that shares a client id will eventually read production candidate mail from staging. | | Credential | **Certificate** or Entra **workload-identity federation**, not a client secret. Stored in Key Vault; the container's managed identity reads it at boot. | Secrets rotate on a calendar; certificates and federated credentials remove the "secret expired at 2am" incident class. Never in a repository file — findings §B confirms no `.env` exists today and none should be added with real values. | | Graph permission: `Mail.Read` | **Application.** Read messages and attachments. | Required for attachment bytes; `Mail.ReadBasic.All` cannot read body or attachments. | | Graph permission: `Mail.ReadWrite` | **Application.** Move processed messages to folders, set categories, set `isRead`. | The processed/quarantine folder moves in §2.9 need write. If the business rejects any mutation of the mailbox, drop this and use a database-side cursor only — stated tradeoff: recruiters then cannot see processing state in Outlook. | | Graph permission: `Mail.Send` | **Application.** Send replies as the careers mailbox. | §2.11. | | Explicitly **not** requested | `Mail.Read.All` beyond the policy scope, `User.Read.All`, `Directory.Read.All`, `Files.*`, `Sites.*`, `Calendars.*`. | Interview calendar integration is Phase 2 (`_decisions.md` module 13) and gets its own scope request then, not a pre-emptive grant now. | | Mailbox scoping | Exchange Online PowerShell: `New-ApplicationAccessPolicy -AppId -PolicyScopeGroupId ats-mail-scope@utopiabrands.com -AccessRight RestrictAccess`, where that group contains **only** the careers mailbox. Verified with `Test-ApplicationAccessPolicy` against both an in-scope and an out-of-scope mailbox, and that verification is a documented go-live check. | Without this, `Mail.Read` application permission reads **every mailbox in the tenant**. This single line is the difference between a defensible and an indefensible integration. | | Network | Egress to `graph.microsoft.com` and `login.microsoftonline.com` only, from the `web`/`worker` egress identity. Graph webhook ingress restricted to Microsoft's published notification source ranges at the platform WAF **in addition to** the token checks in §2.5. | Defence in depth; the ranges change, so the token checks remain the primary control. | | Token handling | Acquired by MSAL with in-process caching, refreshed on expiry. Never logged, never persisted to the database. `ChannelConnection.credential_ref` (`_decisions.md` module 22) stores a **Key Vault reference**, not a credential. | Findings §J: no secret values in the repository. | **Graph service limits must be verified against current Microsoft documentation at implementation time** — they change, and my knowledge has a cutoff. Design to the throttling *behaviour* rather than to a number: honour `Retry-After` on `429` and `503` absolutely, cap concurrency per mailbox at a small fixed number (start at 2), use a single-flight lock per `ChannelConnection` so two workers never sync one mailbox concurrently, and treat sustained throttling as a monitored condition rather than an error to retry blindly. ### 2.3 Sync architecture: three independent mechanisms, deliberately overlapping The requirement "never rely on webhooks alone" is met by running three mechanisms that each alone would eventually get every message, and whose combination has no single point of loss. | # | Mechanism | Cadence | Role | |---|---|---|---| | 1 | **Change notification (webhook)** | Push, seconds | Latency. Notifications are treated as *hints only* — the payload is never trusted as data. | | 2 | **Delta query** | On every hint, plus a periodic tick every 5 minutes | The authoritative incremental fetch. Delta is the only thing that ever advances the cursor. | | 3 | **Reconciliation sweep** | Every 60 minutes, plus a nightly deep pass | Catches everything the other two missed: dropped notifications, expired subscriptions, dead-lettered items, messages present in the mailbox with no `raw_intake` row. | Design rule, stated once and applied everywhere: **the webhook is a doorbell, not a delivery.** A notification body may be truncated, out of order, duplicated or spoofed. The handler's only job is to validate, record, and enqueue a delta run. ### 2.4 Initial mailbox sync (backfill) The careers mailbox will already contain history on day one. Backfill is a distinct job from steady-state sync because it has different failure and volume characteristics. 1. **Bounded by policy, not by "everything".** Operator supplies a `backfill_from` date at connection setup. Default recommendation: **90 days**, because older applications are stale for active requisitions and each one carries retention obligations the moment it is ingested (`_decisions.md` retention decision). Ingesting five years of CVs on day one creates a retention liability and a review queue nobody will clear. Confidence: **Medium** — this is a business call; surface it as a setup choice, do not hardcode. 2. Page `GET /users/{mailbox}/mailFolders/{folderId}/messages` filtered on `receivedDateTime ge {backfill_from}`, `$select` restricted to the fields actually used, `$top=50`, following `@odata.nextLink`. Attachments are **not** expanded inline — a separate per-message attachment job (§2.9) keeps page responses small and makes each attachment independently retryable. 3. Each page is processed in its own transaction: insert `raw_intake` rows and enqueue attachment-download jobs, then persist the page cursor. Postgres-backed queueing (`procrastinate`, `_decisions.md` line 131) makes the insert and the enqueue commit together, so a crash mid-backfill cannot produce an intake row with no job or a job with no row. 4. Backfill runs on the `mail` queue at **low concurrency (1)** and yields to steady-state sync: if a delta run is pending, the backfill job re-enqueues itself with a short delay. New applications must never sit behind a five-year backfill. 5. On completion, call `.../messages/delta` once to obtain the initial `deltaLink`, store it on `ChannelConnection.delta_cursor`, and only then create the webhook subscription. Order matters: a cursor established after the subscription can miss messages that arrive in the gap; establishing the cursor first means the first delta run picks them up. 6. Backfill state is resumable and idempotent — a re-run inserts nothing new because of the uniqueness rules in §2.6. ### 2.5 Incremental sync via delta queries - Cursor: `ChannelConnection.delta_cursor` holds the opaque `@odata.deltaLink`. It is **never parsed** and never reconstructed by hand. - A delta run: `GET {delta_cursor}` → for each `value` entry, dispatch by shape (`@removed` → mark the `raw_intake` row's provider state as removed-at-source; keep the intake row, because a deleted mail does not un-apply a candidate); follow `nextLink` pages; on the final page persist the new `deltaLink` **in the same transaction as the last page's rows**. - **Cursor invalidation.** Graph can reject a stale or too-old delta token (`410 Gone`, `syncStateNotFound` / `resyncRequired`). Handling: clear the cursor, run a **bounded re-sync** over `receivedDateTime ge now() - 7 days`, obtain a fresh `deltaLink`, and raise a `channel_resync` operational event. Bounded, not full — a full re-sync of a five-year mailbox on a token expiry is a self-inflicted incident, and the reconciliation sweep in §2.7 will find anything older. - **Single-flight.** A `procrastinate` queueing lock keyed on `channel:{id}` guarantees one delta run per connection at a time. Without it, two concurrent runs both advance the cursor and one of them loses the messages the other saw. - **Cursor is advanced only on a fully-persisted page.** If any row in a page fails to insert for a non-idempotency reason, the transaction rolls back, the cursor is unchanged, and the page is retried. Losing a page is a correctness bug; re-reading a page is free (§2.6). ### 2.6 Idempotency and duplicate prevention Four layers. Layers 1-2 are database facts, not code paths. | Layer | Mechanism | Catches | |---|---|---| | 1 | `UNIQUE (channel_id, external_message_id)` on `raw_intake`, where `external_message_id` = the Graph **immutable message id** (`Prefer: IdType="ImmutableId"` on every request) and `channel_id` identifies the mailbox. | Webhook redelivery, delta re-read, reconciliation overlap, backfill re-run. Insert uses `ON CONFLICT (channel_id, external_message_id) DO NOTHING RETURNING id`; an empty return means "already have it, nothing to do". | | 2 | `UNIQUE (channel_id, payload_sha256)` on `raw_intake` (`_decisions.md` intake decision). | The same envelope arriving with a different provider id — e.g. after a mailbox migration or an IMAP fallback switch where ids are not stable. | | 3 | `raw_intake_attachment.sha256` + content-addressed storage keys (§6.1). | The same CV attached twice in one mail, or re-sent in a follow-up mail. The blob is stored once; a second `raw_intake_attachment` row points at the same object and is marked `duplicate_of_attachment_id`. | | 4 | `duplicate_candidate_pair` + the global partial unique index on `candidate_email.address_normalised` (`_decisions.md`). | The same *person* applying twice through different channels. This is identity dedupe, not message dedupe, and it is deliberately a separate layer with a human reviewer. | **Why the immutable id specifically.** The default Graph message `id` changes when a message is moved between folders. Since §2.9 moves processed messages into a `Processed` folder, using the mutable id would make every processed message look like a brand-new message on the next reconciliation sweep. `Prefer: IdType="ImmutableId"` is therefore mandatory on every mail request, and it belongs in the HTTP client's default headers so it cannot be forgotten. This is the single most likely implementation mistake in the whole email integration. ### 2.7 Webhook subscriptions, renewal, failure recovery and reconciliation #### Subscription lifecycle | Concern | Design | |---|---| | Create | `POST /subscriptions` with `changeType: created`, `resource: users/{mailbox}/mailFolders/{inboxId}/messages`, `notificationUrl: https:///api/v1/webhooks/graph/mail`, `lifecycleNotificationUrl` pointing at the same host, a per-subscription random `clientState` (stored hashed in Key Vault, compared in constant time), and `expirationDateTime`. | | Validation handshake | Graph immediately POSTs a `validationToken` query parameter; respond `200` with the raw token as `text/plain` within the provider's short timeout. This endpoint must be reachable **before** the subscription is created — a common first-deploy failure. | | Expiry | Outlook message subscriptions have a **maximum lifetime measured in days, not weeks** (at time of writing ~4230 minutes for Outlook resources; **verify against current Graph documentation**). Treat the maximum as unknown-and-short: read `expirationDateTime` from the create response and drive renewal from it rather than from a hardcoded constant. | | Renewal | A periodic `maintenance`-queue job every **15 minutes** renews any subscription whose `expirationDateTime` is inside the next **12 hours**, by `PATCH /subscriptions/{id}` with a new expiry. Renewing at half-life with a 15-minute check means a single failed renewal window is recoverable and does not lose the subscription. | | Renewal failure | 3 attempts with backoff; on continued failure **delete and recreate** the subscription, then force a delta run. Recreation is safe precisely because the notification is only a hint and the delta cursor is authoritative. Raise `channel_subscription_recreated`. | | Lifecycle notifications | Handle all three: `reauthorizationRequired` → refresh token and `PATCH` to reauthorize; `subscriptionRemoved` → recreate then force delta; `missed` → force delta immediately and log a `notifications_missed` counter. `missed` is Graph telling us it dropped notifications; ignoring it is how mail silently stops arriving. | | Secret rotation | `clientState` is regenerated on every recreate. A notification whose `clientState` matches neither the current nor the immediately-previous value is dropped and counted as `webhook_rejected`. | #### Webhook endpoint contract - Path: `POST /api/v1/webhooks/graph/mail`. **Unauthenticated by design** (Graph cannot present our credentials), therefore hardened: `clientState` constant-time comparison, subscription id must exist and be active, body size cap, per-source rate limit, and a request-id in every log line. Optionally validate the JWT in the `Authorization` header when `includeResourceData` is used — not used here, because we deliberately do not accept resource data in the notification. - Behaviour: validate → insert a `webhook_receipt` row (subscription id, provider change id, received_at, raw body, outcome) → enqueue one delta run per affected connection with a **debounce** (coalesce to at most one queued delta run per connection) → return `202` in under 1 second. No mail fetching, no parsing, no candidate work in the request path. - The endpoint returns `202` even when the enqueue is coalesced. Returning `5xx` to Graph triggers provider-side retries and eventually subscription removal; we would rather record the receipt and let reconciliation cover us. #### Reconciliation sweep — the actual safety net Two tiers, both on the `mail` queue: | Tier | Cadence | Method | |---|---|---| | Shallow | Every 60 minutes | List message ids in the Inbox and Processed folders for `receivedDateTime ge now() - 48h`, `$select=id,receivedDateTime,internetMessageId` only. Left-anti-join against `raw_intake.external_message_id`. Any provider id with no intake row is ingested. Any intake row still in `received`/`parsing` for more than 30 minutes is re-enqueued. | | Deep | Nightly, off-hours | Same comparison over `now() - 30 days`, plus: intake rows whose attachments have no blob, blobs with no `virus_scan_status`, `intake_parse_attempt` rows stuck in a non-terminal state, and dead-letter entries older than 24 hours with no operator decision. Emits a single `reconciliation_report` row with counts per anomaly class. | The nightly report having a **non-zero missed-message count is an alert, not a statistic.** If reconciliation is routinely finding messages, the webhook or delta path is broken and the sweep is masking it. #### Diagram — webhook failure and reconciliation recovery ```mermaid sequenceDiagram autonumber participant GR as Microsoft Graph participant WEB as ATS web process participant DB as PostgreSQL participant WK as ATS worker participant OPS as On-call dashboard Note over GR,WEB: Normal path GR->>WEB: POST webhook notification with clientState WEB->>DB: insert webhook_receipt, enqueue delta run (debounced) WEB-->>GR: 202 Accepted WK->>DB: claim delta job, take the per-channel single-flight lock WK->>GR: GET deltaLink GR-->>WK: messages page plus new deltaLink WK->>DB: insert raw_intake rows and advance cursor in one transaction Note over GR,WEB: Failure path A - notifications dropped or endpoint down GR--xWEB: notification lost (deploy restart or WAF block) Note over WEB: no receipt row is written, so nothing is enqueued Note over WK,DB: Hourly shallow reconciliation WK->>GR: list message ids received in the last 48 hours GR-->>WK: id list WK->>DB: left-anti-join against raw_intake.external_message_id DB-->>WK: 3 provider ids with no intake row WK->>DB: ingest the 3 missing messages, increment missed_by_webhook WK->>OPS: alert - reconciliation found missed messages Note over GR,WK: Failure path B - subscription expired or removed GR->>WEB: lifecycle notification subscriptionRemoved WEB->>DB: enqueue subscription repair and a forced delta run WK->>GR: DELETE old subscription then POST new subscription GR-->>WK: new subscription id and expirationDateTime WK->>DB: store subscription, clientState hash and expiry WK->>GR: GET deltaLink (cursor was never lost) GR-->>WK: everything that arrived while the subscription was dead Note over WK,GR: Failure path C - delta cursor rejected WK->>GR: GET deltaLink GR-->>WK: 410 Gone resyncRequired WK->>DB: clear cursor, raise channel_resync WK->>GR: bounded re-sync over the last 7 days GR-->>WK: pages plus a fresh deltaLink WK->>DB: upsert on (channel_id, external_message_id) - duplicates are no-ops ``` ### 2.8 Thread handling Email threads are **not** applications, and conflating them is a real failure mode: a candidate replying "here is the updated CV" would otherwise create a second application. | Signal | Use | |---|---| | `conversationId` (Graph) | Stored on `raw_intake.payload`. Groups intakes into a conversation for recruiter display. | | `internetMessageId`, `inReplyTo`, `references` (RFC 5322) | Stored in the payload; used as the thread key on the IMAP fallback path where `conversationId` does not exist. | | ATS-owned token | Every outbound message from the careers mailbox carries `X-Utopia-Thread: ` and a `Reply-To` of `careers+t.@utopiabrands.com` (sub-addressing) where the mailbox supports it. On inbound, the token is the **primary** thread key because it survives clients that strip `references`. | Resolution rule, in priority order: ATS token → `conversationId` → `references` chain → normalised subject plus sender within 30 days. Each inbound message still gets its **own** `raw_intake` row (arrival is a fact and must be recorded), but the resolution step (§4 step 16) proposes `attach_to_existing_candidate` rather than `create_candidate` when a thread link exists, and attaches new attachments as a **new `candidate_document` revision** rather than a new candidate. A recruiter can override in the triage UI; the override is recorded on `intake_resolution` with actor and reason. Auto-reply and vacation-message suppression: messages with `Auto-Submitted: auto-replied`, `X-Auto-Response-Suppress`, or a `precedence: bulk` header are ingested (arrival is still a fact) but resolved automatically to `reject_unusable` with reason `auto_reply`, and never enter the review queue. Without this, every out-of-office reply to a rejection email becomes a triage item. ### 2.9 Attachment handling Attachments are downloaded by a **separate job per attachment**, never inline with the message fetch. Rationale: a 40 MB attachment must not fail a page of 50 messages, and each attachment needs independent retry, scanning and quarantine state. | Case | Handling | |---|---| | Enumerate | `GET /messages/{id}/attachments?$select=id,name,contentType,size,isInline` — metadata first, bytes second. | | Download | `GET /messages/{id}/attachments/{aid}/$value` streamed **directly to object storage**, computing sha256 during the stream. Never buffered fully in worker memory. Hard cap enforced during the stream, not from the declared `size` (a declared size is attacker-controlled). | | Inline images | `isInline: true` with a `contentId` referenced by the HTML body → signature logos, not CVs. Recorded as attachments with `is_inline = true` and excluded from parse candidacy. | | `itemAttachment` (an embedded email or contact) | Recorded; not parsed in Phase 1. If it contains a nested file attachment, one level of extraction is attempted; deeper nesting is quarantined. Rationale: unbounded nesting is a zip-bomb-shaped attack. | | `referenceAttachment` (OneDrive/SharePoint link) | Recorded with the link; **not followed** in Phase 1 — following it needs `Files.Read.All`, which §2.2 deliberately does not request. Resolution is `needs_review` with a recruiter-visible reason "CV is a cloud link; download manually or ask the candidate to attach the file". Honest limitation, not a silent drop. | | **Multiple attachments** | Every attachment gets its own `raw_intake_attachment` row and its own parse pipeline. Classification (§4 step 9) decides which are CVs. | | **Multiple CVs in one email** | Fully supported and expected (agency submissions). Each CV-classified attachment produces its own `intake_parse_attempt` and its own **proposed** candidate. The intake stays in `needs_review` with `n` proposals; the recruiter confirms each independently. **Never** auto-create *n* candidates from one email — an agency PDF pack that is really one CV plus three reference letters would silently create three ghost identities. Confidence: **High** (this is the exact failure the raw-intake layer exists to prevent). | | **No CV in the email** | Not an error. The intake is recorded, body text is retained, classification finds no CV, and resolution proposes `needs_review` with reason `no_cv_attachment`. A recruiter can still create a candidate from the body text (a plain-text application) or reject it. General enquiries are rejected with reason `not_an_application`. | | Duplicate attachment | sha256 match against `raw_intake_attachment` in the same intake → second row marked `duplicate_of_attachment_id`, parsed once. sha256 match against an existing `candidate_document` → strong duplicate-identity signal fed to §4 step 15 and surfaced to the reviewer as "identical file already on file for candidate X". | | Password-protected / corrupted / unsupported | See the failure matrix in §4.3. All three end in a terminal, recruiter-visible state with a specific reason and a "request a new file" action — never a silent discard. | | Post-processing mailbox move | On terminal resolution, `PATCH` the message's `categories` (`ATS/Processed`, `ATS/NeedsReview`, `ATS/Quarantined`) and move it to the matching folder. Best-effort and idempotent: a failed move never fails the intake, and the reconciliation sweep scans the Processed folder too, so a message in the wrong folder is not lost. | ### 2.10 Diagram — inbound email to intake to candidate and application ```mermaid sequenceDiagram autonumber participant CA as Candidate mail client participant EX as Exchange Online careers mailbox participant GR as Microsoft Graph participant WEB as ATS web participant WK as ATS worker participant OS as Object storage participant DB as PostgreSQL participant REC as Recruiter CA->>EX: email with CV attachment EX->>GR: message available GR->>WEB: change notification (hint only) WEB->>DB: insert webhook_receipt and enqueue delta run WEB-->>GR: 202 WK->>GR: GET deltaLink with ImmutableId preference GR-->>WK: message envelope and headers WK->>DB: insert raw_intake ON CONFLICT DO NOTHING Note over WK,DB: unique on channel plus external_message_id, and on payload hash WK->>DB: insert raw_intake_attachment rows (metadata only) WK->>DB: enqueue one attachment_download job per attachment WK->>GR: GET attachment value (streamed) WK->>OS: PUT object at sha256-addressed key WK->>DB: set object_store_key, sha256, byte_size WK->>DB: enqueue malware_scan WK->>OS: fetch bytes for scanning WK->>DB: virus_scan_status = clean, enqueue parse WK->>DB: insert intake_parse_attempt (append-only) Note over WK: 17-step pipeline of section 4 runs here WK->>DB: parse attempt succeeded, parsed jsonb plus per-field confidence WK->>DB: identity resolution and duplicate scan write proposals WK->>DB: raw_intake.state = needs_review REC->>WEB: open triage queue, review parsed fields and the CV side by side REC->>WEB: confirm - create candidate and apply to JOB-1042 WEB->>DB: BEGIN WEB->>DB: insert intake_resolution (decision_mode = human, decided_by_user_id) WEB->>DB: insert candidate with created_from_raw_intake_id NOT NULL WEB->>DB: insert candidate_email (CHECK on normalised address) WEB->>DB: insert job_application with raw_intake_id and source_channel_id WEB->>DB: insert candidate_document from the attachment WEB->>DB: enqueue ats_scoring for the application WEB->>DB: COMMIT Note over DB: deferrable contactability trigger fires at COMMIT WEB->>DB: audit_event rows written by trigger using SET LOCAL actor WEB-->>REC: candidate CAN-5219 created, application APP-30412 created WK->>DB: score against pinned job_version and scoring_config_version WK->>DB: insert ats_result with is_current true plus criterion rows ``` ### 2.11 Sending replies through the careers mailbox | Concern | Design | |---|---| | Transport | `POST /users/{careers-mailbox}/sendMail` with `saveToSentItems: true`, so every outbound message is visible in the mailbox a recruiter can open. For a genuine threaded reply, `POST /messages/{id}/reply` preserves `references` and `conversationId`; use it whenever the outbound message answers a specific inbound message. | | Record of intent | Every send is an `OutboundMessage` row (`_decisions.md` module 5) written **before** the provider call: recipient, template version, rendered subject and body, `application_id`/`candidate_id`, `thread_token`, `status`, `provider_ref`, `attempt_no`. The row is the record; the provider call is an attempt against it. | | Transactional enqueue | The row insert and the send job enqueue commit together (Postgres queue). No dual-write, so "we recorded a rejection email that was never sent" and "we sent an email we have no record of" are both impossible. | | Idempotency | Idempotency key `outbound:{OutboundMessage.public_id}`, plus a `sent_at IS NULL` guard checked inside the job's transaction. A retried job that finds `provider_ref` already set is a no-op. Graph `sendMail` is not idempotent server-side, so this guard is the only thing preventing a duplicate rejection email — the single most reputationally damaging duplicate this system can produce. | | Human gate | Candidate-facing rejection and offer communications require a human action to enter `queued`. `_decisions.md` makes AI structurally incapable of writing domain state and forbids terminal-negative transitions by a non-human actor; this document adds the matching rule on the delivery side: **no AI-originated or rule-originated job may enqueue a candidate-facing rejection.** Enforced by the send service refusing an `OutboundMessage` whose `category = rejection` and `created_by_actor_kind <> 'user'`. **`'user'`, not `'human'`** — `actor_kind` is one closed set repository-wide, `CHECK in ('user', 'system', 'integration', 'ai_agent')` (Part 2 §9.5/§28.1, `_decisions.md` audit and enum-strategy decisions), and `'user'` is the only member that carries an `actor_user_id` identifying a real person. A guard written against `'human'` would compare against a value the CHECK cannot hold, so it would either always refuse or always pass depending on the comparison direction — see the vocabulary note in *Status / Scope*. | | Suppression list | Checked before every send: unsubscribed, hard-bounced, retention-pseudonymised, and legal-hold candidates. A pseudonymised candidate has no deliverable address by construction, and the send fails closed rather than mailing a token. | | Rate and batch | Per-mailbox send concurrency 1, with a modest per-minute cap. Bulk sends (e.g. 200 rejections after a requisition closes) are individual queued jobs, never one loop — so one bad address cannot abort the batch, and the throttle is enforced by the queue rather than by `sleep`. | | Templates | `config.render_template()` with a **pinned `TemplateVersion`** recorded on the row, so "what exactly did we tell this candidate in March" is answerable. | | Attachments outbound | Offer letters go out as generated PDFs from `files`, referenced by `StoredFile` id on the outbound row. Never a signed URL to storage in an email to a candidate — candidate-facing document access goes through `candidate_access_token` (`_decisions.md`), which has expiry and revocation. | #### Delivery status — what is actually knowable, stated honestly Graph `sendMail` returning `202` means **Exchange accepted the message for delivery**. It does not mean delivered, and there is no Graph webhook for delivery. Overclaiming here would mislead recruiters into believing a candidate received a rejection they never got. The state machine therefore names its own uncertainty: | State | Meaning | Set by | |---|---|---| | `draft` | Row exists, not yet approved for sending | Application | | `queued` | Approved; job enqueued | Human action (or a rule, for non-candidate-facing mail) | | `accepted_by_provider` | Graph returned success | Send job | | `delivery_unconfirmed` | Accepted, no DSN after 24 hours | Timer job. **This is the normal terminal state for a successful send** and the UI must say "sent" rather than "delivered". | | `bounced_hard` | NDR classified as permanent (5.x.x) | Inbound NDR classifier | | `bounced_soft` | NDR classified as transient (4.x.x) | Inbound NDR classifier; one automatic re-send after 6 hours, then `failed` | | `rejected_by_provider` | 4xx/5xx from Graph after max attempts | Send job | | `suppressed` | Blocked by the suppression list before sending | Send service | **Bounce detection is the same inbound pipeline, reused.** NDRs arrive in the careers mailbox as ordinary messages. The inbound classifier detects `multipart/report; report-type=delivery-status`, extracts the DSN status code and the original recipient plus our `X-Utopia-Thread` token, links back to the `OutboundMessage`, and sets the bounce state. This is the reason §2.8's ATS-owned thread token exists: without it, matching an NDR back to the right outbound row is guesswork. NDR intakes are resolved to `reject_unusable` with reason `bounce_report` so they never enter the recruiter triage queue. For operational investigation only, Exchange Online message trace (Reporting API) can confirm delivery to the recipient's server. It is a **manual diagnostic**, not a per-message pipeline — polling it for every send is disproportionate at A4 volumes. Open-tracking pixels and click-tracking on candidate mail are **rejected**: they are candidate surveillance without a lawful basis, and they degrade deliverability. Audit: every state transition writes an `audit_event` with `actor_kind` (`user` for a recruiter action, `system` for a timer or classifier transition — the closed set is in *Status / Scope*), `entity_table = 'outbound_message'`, `before`/`after`, and the `request_id`. Body content of candidate-facing mail is classified `personal`, so the audit payload stores the **template version and a body hash**, not the rendered body — the rendered body already lives on the outbound row where the retention purge can reach it (`_decisions.md` audit PII rule). #### Diagram — outbound candidate communication with delivery tracking ```mermaid sequenceDiagram autonumber participant REC as Recruiter participant WEB as ATS web participant DB as PostgreSQL participant WK as ATS worker participant GR as Microsoft Graph participant EX as Exchange Online participant CA as Candidate REC->>WEB: send "interview invitation" for APP-30412 WEB->>DB: BEGIN WEB->>DB: check suppression list (unsubscribed, bounced, pseudonymised) WEB->>DB: insert outbound_message status queued, pinned template_version, thread_token WEB->>DB: enqueue send_email keyed on the outbound message public id WEB->>DB: COMMIT WEB-->>REC: queued for sending WK->>DB: claim job, re-check provider_ref IS NULL WK->>GR: POST sendMail with X-Utopia-Thread header and saveToSentItems GR-->>WK: 202 Accepted with request id WK->>DB: status accepted_by_provider, provider_ref, sent_at WK->>DB: audit_event (actor_kind user, template version, body hash) GR->>EX: place in Sent Items EX->>CA: deliver Note over WK,DB: 24h timer with no DSN WK->>DB: status delivery_unconfirmed Note over REC: UI shows "Sent" - never "Delivered" Note over EX,CA: Failure path - mailbox does not exist EX->>EX: generate NDR 5.1.1 to the careers mailbox EX->>GR: NDR message available GR->>WEB: change notification WK->>GR: delta fetch WK->>DB: insert raw_intake for the NDR WK->>DB: classify as delivery-status report, extract thread token and DSN code WK->>DB: outbound_message status bounced_hard, add address to suppression list WK->>DB: resolve intake as reject_unusable reason bounce_report WK->>DB: create worklist task "invitation to CAN-5219 bounced - obtain a valid address" WK-->>REC: in-app notification ``` ### 2.12 IMAP/SMTP fallback design Required if A1 or A3 fails (non-M365 provider, or IT refuses application permissions). Same ports (§7), a different adapter. Confidence: **High** that this works; **Medium** on effort — budget **5-8 developer-days** for the adapter plus its own reconciliation tests. | Concern | Graph | IMAP/SMTP fallback | |---|---|---| | Auth | Entra app + certificate, mailbox-scoped policy | **OAuth 2.0 XOAUTH2 where the provider supports it.** Basic auth with a stored mailbox password is a last resort and must be escalated as an accepted risk, not adopted quietly — it is a full-mailbox credential with no scoping. | | Push | Change notifications | `IDLE` on a long-lived connection, one connection per mailbox, with a 29-minute re-`IDLE` cycle and reconnect-with-backoff. Same doorbell semantics: `IDLE` wakes a fetch, it does not deliver data. | | Cursor | Opaque `deltaLink` | `UIDVALIDITY` + `UIDNEXT` per folder; `CONDSTORE`/`HIGHESTMODSEQ` when the server advertises it. **A `UIDVALIDITY` change invalidates all stored UIDs** and forces a bounded re-sync — the exact analogue of a `410 resyncRequired`, and the fallback's sharpest edge. | | Stable id | Immutable id | `Message-ID` header, normalised. Weaker: it is client-generated, occasionally missing and occasionally duplicated. Compensated by making `UNIQUE (channel_id, payload_sha256)` (dedupe layer 2) the primary defence rather than the backstop, with a synthetic id `sha256(Message-ID || Date || From || Subject || body-hash)` when `Message-ID` is absent. | | Fetch | REST, `$select` | `FETCH (BODY.PEEK[])` — `PEEK` is mandatory so reading does not set `\Seen` and change what a recruiter sees in the mailbox. Parse MIME with Python `email` / `mailparser`. | | Attachments | Per-attachment endpoint | Extracted from the MIME tree in one pass. Hard limits on part count and nesting depth (recommend 50 parts, depth 5) as an anti-zip-bomb measure — no equivalent server-side guard exists. | | Threading | `conversationId` | ATS thread token → `In-Reply-To`/`References` → subject+sender heuristic. Materially worse; the ATS-owned token carries most of the weight. | | Processed state | Folder move + categories | `UID MOVE` into `ATS/Processed` (`UID COPY` + `\Deleted` + `EXPUNGE` on servers without MOVE). `EXPUNGE` is the only destructive operation in the whole integration and needs an explicit config flag, default off. | | Send | Graph `sendMail` | SMTP submission on 587 with STARTTLS, one connection per batch, DKIM/SPF/DMARC alignment verified for the careers domain **before** first production send. Same `OutboundMessage` state machine, same idempotency guard. | | Reconciliation | §2.7 | Unchanged in shape and **more important**: `SEARCH SINCE ` for the id set, anti-joined against `raw_intake`. IMAP loses notifications more often than Graph does. | The fallback's design cost is concentrated in exactly two places — id stability and cursor invalidation — and both are already covered by mechanisms the Graph path uses anyway (payload-hash uniqueness, bounded re-sync, reconciliation sweep). That is the payoff for building those as first-class layers rather than as Graph-specific workarounds. --- ## 3. Careers website integration ### 3.1 Shape of the integration The careers website is a **client of a versioned REST API**, exactly like the recruiter SPA. Three interaction directions: | Direction | Mechanism | Why | |---|---|---| | Job listings (ATS → site) | **Pull.** `GET /api/v1/public/postings` with `ETag`/`If-None-Match`, 60-second site-side cache. | Pull survives an ATS outage — the site serves its cache and stays up. Push would make a failed webhook silently freeze the listings, and reconciling that requires a pull anyway. | | Cache invalidation (ATS → site) | Optional `POST` to a site-supplied webhook on publish/update/close, with HMAC signature. Best-effort. | Reduces publish latency from 60s to ~1s. It is an optimisation; correctness never depends on it. | | Application submission (site → ATS) | `POST /api/v1/public/postings/{posting_public_id}/applications`, multipart, service-authenticated, idempotency-keyed. | The only write the site is permitted to make. | ### 3.2 API versioning and contract - Namespace `/api/v1/`, per `_decisions.md` (line 202). Public endpoints sit under `/api/v1/public/` with their **own** DRF authentication class, throttle scope, serializers and OpenAPI tag — so a change to internal serializers cannot accidentally widen the public payload. This separation is the whole point: the public contract is a deliberate, reviewed, narrow projection. - Breaking-change policy: additive within `v1`; a breaking change mints `/api/v2/` and `v1` is supported for **90 days** with a deprecation date in a `Sunset` header. Confidence: **High** — there is exactly one external consumer under A6, so this is cheap. - Schema published by drf-spectacular; the careers site's client is **generated** from it, not hand-written. - One error envelope across the whole API: `{ "error": { "code", "message", "field_errors", "request_id" } }`. `request_id` is the same value in the ATS logs, which turns "the site got an error" into a single log query. ### 3.3 Shared database access between the public site and the ATS is forbidden **Rule: the careers website has no database credential, no connection string, no read replica and no direct table access — now or in any later phase. Its only interface is `/api/v1/public/`.** Six reasons, in order of severity: 1. **Every invariant in `_decisions.md` Part 2 lives in the application service layer or in database triggers that need transaction-local context.** History triggers read `current_setting('app.actor_user_id')`, set by ATS middleware. A direct-writing careers site would write history rows attributed to `system` with `actor_unknown = true`, or would insert a `candidate` row without going through the resolution flow. The raw-intake-before-candidate ordering, the deferrable contactability trigger, the audit chain and the reapplication cooling-off trigger are all premised on writes arriving through one application role from one codebase. Shared access does not weaken these guarantees — it voids them. 2. **The public site is the most likely component to be compromised.** It is internet-facing, CMS-shaped, and owned by a different team under A6. A database credential there is a credential to the entire candidate PII estate — `candidate_email`, `candidate_document` extracted text, `offer_version` compensation, the lot. Through the API, a compromise yields only what the public endpoints expose: posting reads and application writes. 3. **Column-level `GRANT`s are the enforcement mechanism for append-only `ats_result` and `audit_event`** (`_decisions.md`). Those grants are attached to *one* application role. A second role means either duplicating and maintaining the grant matrix in two places, or — what actually happens — granting the site broader privileges "for now". 4. **Schema coupling freezes migrations.** The site's queries become an undocumented consumer of table shapes. Splitting candidate from application, versioning requirements, and the merge re-pointing model all change table shapes; each one would then require a coordinated deploy with a team that does not report to this project. Two developers cannot absorb that. 5. **No chokepoint for authorization, validation, rate limiting, file scanning or audit.** All five exist in the API layer. SQL bypasses all five simultaneously. 6. **Data-protection scope.** Under A6 the careers site is operated by a different team, possibly on different infrastructure. Direct database access puts that infrastructure inside the processing boundary for all candidate data, not just submissions — a materially larger DPIA scope for zero engineering benefit. **Also forbidden, for the same reasons:** a read-only replica for the site ("just for job listings" — job listings are a cached JSON endpoint, which is cheaper and faster anyway); a shared object-storage bucket with site write access (uploads go through the API so they are scanned before they exist as a candidate document); and a nightly SQL dump to the site's database (a stale unencrypted copy of candidate PII outside the retention machinery). ### 3.4 Service-to-service authentication | Layer | Mechanism | |---|---| | Primary | **mTLS** between the careers site and the ATS ingress where the platform supports it, with the client certificate pinned to a named service identity. | | Credential | A per-environment **service principal** with a single scoped permission, `public_application:create`, and no user identity. Presented as a short-lived bearer token from a client-credentials exchange — **not** a static API key. If the platform makes token exchange awkward, the fallback is a static key with a documented 90-day rotation and two-key overlap, recorded as an accepted risk. | | Request integrity | HMAC-SHA256 over `timestamp + method + path + sha256(body)` in an `X-Utopia-Signature` header, timestamp window ±300 seconds, nonce cached in Redis for the window to block replay. Belt and braces with mTLS, and the only protection if the fallback static key is in use. | | Authorization | The service principal can create applications and read public postings. It **cannot** read candidates, cannot read applications back, cannot list submissions and cannot read scores. The submission response contains only the external reference and status. This is the important part: a compromised site cannot enumerate the candidate base. | | Audit | Every public submission writes an `audit_event` with `actor_kind = 'integration'` and the service principal id. | ### 3.5 Submission endpoint: validation, scanning, limits, idempotency `POST /api/v1/public/postings/{posting_public_id}/applications`, `multipart/form-data`. | Control | Specification | |---|---| | Input validation | DRF serializer, allowlist fields only, unknown fields **rejected** (not ignored) so a site-side bug surfaces immediately. Name 1-200 chars; email against the same regex as the `candidate_email` CHECK (`_decisions.md`) so the API and the database agree — a mismatch means the API accepts what the database will refuse; phone normalised to E.164 or rejected; free-text fields length-capped and stored raw. **No sanitisation on write** — `_decisions.md` requires the original be preserved, and findings §E requires escaping at the rendering layer. Both are true simultaneously and neither substitutes for the other. | | File constraints | Max 5 files, each ≤ 10 MB, total ≤ 25 MB. Accepted true types: PDF, DOCX, DOC, RTF, ODT, TXT. **True type from magic bytes** (`libmagic`), never from the extension or the client `Content-Type`; a mismatch between declared and detected type is logged as a suspicious-upload signal. Filenames are stored raw for display and **never** used to build a storage key (§6.1). | | Streaming | Files stream to a **quarantine prefix** in object storage with a size cap enforced during the stream. Nothing is written to the worker filesystem. | | File scanning | Malware scan is **synchronous-blocking on the response only for a fast verdict path with a 5-second budget**; on timeout the submission is accepted and the file stays quarantined until the async scan completes. The candidate is never made to wait on a slow scanner, and an unscanned file is never readable by a recruiter — quarantine-prefix objects are not signable (§6.4). | | Rate limiting | Redis token buckets (`_decisions.md`: Redis is cache and rate-limit only), layered: per source IP 5/hour and 20/day; per normalised email 3/day per posting and 10/day overall; per service principal 300/hour as a circuit breaker against a looping site bug; global public-endpoint ceiling. Exceeded → `429` with `Retry-After`. | | CAPTCHA / abuse | A privacy-preserving challenge on the **site**, verified **server-side** by the ATS (`Cloudflare Turnstile` recommended: no cookies, no behavioural profiling, and `siteverify` is a single server call). The ATS trusts nothing the site asserts about the challenge — it verifies the token itself against the provider, checks the `action` and `hostname` claims, and single-uses the token. **Verifier-unavailable policy:** fail *open* into a stricter mode (rate limits divided by 5, submissions flagged `abuse_check_unavailable` and forced to `needs_review`) rather than fail closed. A CAPTCHA provider outage must not stop real people applying for jobs; the honest tradeoff is a slightly noisier review queue for the duration. | | Honeypot + timing | A hidden field that must stay empty, and a minimum form-fill time of 3 seconds. Cheap, no privacy cost, catches naive bots before the CAPTCHA quota is spent. | | Idempotency | Mandatory `Idempotency-Key` header (site-generated UUID per submission attempt). Stored in `public_submission_idempotency (key, service_principal_id, request_fingerprint, response_status, response_body, raw_intake_id, created_at)` with `UNIQUE (service_principal_id, key)`. A replay with the same key **and** the same fingerprint replays the stored response; the same key with a *different* fingerprint returns `409 Conflict`. Retention 30 days. This is what makes the site's own retry-on-timeout safe. | | Duplicate submissions | Three distinct cases, three different answers. (a) *Network retry* — same idempotency key → replayed response, one intake. (b) *Impatient double-click, new key* — caught by `raw_intake` `UNIQUE (channel_id, payload_sha256)`; the second insert conflicts and the first intake's reference is returned, so the candidate sees success both times. (c) *Genuine reapplication* — passes to the reapplication rule (`_decisions.md`: one live application per candidate+job, cooling-off, `attempt_no`), which may return `409` with a plain-language reason such as "you already have an active application for this role". | | Response | `202 Accepted` with `{ external_application_id, posting_public_id, received_at, status: "received" }`. **`202`, not `201`** — nothing durable has been created except a `raw_intake` row, and promising a created application would be a lie: parsing may fail and a recruiter may reject the submission. | | Source attribution | `source_channel_id` → `ref.source_channel` value `careers_website`. `raw_intake.payload` captures referrer, UTM parameters, posting id, site locale and a coarse user-agent class. On resolution, `job_application.raw_intake_id` carries the attribution forward, so source reporting is a single FK hop (`_decisions.md`). No IP address is stored in `payload` beyond what the abuse controls need, and that copy expires with the idempotency record. | | External application id | `external_application_id` = the `raw_intake.public_id` (UUIDv7), returned to the site and shown to the candidate. Deliberately **not** `job_application.reference_code` (`APP-30412`), which `_decisions.md` classifies as internal-only and enumerable, and which does not exist yet at `202` time. | ### 3.6 Job publish, update and closure flows | Flow | Mechanism | Failure handling | |---|---|---| | **Publish** | HR-admin publishes a `job_version` → `job_posting` row created for the `careers_website` platform with state `published` and a pinned `job_version_id`. It becomes visible in `GET /api/v1/public/postings` on the next cache cycle; the optional invalidation webhook makes it near-immediate. | Webhook failure is logged, not retried past 3 attempts — the 60-second pull covers it. Publishing is never blocked by the site being unreachable. | | **Update** | A content change mints a **new `job_version`** (immutable version rows, `_decisions.md`) and re-points the posting. `ETag` changes; the site refetches. | The posting always resolves to exactly one `job_version`, so an application submitted at 10:00 pins the text the applicant actually read, even if the description changed at 10:01. This is the "defensible versus indefensible rejection" property. | | **Closure** | Posting state → `closed`, `closed_at` set. Removed from the list endpoint. `GET /api/v1/public/postings/{id}` returns `410 Gone` with a body naming the closure reason class (`filled` / `withdrawn` / `expired`) so the site can render a proper page rather than a 404. | **Submissions to a closed posting are accepted, not rejected**, when they arrive within a 10-minute grace window — the candidate was filling the form when it closed and the fault is not theirs. The intake is flagged `posting_closed_at_submission` and routed to `needs_review` for talent-pool consideration. After the grace window, `409` with a plain-language message and a link to open roles. Confidence: **Medium** — the grace window length is a business preference; make it configurable. | | **File upload** | Part of the same multipart submission, not a separate pre-signed upload. | A pre-signed direct-to-storage upload was considered and **rejected** for Phase 1: it hands an internet-facing site a storage write capability, and it makes the "scan before it is ever a candidate document" guarantee racy. At 10 MB and A4 volumes the proxied upload costs nothing. Revisit only if uploads exceed ~50 MB or volume exceeds ~10k/day. | | **Listing payload** | Only fields intended for the public: title, department, business unit, location, employment type, description, posted date, closing date, and the salary range **only where `job_version` marks it publishable**. Never `job_requirement` weights, never internal grade, never `scoring_config` binding. | Weighted requirements are scoring inputs. Publishing them tells applicants exactly what to keyword-stuff, which corrupts the score. | ### 3.7 Diagram — website application submission ```mermaid sequenceDiagram autonumber participant AP as Applicant browser participant SITE as Careers website participant CAP as CAPTCHA verifier participant API as ATS public API participant RD as Redis participant OS as Object storage participant DB as PostgreSQL participant WK as ATS worker AP->>SITE: GET job listing SITE-->>AP: page rendered from the 60s cache of /api/v1/public/postings AP->>SITE: submit form with CV file and challenge token SITE->>API: POST /api/v1/public/postings/{id}/applications Note over SITE,API: mTLS, bearer from client credentials, HMAC signature, Idempotency-Key API->>RD: check nonce and rate-limit buckets (IP, email, principal) RD-->>API: within limits API->>CAP: verify challenge token server-side CAP-->>API: valid for action apply and the expected hostname API->>DB: SELECT posting - is it published or inside the grace window DB-->>API: published, pinned job_version 7 API->>API: validate fields, detect true file type from magic bytes API->>OS: stream file to the quarantine prefix, hashing while streaming OS-->>API: object key and sha256 API->>DB: BEGIN API->>DB: insert idempotency record API->>DB: insert raw_intake ON CONFLICT (channel_id, payload_sha256) DO NOTHING API->>DB: insert raw_intake_attachment with quarantine key and sha256 API->>DB: enqueue malware_scan API->>DB: insert audit_event actor_kind integration API->>DB: COMMIT API-->>SITE: 202 with external_application_id (raw_intake public_id) SITE-->>AP: confirmation page showing the reference WK->>OS: fetch quarantined bytes WK->>WK: malware scan alt clean WK->>OS: move object to the active prefix WK->>DB: virus_scan_status clean, enqueue parse Note over WK,DB: the 17-step pipeline of section 4 runs here else infected WK->>OS: move object to the infected prefix (never deleted in Phase 1) WK->>DB: virus_scan_status infected, raw_intake.state quarantined WK->>DB: security audit_event, notify security owner Note over WK: terminal - no candidate is ever created, applicant is not told why end ``` Note on that last point: the applicant receives the same neutral confirmation whether the file was clean or infected. Telling a submitter "your file was flagged as malware" is a free oracle for tuning an evasion attempt. --- ## 4. CV and AI processing pipeline (§15) ### 4.1 Structure One asynchronous pipeline, **17 steps**, from an attachment arriving to a scored application. Each step is a separately queued, separately retryable job that reads and writes explicit rows — not a single long function. Rationale: a 17-stage in-process function fails as a unit, so an OCR timeout would discard the successful download, scan and extraction that preceded it. Every step, without exception, obeys four rules: 1. **Idempotent.** Re-running a step with the same inputs produces the same row or no new row. 2. **Append-only where it matters.** `intake_parse_attempt` is append-only (`_decisions.md`), so a re-parse with a better parser is a new attempt, never an overwrite. The original payload and blob are never mutated. 3. **Fails to a named, recruiter-visible state.** No silent drops. Findings §F establishes that the prototype has no representation for "arrived but cannot become a candidate"; that state is the point of this design. 4. **Records its own version.** `parser_name`, `parser_version`, `ai_model_version`, `prompt_template_version` on the attempt, so any output is attributable and replayable. Column key: **DET** = deterministic (same input, same output, no model). **AI** = model-based. **HYB** = deterministic path with a model fallback, explicitly labelled per field. ### 4.2 The 17 steps | # | Step | Type | Input | Output | Queue | Notes | |---|---|---|---|---|---|---| | 1 | Attachment enumeration | **DET** | `raw_intake` | one `raw_intake_attachment` per part, inline/item/reference classified | `ingest` | Part count and nesting depth capped. Inline signature images excluded from parse candidacy. | | 2 | Blob persist, hash, content dedupe | **DET** | provider attachment stream | object in storage, `sha256`, `byte_size`, `object_store_key`; `duplicate_of_attachment_id` on a hash hit | `ingest` | Streamed, size-capped mid-stream, never buffered whole. A hash hit skips steps 3-13 entirely and reuses the existing `intake_parse_attempt`. | | 3 | Malware scan | **DET** | blob | `virus_scan_status` ∈ `pending`/`clean`/`infected`/`unscannable`; infected → `quarantined` (terminal) | `ingest` | Scanner unavailable → `pending`, retried; the blob stays in the quarantine prefix and is not signable. Never "assume clean on scanner failure". | | 4 | True-type detection and structural safety gate | **DET** | blob | `detected_mime`, `is_encrypted`, `page_count`, `has_macros`, `declared_vs_detected_mismatch` | `parse` | Magic bytes, not extension. Rejects at the gate: encrypted, over the page cap (recommend 100), macro-bearing DOCM, or a type not on the allowlist. Runs before any parser touches the file, because the parser is the attack surface. | | 5 | Native text and layout extraction | **DET** | blob + detected type | `extracted_text`, `layout_metadata` jsonb, `char_count`, `text_layer_ratio` | `parse` (Phase 2: `untrusted`) | PDF via pdfplumber/PyMuPDF; DOCX via python-docx/mammoth; RTF/ODT/TXT via their own readers. Hard wall-clock and memory caps per document. `_decisions.md` line 271 flags this as the most likely early process-split trigger — parsers over attacker-supplied files are an RCE surface. | | 6 | Text sufficiency evaluation | **DET** | step 5 output | `needs_ocr` boolean plus the reason | `parse` | Thresholds, not a model: `char_count < 300`, or `text_layer_ratio < 0.1`, or a page count with near-zero extractable characters. Held in versioned config, not hardcoded, so tuning is attributable. | | 7 | OCR fallback | **AI** (local OCR model, versioned) | rasterised pages | `extracted_text`, `ocr_confidence` per page, `ocr_engine_version` | `parse` | Only when step 6 says so. Page cap and per-page timeout. Runs entirely in our infrastructure — CV images do not go to a third party for OCR. Low mean confidence marks the whole attempt `partial`, which forces human review rather than trusting a guess. | | 8 | Normalisation and sectioning | **DET** (AI fallback) | text | normalised text, section offsets (`experience`, `education`, `skills`, `contact`) | `parse` | Unicode NFKC, de-hyphenation, header/footer stripping, column-order repair from `layout_metadata`. Rule-based first because it is free and explainable; the AI section-labelling fallback fires only when the rules find fewer than two sections, and that fact is recorded on the attempt. | | 9 | Document-type classification | **AI** | normalised text + filename + section signals | `document_kind` ∈ `cv`/`cover_letter`/`certificate`/`portfolio`/`id_document`/`not_a_cv`, with confidence | `ai` | This is what makes "multiple attachments, one of which is a CV" work, and what stops a reference letter becoming a candidate. Cheap deterministic pre-filters (a 40-page slide deck, a 2 KB text file) short-circuit before the model call. Low confidence → `needs_review`, never a guess. | | 10 | Deterministic entity extraction | **DET** | normalised text | emails, phones, URLs, LinkedIn handles, dates, postal locations — each with a character offset | `parse` | Regex and libphonenumber. Runs **before** the model and its results **win** on conflict (step 12), because a regex-matched email address is more reliable than a generated one and cannot hallucinate. | | 11 | Structured field extraction | **AI** | normalised text + section offsets | schema-constrained JSON: name, contacts, employment history, education, skills, total experience — each field with a confidence and a source offset | `ai` | Strict output schema (JSON-schema-constrained decoding or a validated retry). Every invocation writes an `AiRun` with capability, model id and version, prompt template version, input reference, raw output, tokens, cost, latency (`_decisions.md` AI boundary). Output is a **suggestion**, never a domain write. | | 12 | Reconciliation, confidence, provenance | **DET** | steps 10 + 11 | merged field set, per-field `value`, `confidence`, `source` ∈ `regex`/`model`/`layout`, `char_offset` | `parse` | Precedence: deterministic > model. Disagreement is recorded, not resolved silently. **Fields below the confidence floor are left empty rather than guessed** — `_decisions.md` line 270 flags the gap between BRD §11's "no re-keying" and real-world parsing accuracy, and this is where that honesty is implemented. An empty field a recruiter fills is recoverable; a confidently wrong employer name is not. | | 13 | Skill canonicalisation | **HYB** | extracted skill labels | `candidate_skill` proposals with `skill_id` or `raw_label`, `confidence`, `source` | `parse` | Exact → alias (`ref.skill_alias`) → trigram similarity above threshold → **AI suggestion for unmapped labels only**, proposed as a new alias for admin approval, never auto-added to the vocabulary. `raw_label` is always retained (`_decisions.md`: parser output that maps to nothing must still be storable). | | 14 | Identity resolution proposal | **DET** | reconciled fields | proposal ∈ `new_candidate` / `existing_candidate(id)` / `ambiguous`, with per-signal evidence | `ingest` | Normalised email exact → E.164 phone exact → normalised LinkedIn URL → `candidate_document.sha256` identical → name+employer trigram. Deterministic and fully explainable; the reviewer sees which signal fired. `confirmed_distinct` pairs are skipped (`_decisions.md`). | | 15 | Duplicate pair detection | **DET** | proposal + candidate corpus | `duplicate_candidate_pair` rows with `match_score`, per-signal `signals` jsonb, `detector_version`, `matching_config_version_id` | `ingest` | Trigram GIN + rapidfuzz. **Never merges.** `candidate_merge.performed_by_user_id` is `NOT NULL` (`_decisions.md`), so an automatic merge is a database impossibility, not a policy. | | 16 | Intake resolution | **DET** | everything above | `intake_resolution` row, then `candidate` + `job_application` + `candidate_document`, or a terminal non-candidate state | `ingest` / web | Default is **`needs_review` with a human decision**. Automatic `create_candidate` is permitted only with `auto_create_evidence` populated (a database CHECK, `_decisions.md`) — recommended Phase 1 policy: automatic creation only when the email matches no existing candidate, `document_kind = cv` with high confidence, name and email both above the confidence floor, no duplicate pair above the review threshold, and a job title resolvable to exactly one open posting. Everything else is human. Confidence: **Medium** on the thresholds; they belong in a versioned matching config and will be tuned. | | 17 | ATS scoring | **HYB** | `job_application` + pinned `job_version` + `scoring_config_version` + `candidate_document` + `parse_attempt` | `ats_result` with `is_current`, plus `ats_result_criterion` rows carrying `weight_applied`, `contribution`, `matched_evidence` | `score` (+`ai`) | Aggregation is **deterministic arithmetic** over criterion scores; per-criterion evidence extraction for unstructured requirements is **AI**, with its `AiRun` id recorded. All five version identities are pinned on the row (`_decisions.md`), so a displayed historical score can never drift. `input_fingerprint` makes "has anything changed" an index lookup, preventing pointless rescore churn. **No scoring job may set any status to rejected** — `review_outcome` requires `reviewed_by_user_id`, enforced by CHECK. | **Where a job title is missing** (assignment's "CV with no job title" case): steps 16-17 need a posting to score against. Resolution order — an ATS thread token or posting id in the email body or subject → exact/fuzzy match of a stated title against open `job_posting` titles → a single open posting in the mailbox's configured default department → otherwise the candidate is created (if the identity signals are strong) as a **general application** with **no `job_application`** and therefore **no score**. General applications are a first-class outcome, not an error: the candidate enters the talent pool, appears in `rematch` runs (`_decisions.md` module 16), and a recruiter can attach them to a requisition later, which creates the application and triggers scoring then. A speculative application to a guessed requisition would produce a low score that misrepresents the candidate — which is worse than no score. **Repeat applicants** are handled entirely by the model rather than by this pipeline: step 14 resolves to the existing candidate, a new `candidate_document` revision is added, and the reapplication rule (`_decisions.md`: one live application per candidate+job, `attempt_no`, cooling-off with an audited override) decides whether a new application is permitted. The recruiter sees the prior attempts and their outcomes in the triage view before deciding. ### 4.3 Failure and edge-case matrix Every row ends in a **named terminal or reviewable state**. Nothing is discarded. | Case | Detected at | Behaviour | Retry | Terminal state | Recruiter sees | |---|---|---|---|---|---| | Password-protected PDF | Step 4 | No decryption attempted — no password guessing, ever | No | `parse_failed` reason `encrypted_document` | "This CV is password protected. Request an unprotected copy." One-click templated reply. | | Image-only PDF (scan) | Step 6 | OCR path, steps 7-8 | Yes, 2× | Normal flow; `partial` if OCR confidence is low | Attempt badged "OCR — verify fields" and low-confidence fields left empty | | OCR yields nothing usable | Step 7/8 | Attempt `failed` | No | `parse_failed` reason `no_extractable_text` | "Could not read this document. Ask for a text-based PDF or DOCX." | | Corrupted file | Step 4/5 | Parser exception caught and classified | 1 retry (transient I/O) | `parse_failed` reason `corrupt_file` | "File appears damaged. Request a re-send." | | Unsupported type (`.pages`, `.zip`, `.heic`, `.jfif`) | Step 4 | Rejected at the gate; blob retained | No | `parse_failed` reason `unsupported_format`, with the detected type named | "We cannot read `.pages` files. Ask for PDF or DOCX." Recruiter may convert manually and re-upload, which creates a new attempt against the same intake. | | Declared/detected type mismatch | Step 4 | Processed by **detected** type; mismatch logged as a suspicious-upload signal | n/a | Continues if the detected type is allowed | Nothing (an ops signal, not recruiter noise) | | Very large file | Step 2 (bytes) / step 4 (pages) | Rejected during the stream at the byte cap; page cap at the gate | No | `parse_failed` reason `file_too_large` / `too_many_pages` | "File exceeds 10 MB / 100 pages." | | Zip-bomb / decompression bomb | Step 4/5 | Output-size ratio cap plus wall-clock cap during extraction | No | `parse_failed` reason `resource_limit_exceeded`; blob quarantined | "File rejected for safety." Security event raised. | | Multiple CVs in one email | Step 9 | *n* independent attempts, *n* proposals, one intake | n/a | `needs_review` with *n* proposals | A list to confirm or reject individually | | Non-CV attachment | Step 9 | Classified and retained; not a candidate source | n/a | Attachment `not_parsed_as_cv` | Shown as a supporting document, attachable to the candidate | | Duplicate attachment | Step 2 | Blob stored once, second row linked | n/a | `duplicate_of_attachment_id` set | "Identical file already received" | | No text at all after all paths | Step 8 | Attempt `failed` | No | `parse_failed` reason `no_extractable_text` | As above | | **AI provider failure** (5xx, auth, malformed output) | Step 9/11/17 | Circuit breaker per capability (`_decisions.md` BRD NFR-7). Open → AI steps are **skipped**, not failed | 3 attempts, exponential backoff + jitter | Attempt `partial` reason `ai_unavailable`; deterministic fields from step 10 are kept | "Parsed with basic extraction only — AI enrichment unavailable. Retry?" The application still exists and is still reviewable. Scoring is deferred, not faked. | | **AI timeout** | Step 9/11/17 | Per-call deadline (recommend 30 s extraction, 60 s scoring), hard-cancelled | 2 attempts with a longer deadline | As above | As above | | **AI rate limit (429)** | Step 9/11/17 | Honour `Retry-After`; token-bucket admission control in front of every provider call so we self-throttle before the provider does | Backoff until the window clears, max 6 attempts | `partial` if still limited | "AI enrichment queued" — a delay, not an error | | AI returns schema-invalid JSON | Step 11 | One constrained retry with the validation error fed back; then abandon the AI path | 1 | `partial` reason `ai_schema_violation`; the raw output is retained on the `AiRun` for debugging | As `ai_unavailable` | | AI output contradicts a regex match | Step 12 | Deterministic wins; disagreement recorded | n/a | Continues | Field badged "AI and pattern extraction disagreed" | | Scanner unavailable | Step 3 | `pending`, blob stays quarantined and unsignable | Every 5 min, 12 attempts | `scan_unavailable` → dead-letter, ops alert | "File awaiting security scan" — deliberately **not** viewable | | Worker crash mid-step | any | Job lease expires, job is re-claimed; steps are idempotent | Per the step's policy | Unchanged | Nothing | | Deploy during processing | any | In-flight jobs finish or lease-expire and are retried on the new revision | n/a | Unchanged | Nothing | ### 4.4 Diagram — CV parsing and ATS scoring pipeline ```mermaid sequenceDiagram autonumber participant Q as Queue procrastinate on Postgres participant WK as Worker participant OS as Object storage participant SC as Malware scanner participant PX as Parser libraries participant OCR as Local OCR engine participant AIO as ai_orchestration participant PV as AI provider participant DB as PostgreSQL Q->>WK: attachment_download job WK->>OS: stream bytes to the quarantine prefix, hash while streaming WK->>DB: object_store_key, sha256, byte_size WK->>DB: sha256 already known? DB-->>WK: no WK->>Q: enqueue malware_scan Q->>WK: malware_scan WK->>SC: scan SC-->>WK: clean WK->>OS: move to the active prefix WK->>DB: virus_scan_status clean WK->>Q: enqueue parse_document Q->>WK: parse_document WK->>DB: insert intake_parse_attempt (parser_name, parser_version) status running WK->>PX: detect true type, check encryption, page count, macros PX-->>WK: application/pdf, not encrypted, 3 pages WK->>PX: extract text and layout PX-->>WK: 180 characters, text_layer_ratio 0.02 Note over WK: step 6 thresholds say this is a scan - needs_ocr WK->>OCR: rasterise and OCR 3 pages OCR-->>WK: 4200 characters, mean confidence 0.86 WK->>WK: normalise, section, deterministic entity extraction (steps 8 and 10) WK->>AIO: invoke document_classification, actor_kind = system, allows_system_actor = true AIO->>DB: insert AiRun (actor_user_id NULL, trigger_kind batch, model id and version, prompt version, input ref) AIO->>PV: classify PV-->>AIO: cv, confidence 0.94 AIO->>DB: AiRun output, tokens, cost, latency AIO-->>WK: cv WK->>AIO: invoke cv_field_extraction, actor_kind = system, strict output schema AIO->>PV: extract PV--xAIO: 429 Too Many Requests with Retry-After 20 Note over AIO: honour Retry-After, token bucket throttles admission AIO->>PV: retry after the window PV-->>AIO: schema-valid JSON with per-field confidence AIO->>DB: AiRun row AIO-->>WK: suggested fields WK->>WK: reconcile - regex wins over model, drop fields below the confidence floor WK->>DB: intake_parse_attempt succeeded, parsed jsonb, confidence, versions WK->>DB: canonicalise skills against ref.skill and ref.skill_alias WK->>DB: identity resolution proposal plus duplicate_candidate_pair rows WK->>DB: raw_intake.state = needs_review Note over DB: a human resolves the intake - candidate and application are created Q->>WK: ats_scoring job for the new application WK->>DB: load pinned job_version, job_requirement rows, scoring_config_version, parse attempt WK->>DB: compute input_fingerprint - changed since the last score? DB-->>WK: no current score exists WK->>AIO: invoke requirement_evidence for unstructured criteria only AIO->>PV: evidence extraction PV-->>AIO: per-criterion evidence with text offsets AIO->>DB: AiRun row AIO-->>WK: evidence set WK->>WK: deterministic weighted aggregation using weight_applied per criterion WK->>DB: BEGIN WK->>DB: insert ats_result (pins job_version, config version, document, parse attempt, model version) WK->>DB: insert ats_result_criterion rows with weight_applied, contribution, matched_evidence WK->>DB: flip any prior current row to superseded WK->>DB: audit_event carrying ai_run_id WK->>DB: COMMIT Note over WK,DB: no status is ever set to rejected here - review_outcome needs reviewed_by_user_id ``` **`actor_kind = 'system'` on these three invocations, and why that does not weaken the chatbot guarantee.** The parse of an emailed CV has **no human present** — the mail poller found it, the worker parsed it, and no `app_user` caused any of it. `03-database-design.md` §27.2 originally made `ai.ai_model_invocation.actor_user_id` `NOT NULL` with `actor_kind` restricted to (`user`, `ai_agent`) on the grounds that "there is no service account and no system principal", which would have made these runs **unable to write a ledger row at all** — i.e. `document_classification` and `cv_field_extraction`, the two capabilities the Phase 1 vertical slice depends on, could not run. The two documents now agree, and the boundary is expressed in the schema rather than in prose: | Rule | Enforced by (`03` §27.2) | |---|---| | A `system` run has **no** `actor_user_id` — it cannot borrow or impersonate a user | `ck_run_actor_pairing` | | A `system` run can never be `interactive`; only `batch`, `scheduled` or `webhook` | `ck_run_system_trigger` | | A `system` run is permitted only on a capability with `ai_capability.allows_system_actor = true`, which is **`false` for every assistant, answering or ranking-for-display capability** | deferred constraint trigger | | Where a human *did* cause the work — a manual CV upload through the Import screen, a publish-triggered rescore, a recruiter-requested match — the pipeline **propagates that user** and must not fall back to `system` | service rule, asserted by a test | So a chatbot answer still cannot be produced by a principal with no permissions: a chatbot capability can never carry `actor_kind = 'system'`. The absolute wording in `_decisions.md`'s AI-boundary decision is amended to match — there is no system principal **for any capability that returns data to a user**, which is the property the constraint was protecting. --- ## 5. Background processing (§16) ### 5.1 Queue technology **Postgres-backed durable queue via `procrastinate`.** This is a binding decision in `_decisions.md` (line 131) and this document does not diverge. Restating the two properties this section depends on: - **Transactional enqueue.** Insert-and-enqueue commit together, so "an intake row with no parse job" and "a parse job for a rolled-back row" are both impossible with no outbox table. Every job in the table below relies on this. - **Queueing locks** serialise per-key work declaratively: one delta run per mailbox, one dedupe scan per candidate, one rescore per requisition version. This is where a naive queue produces races. Queues, per `_decisions.md`: `ingest`, `parse`, `score`, `ai`, `mail`, `maintenance`, plus `untrusted` from Phase 2 (parsing under a restricted OS user with no outbound network). Redis is cache, rate limiting and sessions only — never a broker. Conventions applied to every job below: exponential backoff with **full jitter** (bare exponential backoff synchronises retries into a thundering herd); an explicit `max_attempts`; a `failed` terminal state that surfaces in the UI rather than a silent drop; an idempotent body keyed on a domain row; a per-job wall-clock timeout; and structured logs carrying `request_id`, `job_id`, `idempotency_key` and the domain public id. **Dead-letter handling** is one shared mechanism, not per-job code. On final failure a job writes `dead_letter (queue, task_name, idempotency_key, payload jsonb, first_failed_at, last_error, attempt_count, subject_table, subject_pk, state ∈ open/retried/abandoned, triaged_by, triaged_at, note)` and sets the owning domain row to its named failure state. The dead-letter queue is an **operator surface with actions** (retry one, retry a filtered batch, abandon with a reason), reviewed as part of the nightly reconciliation report. A dead-letter row is never auto-retried forever — auto-retry is the retry policy's job, and a DLQ that silently retries is just a slower infinite loop. ### 5.2 Async job catalogue | Job | Trigger | Input | Output | Status writes | Idempotency key | Retry / max attempts / backoff | Failure state | Dead-letter | User-visible error | Monitoring | |---|---|---|---|---|---|---|---|---|---|---| | `mail.sync_delta` | Webhook hint (debounced) + 5-min tick | `channel_id`, `delta_cursor` | `raw_intake` rows; advanced cursor | `ChannelConnection.health`, `last_sync_at`, `IngestionRun` | `channel:{id}` (queueing lock, single-flight) | 5 attempts; 30 s → 8 min; honour `Retry-After` | `ChannelConnection.health = degraded` | Yes, per run | Admin banner "Careers mailbox sync degraded — last successful sync 14:20" | **Alert if no successful sync in 30 min.** Metrics: lag, messages/run, 429 rate, cursor-invalidation count | | `mail.backfill_page` | Connection setup | `channel_id`, page cursor, `backfill_from` | `raw_intake` rows; page cursor | `IngestionRun` progress | `channel:{id}:backfill:{page_cursor}` | 5; 30 s → 8 min | `IngestionRun.failed` | Yes | Setup progress bar with a resume action | Alert if stalled > 2 h | | `mail.renew_subscriptions` | Every 15 min | active subscriptions | renewed or recreated subscriptions | `subscription.expires_at` | `subscription:{id}:renew` | 3, then delete-and-recreate; 1 → 4 min | `subscription.state = lost` → forced delta | Yes | Admin "Mail push notifications unavailable — falling back to polling" | **Alert on any subscription within 2 h of expiry, and on every recreate** | | `mail.reconcile` | Hourly (shallow) / nightly (deep) | window, `channel_id` | ingested missed messages; anomaly report | `reconciliation_report` | `channel:{id}:reconcile:{window}` | 3; 5 → 20 min | report `partial` | Yes | None (ops) | **Alert on any non-zero missed-message count** — this is the webhook-broken signal | | `ingest.download_attachment` | After `raw_intake` insert | `attachment_id` | blob in storage, `sha256`, key | `raw_intake_attachment.download_state` | `attachment:{id}:download` | 5; 10 s → 5 min | `download_failed` | Yes | "Attachment could not be retrieved — retry" | Alert if p95 > 60 s or failure rate > 1 % | | `ingest.malware_scan` | After download | `attachment_id` | verdict; blob moved to active/infected prefix | `virus_scan_status` | `attachment:{id}:scan` | 12; 30 s → 5 min (scanner-down tolerant) | `unscannable` → intake `quarantined` | Yes | "Awaiting security scan" / "File blocked by security scan" | **Alert on any `infected`. Alert if scanner unavailable > 15 min.** Metric: unscanned backlog age | | `parse.extract_text` | After clean scan | `attachment_id` | `intake_parse_attempt` with text and layout | attempt status | `attachment:{id}:parse:{parser_version}` | 2; 1 → 5 min | `parse_failed` + reason | Yes | Specific per §4.3 | p95 duration, failure rate by reason, **memory/CPU per document** | | `parse.ocr_fallback` | Step 6 says `needs_ocr` | `attachment_id`, page range | OCR text + per-page confidence | attempt `partial`/`succeeded` | `attachment:{id}:ocr:{engine_version}` | 2; 2 → 10 min | `parse_failed` reason `no_extractable_text` | Yes | "Could not read this scanned document" | Alert if OCR share of documents > 40 % (mailbox quality regression) or p95 > 5 min | | `parse.cv_fields` | After sufficient text | `attempt_id` | parsed jsonb, per-field confidence, `AiRun` | attempt status; `AiRun` | `attempt:{id}:fields:{prompt_version}` | 3; 20 s → 4 min; honour `Retry-After` | `partial` reason `ai_unavailable` | Yes | "Parsed with basic extraction only — AI unavailable" | **Alert on circuit-breaker open.** Metrics: token cost/day, p95 latency, schema-violation rate | | `ingest.resolve_identity` | After parse | `attempt_id` | resolution proposal + evidence | `raw_intake.state = needs_review` | `intake:{id}:resolve` | 3; 30 s → 5 min | `resolution_failed` | Yes | "Could not match this application — review manually" | Queue depth of `needs_review`; **alert if any intake sits > 48 h** | | `ingest.detect_duplicates` | New/updated candidate; nightly rescan | `candidate_id` or window | `duplicate_candidate_pair` rows | pair `state = open` | `candidate:{id}:dedupe:{detector_version}` (lock) | 3; 1 → 10 min | `detector_failed` | Yes | None (queue simply lacks new items) | Open-pair count trend; **alert on a step change after a detector or threshold version bump** | | `score.ats_score` | Application created; requisition version published; scoring config activated; manual rescore | `application_id`, config version | `ats_result` + criterion rows | `ats_result.is_current` | `application:{id}:score:{fingerprint}` | 3; 30 s → 6 min | `score_failed` | Yes | "Score unavailable — retry" (**never a fake score, never a default of 0**) | Unscored-application count; p95 score latency; **alert if any application is unscored > 4 h** | | `score.rescore_batch` | Requisition version publish / config activation | `job_version_id` or `config_version_id` | fan-out of per-application score jobs | batch progress row | `rescore:{job_version_id}:{config_version_id}` (lock) | 3; 1 → 15 min | `batch_partial` with a per-application failure list | Per child job | "Rescoring 240 applications — 12 failed, retry" | Batch duration; failure share | | `outbound.publish_posting` | Posting published/updated/closed | `posting_id`, target platform | external id, posting state | `PublishAttempt` | `posting:{id}:publish:{platform}:{version}` | 5; 1 → 30 min | `publish_failed` | Yes | "Not yet live on the careers site — retry" | **Alert if a posting is `published` in the ATS but not confirmed live within 15 min** | | `notify.send_email` | Human action, or a non-candidate-facing rule | `outbound_message_id` | provider accept, `provider_ref` | `OutboundMessage.status` | `outbound:{public_id}` + `sent_at IS NULL` guard | 5; 1 → 30 min; honour `Retry-After` | `rejected_by_provider` | Yes | "Message could not be sent — recruiter action required" | **Alert on any candidate-facing send failure.** Metrics: hard-bounce rate, queue age | | `notify.classify_bounce` | Inbound NDR intake | `raw_intake_id` | linked `OutboundMessage` bounce state; suppression entry | `OutboundMessage.status` | `intake:{id}:ndr` | 3; 1 → 10 min | `ndr_unmatched` → `needs_review` | Yes | "Delivery failure could not be matched to a sent message" | Hard-bounce rate; **alert if unmatched NDRs > 5 % (the thread token is broken)** | | `notify.reminders` | Scheduled (interviews, offer expiry, SLA, stale intake) | window | notifications + worklist tasks | `Notification`, `Task` | `reminder:{kind}:{subject_pk}:{due_bucket}` | 3; 1 → 10 min | `reminder_failed` | Yes | None (in-app) | **Alert if a scheduled tick is missed** — a silent scheduler is the worst failure here | | `analytics.refresh_read_models` | Nightly + on demand | — | refreshed materialised views | `ReportRun` | `analytics:refresh:{date}` (lock) | 2; 10 → 40 min | `refresh_failed`; last good data retained | Yes | "Dashboard data as at 02:00 — refresh in progress" | Refresh duration; **alert if data staleness > 26 h** | | `ai.generate_embeddings` (**Phase 2**) | New/updated `candidate_document` | `document_id`, model version | `candidate_embedding` row | `embedding_state` | `document:{id}:embed:{model_version}` | 3; 1 → 15 min | `embedding_failed` | Yes | None (semantic search silently narrows) | Coverage % of candidates with a current-model embedding; cost/day | | `ai.index_assistant_corpus` (**Phase 2**) | Content change + nightly | changed refs | refreshed retrieval index entries | `index_state` | `corpus:{ref}:{index_version}` | 3; 1 → 15 min | `index_stale` | Yes | Assistant states "my index was last updated at 02:00" | Index lag; **alert if lag > 24 h** | | `maintenance.retention_purge` | Nightly | `retention_due_on <= today`, minus `retention_hold` | pseudonymised rows, deleted blobs, `retention_action` rows | `retention_action` | `retention:{policy}:{date}` (lock) | 2; 1 h | `purge_partial` | Yes | None | **Alert on any failure — this is a legal obligation.** Metrics: rows purged, blobs deleted, holds skipped | | `maintenance.orphan_blob_sweep` | Weekly | storage listing vs DB keys | orphan report; delete after a 30-day grace | `orphan_candidate` rows | `orphan_sweep:{week}` | 2; 1 h | `sweep_partial` | Yes | None | Orphan count trend; **alert on any deletion of an object with a live DB reference (never expected)** | | `maintenance.audit_partition_maintain` | Monthly | — | new partition; detached/archived old partition + final `row_hash` | `partition_registry` | `audit_partition:{yyyymm}` | 3; 1 h | `partition_failed` | Yes | None | **Alert if next month's partition does not exist by the 25th** | | `maintenance.audit_hash_verify` | Nightly | partition | verification result | `verification_run` | `audit_verify:{partition}:{date}` | 2; 1 h | `verification_failed` | Yes | None | **Page immediately on a hash-chain mismatch** — this is tamper evidence | | `maintenance.dead_letter_report` | Nightly | open DLQ rows | digest to admins | — | `dlq_report:{date}` | 2; 30 min | — | No | Admin digest | **Alert if open DLQ rows > 20 or any row is older than 72 h** | | `maintenance.session_cleanup` | Nightly | expired sessions/tokens | rows deleted | — | `session_cleanup:{date}` | 2; 30 min | — | No | None | Row count | Two policy notes that apply across the table: - **User-visible errors name the object and the next action.** "Processing failed" is useless to a recruiter. "Could not read `Ahmed_CV.pdf` — the file is password protected. Request an unprotected copy?" is actionable. Copy for every failure reason in §4.3 is a deliverable of the intake UI, not an afterthought. - **Interactive-path monitoring differs from batch.** `_decisions.md` sets a soft process-split trigger at "p95 time-to-start for interactive AI tasks > 10 s while web p95 < 300 ms". That requires **time-to-start** (enqueue → claim) as a first-class metric per queue, not just duration. Instrument it in Phase 1 or the trigger cannot be evaluated when it matters. --- ## 6. File storage (§17) ### 6.1 Key convention, metadata and hashing **Content-addressed keys, environment- and class-prefixed, with no user-controlled component:** ``` {env}/{class}/{yyyy}/{mm}/{sha256[0:2]}/{sha256[2:4]}/{sha256} ``` Examples: ``` prod/quarantine/2026/07/9f/3c/9f3c…e1 (freshly received, unscanned) prod/active/2026/07/9f/3c/9f3ca7…e1 (scanned clean, referenced by a domain row) prod/infected/2026/07/9f/3c/9f3ca7…e1 (quarantined, restricted ACL, never signable) prod/generated/2026/07/… (offer-letter PDFs, exports) ``` Decisions embedded in that convention, each for a reason: | Choice | Why | |---|---| | Content-addressed on sha256 | Storage-level dedupe for free (the same CV sent five times is one object), and a key that cannot collide. | | **No filename in the key, ever** | Candidate filenames are attacker-controlled: path traversal, unicode direction overrides, `.pdf.html`, 400-character names. The original filename is stored as a **database column** for display and set as a `Content-Disposition` filename **only after** sanitisation. This is the single most common file-storage vulnerability and the convention removes it structurally. | | Two-level hash-prefix fan-out | Keeps listing and any future prefix-partitioned operation sane at millions of objects. Costs nothing. | | Date segment | Makes lifecycle rules, cost attribution and "everything received in July" trivially expressible. | | Explicit `quarantine` / `active` / `infected` / `generated` classes | The scan state is expressed in the key, so "unscanned files are unsignable" is enforced by a prefix check in one function rather than by remembering to check a column. | | Environment prefix **plus a separate bucket/container per environment** | Belt and braces; see §6.7. | Metadata, split deliberately between store and database: - **Object metadata** (immutable, set at write): `sha256`, `byte_size`, `detected_mime`, `received_at`, `source_channel`, `raw_intake_public_id`, `retention_class`, `original_filename_sanitised`. Enough to reconstruct provenance if the database were lost — and enough for a storage-side lifecycle rule to act on `retention_class` without a database lookup. - **Database of record**: **`app.stored_file`** (`03` §8.1) is the registry, keyed uniquely on `sha256`, and it owns `storage_key`, `byte_size`, `mime_type`, **`virus_scan_status`** and `retention_class`. `raw_intake_attachment` / `candidate_document` hold `stored_file_id` plus their own per-occurrence data — `original_filename` (raw, for display), `attachment_index`, `deleted_at` — and a denormalised `sha256`/`byte_size` copy for local checks. **Scan status exists in exactly one place**, which is the load-bearing half of §9.1 row 4's resolution. One `stored_file` row can be shared by N occurrences (content-addressed), so the retention sweep deletes a blob only when no non-purged row still references it (`03` §8.1). `files.store()` / `files.signed_url()` in this document name the Part 1 **module facade**, not a schema — there is no `files` schema. - **No PII in keys or object metadata.** No candidate name, no email. Keys appear in logs, storage access logs and CDN traces, all of which have different retention rules from candidate data. Hashing: sha256 computed **during** the upload stream, never by re-reading the object. Verified on first read after write; a mismatch is a `storage_integrity_error`, pages ops, and blocks the parse rather than parsing a possibly-truncated file. ### 6.2 AWS S3 vs MinIO — evaluated against the actual deployment environment The assignment asks for an S3-compatible design and an S3-vs-MinIO evaluation. The honest answer requires naming a conflict first: **`_decisions.md` (line 233) recommends Azure** (Container Apps, PostgreSQL Flexible Server, **Blob Storage**, Key Vault, Entra ID), for the strong reason that Outlook is inbound channel #1 so mail, identity and hosting land in one tenant. **Azure Blob Storage is not S3-compatible.** So "S3-compatible" cannot be both the literal implementation and consistent with the binding deployment decision. Resolution, stated as a decision rather than a fudge: **"S3-compatible" is adopted as the design *contract*, not as a vendor commitment.** The `files` module exposes an object-store port (§7.3) whose semantics are the S3 subset every provider supports — put, get, presign-get, presign-put, head, copy, delete, list-prefix, plus server-side encryption and object versioning. Three adapters implement it, and the platform decision selects one: | Option | Fit against A7 (Azure) | Verdict | |---|---|---| | **Azure Blob Storage** | Same tenant, same identity model (managed identity, no keys), same portal, same Key Vault, Private Endpoint to the container platform, no egress cost between tiers, immutability policies for the audit archive, versioning and soft delete built in, lifecycle rules for retention tiering. Not S3-API. | **Recommended production store if A7 holds.** The adapter is ~200 lines and everything above it is provider-agnostic. | | **AWS S3** | Best-in-class API, Object Lock (which `_decisions.md` names for the audit archive), mature presigning, `s3:ObjectCreated` events. But under A7 it means a second cloud account, a second IAM model, cross-cloud credentials in Key Vault, cross-cloud egress on every attachment read, and two consoles for two developers with no ops staff. | **Recommended production store only if the cloud decision moves to AWS.** Then it is the default and MinIO is local-only. | | **MinIO (self-hosted)** | S3-API-compatible, runs anywhere, no vendor lock-in. But in production it is a stateful distributed system that this team would own: disk provisioning, erasure coding, upgrades, TLS certificates, capacity planning, and its own backup story for candidate CVs. `_decisions.md` rejects self-managed Postgres precisely because "backup, patching and failover become the senior developer's unpaid second job" (line 237); the identical argument applies here, and CV blobs are the least replaceable data in the system. | **Rejected for production. Adopted for local development and CI**, where its value is real: one `docker compose` service, no cloud credential in a developer's environment, and deterministic tests. | **Decision.** Production: the managed object store of whichever cloud is chosen — **Azure Blob Storage under A7**. Local and CI: **Azurite** when production is Azure Blob (same API, same adapter, true parity), or **MinIO** when production is S3. Choosing the emulator that matches the production adapter matters more than choosing the S3 API for its own sake: a MinIO-local plus Azure-production pairing means the local environment exercises an adapter that never runs in production, which is how "works locally" stops meaning anything. **Explicitly rejected: AWS Glue**, and every ETL/catalogue service like it. There is no data lake, no cross-source ETL and no schema-discovery problem here. Analytics is read-only SQL views inside the one PostgreSQL database (`_decisions.md` module 24); adding a catalogued ETL layer would be a second data model, a second cloud dependency and a second thing to secure, for zero Phase 1 requirement. ### 6.3 Encryption | Layer | Design | |---|---| | In transit | TLS 1.2+ enforced at the store; HTTPS-only bucket/container policy; Private Endpoint or VPC endpoint so attachment traffic never traverses the public internet. | | At rest | Provider-managed server-side encryption (SSE) on **by default at the container level**, so an object cannot be written unencrypted. Phase 1 uses provider-managed keys. | | Customer-managed keys | Phase 2 upgrade: a Key Vault / KMS key with rotation, so key destruction becomes an additional erasure lever. **Recommended, not Phase 1** — it adds a key-availability failure mode, and Phase 1 already has enough new moving parts. Confidence: **Medium**; escalate if legal requires CMK for the six jurisdictions. | | Client-side / envelope encryption | **Rejected for Phase 1.** It would break server-side scanning, provider-side lifecycle rules and any storage-side preview, and it puts key management on two developers. Revisit only if a jurisdiction mandates it. | | Application-level | None on the blob. Field-level protection is a database concern (`pii_classification`, `masking_strategy`); double-encrypting the blob buys nothing and complicates the retention purge. | ### 6.4 Signed URLs and authorization **Rule: the authorization decision happens in the application, before a URL is minted. The signed URL is the transport, never the permission.** | Control | Specification | |---|---| | Mint path | `files.signed_url(actor, document_ref, purpose)` → calls `iam.can(actor, 'read', owning_domain_object)` → refuses if the object is in the `quarantine` or `infected` prefix or `virus_scan_status != 'clean'` → mints. Access is derived from the owning domain object (`_decisions.md` module 3), never from possession of a key. | | TTL | **5 minutes** for read, **15 minutes** for a direct upload (not used in Phase 1, §3.6). Short enough that a leaked URL in a chat log or a support ticket is dead on arrival. | | Response headers | Forced `Content-Disposition: attachment; filename=""` and `Content-Type` from **our** detected type, never the uploader's declared type. Prevents an HTML or SVG "CV" executing script in a recruiter session — which matters enormously given findings §E establishes there is no escaping anywhere in the existing frontend. | | Serving origin | CV blobs are served from a **separate hostname** from the application. Even with `Content-Disposition: attachment`, a same-origin file-serving path is one misconfiguration away from a stored-XSS vector against a recruiter session. | | Audit | Every mint writes an `audit_event` with `action = 'document.access'`, actor, document ref and purpose. `_decisions.md` requires explicit application writes for access events because no trigger can observe a read; a CV download is exactly such an event and is the compliance-relevant one. | | Candidate-facing access | **Never a signed URL in an email.** A `candidate_access_token` (hashed, scoped, expiring, revocable, `_decisions.md`) is exchanged server-side for a short-lived stream. Entity ids are identifiers, not capabilities. | | Bucket policy | Public access **blocked at the account and container level**, not merely absent. Anonymous listing and anonymous read denied explicitly, and a CI check asserts it per environment. | | Enumeration | Keys are sha256-derived and unlisted; a signed URL grants access to one key only; `list` is not exposed through the port to any caller except the orphan sweep, which runs as a distinct storage identity. | ### 6.5 Malware scanning | Concern | Design | |---|---| | Placement | Between download/upload and any parse. A blob in the `quarantine` prefix is unsignable and unparseable — enforced by a prefix check in `files.signed_url()` and in the parse job's precondition, not by convention. | | Engine | ClamAV in the worker image (or a sidecar) for Phase 1: no per-file cost, no candidate CV leaving controlled infrastructure (BRD §7.4), signature updates by scheduled job. **Stated limitation honestly:** ClamAV's detection rate on targeted or novel malware is materially below a commercial multi-engine service. It is a hygiene control, not a guarantee. | | Upgrade path | A cloud-native scanner (Microsoft Defender for Storage under A7) or a multi-engine service, behind the same `MalwareScanner` port (§7.4). Trigger to upgrade: any confirmed infected submission, or a legal/insurance requirement. | | Signature freshness | Scanner signature version recorded on every scan. Signatures older than 48 h raise an alert; older than 7 days, scanning **fails closed** to `unscannable` rather than issuing a verdict a stale engine cannot support. | | Rescan on signature update | Weekly rescan of blobs received in the last 30 days (a cheap window, catches same-day-zero-day). A newly-infected verdict on an already-active blob moves the object to the `infected` prefix, revokes access, alerts security, and creates a worklist item — it does **not** delete the candidate. | | Infected handling | Object moved to the `infected` prefix with a restricted ACL. **Retained, not deleted**, in Phase 1: it is evidence, and BRD-side incident handling may need it. Retention 90 days, then purge. `raw_intake.state = quarantined` (terminal, no candidate). Applicant receives the same neutral confirmation as everyone else (§3.7). | | Beyond signatures | Structural gate at step 4 (macros, embedded JavaScript in PDFs, embedded executables, decompression ratio caps) catches a class ClamAV does not. Parsing runs with a hard timeout, memory cap, restricted OS user and — from Phase 2 — no outbound network (`_decisions.md` line 110). `_decisions.md` line 271 is right to name this the most likely early process-split trigger. | ### 6.6 Versioning, backups, retention, deletion and orphans | Concern | Design | |---|---| | Object versioning | **On** for `active` and `generated`; not needed for `quarantine`. Content-addressed keys mean a "new version" of a CV is a new object anyway, so versioning here is protection against accidental overwrite or deletion, not a revision mechanism. CV revisions are modelled as `candidate_document` rows (`_decisions.md`), which is where they belong. | | Soft delete (storage-level) | Provider soft-delete / delete-retention **14 days**, matching the database PITR window (`_decisions.md` line 233) so a point-in-time restore always has its blobs. | | Soft delete (application-level) | `deleted_at` on `candidate_document`; the blob is untouched. Excluded from the live views (`v_candidate_live`) and unsignable. **Soft delete is not erasure and the UI must never call it "deleted permanently"** — `_decisions.md` is explicit on this. | | Permanent delete | Only two paths: (a) the **retention purge**, whose action is pseudonymisation of database columns **plus real deletion of CV blobs**, recording `blob_keys_deleted` on `retention_action`; (b) an **explicit legal erasure request**, executed by the same code path with a distinct policy key. Both are audited, both skip `retention_hold` subjects, and both require the blob-delete to succeed or the whole subject's purge is marked `partial` and re-queued — a pseudonymised database row whose CV blob still exists is a false claim of erasure. | | Backups | Geo-redundant storage (GRS/GZRS or cross-region replication) with a documented **RPO 15 min / RTO 4 h** for blobs. Blob backup and database PITR must be restorable to the **same** point, or a restore produces `candidate_document` rows with dangling keys. Restore is a rehearsed runbook, tested once per phase; an untested restore is not a backup. | | Retention classes | `raw_attachment` (retain with the intake), `candidate_document`, `generated_document` (offer letters — longer, contractual), `audit_archive` (immutable, WORM, `_decisions.md` layer 4). Class is on the object metadata **and** the database row, so a provider lifecycle rule and the application purge agree. | | Immutability for audit archive | Azure immutable blob policy / S3 Object Lock in compliance mode on the `audit_archive` container, with the closed partition's final `row_hash` recorded alongside. This is the only genuinely independent tamper check (`_decisions.md` is explicit that hash chaining is evidence, not prevention). | | **Orphan cleanup** | Weekly `maintenance.orphan_blob_sweep`: list the store by prefix, anti-join against `object_store_key` across `raw_intake_attachment`, `candidate_document` and generated-document tables. An object with no reference and older than **30 days** is reported, then deleted on the following run if still orphaned. **Two-pass with a grace period is deliberate** — a single-pass delete races an in-flight upload whose database row has not yet committed, and the failure mode is destroying a candidate's only CV. The reverse anti-join (a database row pointing at a missing object) is a **`storage_integrity_error` and pages immediately**; that direction is data loss, not garbage. | ### 6.7 Environment separation | Rule | Detail | |---|---| | Separate storage accounts/buckets per environment | `utopia-ats-prod`, `utopia-ats-staging`, `utopia-ats-dev`. **Not** prefixes in one bucket. A prefix is one IAM-policy typo away from staging writing to production; separate accounts make cross-environment access require a credential that does not exist. | | No production credential outside production | Distinct managed identities; production storage grants access only to the production container revisions. Developers have no production data credential; production diagnosis is by log and metric, or by an audited break-glass grant. | | Non-production data | **Never a copy of production blobs.** Staging gets synthetic CVs plus a pseudonymised extract produced by the anonymisation script that reads `pii_classification` (`_decisions.md`). Copying real CVs into staging silently widens the processing boundary and defeats the retention machinery. | | Local development | Azurite (or MinIO, per §6.2) in `docker compose`, seeded with synthetic fixtures. No cloud credential on a laptop. | | Test mailbox | The staging Graph app registration points at a dedicated **test** mailbox (`_decisions.md` line 233). A staging deploy must be structurally incapable of reading production candidate mail — separate app registration, separate access policy, separate mailbox. | | CI | Ephemeral MinIO/Azurite service container plus the real Postgres service container `_decisions.md` mandates. No external provider is contacted from CI; email, AI, storage and scanner ports are all faked at the port boundary (§7). | --- ## 7. External provider abstraction ### 7.1 Rule Every external dependency sits behind a **port** — a `typing.Protocol` in the owning platform module — with adapters selected by configuration. Only the owning module may import a provider SDK; an `import-linter` forbidden-contract makes an SDK import anywhere else a **failed build** (`_decisions.md` line 19 already enforces boundaries this way, so this is the same mechanism, not a new one). Four ports, four owners: | Port | Owner module | Adapters | Selected by | |---|---|---|---| | `MailProvider` | `integrations_inbound` (+ `notifications` for send) | `GraphMailProvider`, `ImapSmtpMailProvider`, `FakeMailProvider` | `ChannelConnection.type` | | `AiProvider` | `ai_orchestration` — **the only package permitted a model client** (`_decisions.md` line 138) | `HostedApiAiProvider`, `LocalOcrProvider`, `FakeAiProvider` | `ModelConfigVersion` | | `ObjectStore` | `files` | `AzureBlobObjectStore`, `S3ObjectStore` (AWS or MinIO), `FakeObjectStore` | env config | | `MalwareScanner` | `files` | `ClamAvScanner`, `DefenderForStorageScanner`, `FakeScanner` | env config | Three properties the ports must have, or they are decoration: 1. **Provider-neutral vocabulary.** No `messageId`, `deltaLink`, `Bucket` or `blobClient` in a signature. Cursors are opaque `str`; ids are `provider_message_id: str`. 2. **Provider-neutral errors.** Every adapter maps its SDK's exceptions onto one taxonomy: `Transient`, `RateLimited(retry_after)`, `AuthFailure`, `NotFound`, `CursorInvalid`, `Permanent`. Retry policy is written **once** against that taxonomy. Without this, retry logic leaks provider knowledge into every call site — which is how a "swappable" provider turns out not to be. 3. **The fake is a first-class adapter**, shipped in the main package and used by CI and local development. Its fixtures include the ugly cases: a 410 cursor invalidation, a 429 with `Retry-After`, a message with three attachments and one CV, an encrypted PDF, an image-only PDF, an NDR, and a schema-violating AI response. A fake that only does the happy path tests nothing worth testing. ### 7.2 `MailProvider` ```python from typing import Protocol, Iterator, Sequence from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class MailAttachmentRef: provider_attachment_id: str filename: str declared_mime: str declared_size: int is_inline: bool kind: str # "file" | "item" | "reference" @dataclass(frozen=True) class MailMessage: provider_message_id: str # stable across folder moves (Graph immutable id) thread_key: str | None # conversationId, or the ATS thread token internet_message_id: str | None in_reply_to: str | None references: Sequence[str] received_at: datetime # tz-aware, UTC sender_address: str sender_display_name: str | None to: Sequence[str] cc: Sequence[str] subject: str body_text: str | None body_html: str | None # stored raw; escaped at render (findings section E) headers: dict[str, str] attachments: Sequence[MailAttachmentRef] raw_envelope: dict # -> raw_intake.payload @dataclass(frozen=True) class MailPage: messages: Sequence[MailMessage] next_cursor: str | None # opaque; None means the page set is exhausted delta_cursor: str | None # opaque; set only on the final page @dataclass(frozen=True) class SubscriptionInfo: provider_subscription_id: str expires_at: datetime client_state_hash: bytes @dataclass(frozen=True) class SendResult: provider_ref: str accepted_at: datetime class MailProvider(Protocol): def list_since(self, folder: str, since: datetime, cursor: str | None) -> MailPage: ... def delta(self, cursor: str) -> MailPage: ... # raises CursorInvalid def initial_delta_cursor(self, folder: str) -> str: ... def fetch_attachment(self, provider_message_id: str, provider_attachment_id: str) -> Iterator[bytes]: ... def list_message_ids(self, folder: str, since: datetime) -> Iterator[tuple[str, datetime]]: ... # reconciliation def move_message(self, provider_message_id: str, folder: str) -> str: ... def tag_message(self, provider_message_id: str, categories: Sequence[str]) -> None: ... def send(self, *, to: Sequence[str], subject: str, body_html: str, body_text: str, headers: dict[str, str], attachments: Sequence[tuple[str, str, Iterator[bytes]]], reply_to_provider_message_id: str | None) -> SendResult: ... def create_subscription(self, folder: str, notification_url: str, lifecycle_url: str, client_state: str) -> SubscriptionInfo: ... def renew_subscription(self, provider_subscription_id: str) -> SubscriptionInfo: ... def delete_subscription(self, provider_subscription_id: str) -> None: ... def capabilities(self) -> frozenset[str]: ... ``` `capabilities()` is how the IMAP adapter declares honestly what it cannot do — e.g. `{"push_idle", "server_threading_absent", "no_immutable_ids"}` — so the sync orchestrator can lean harder on payload-hash dedupe and the ATS thread token instead of silently producing worse results. `list_message_ids` exists purely for reconciliation and is deliberately cheap (ids and timestamps only) so the hourly sweep costs almost nothing. ### 7.3 `ObjectStore` ```python class ObjectStore(Protocol): def put_stream(self, key: str, stream: Iterator[bytes], *, content_type: str, metadata: dict[str, str], max_bytes: int) -> "PutResult": ... # computes sha256 while streaming; raises SizeLimitExceeded mid-stream def get_stream(self, key: str) -> Iterator[bytes]: ... def head(self, key: str) -> "ObjectInfo": ... def copy(self, src_key: str, dst_key: str) -> None: ... def delete(self, key: str, *, permanent: bool = False) -> None: ... def presign_get(self, key: str, *, ttl_seconds: int, filename: str, content_type: str) -> str: ... def presign_put(self, key: str, *, ttl_seconds: int, max_bytes: int) -> str: ... def list_prefix(self, prefix: str, *, since: datetime | None = None) -> Iterator["ObjectInfo"]: ... ``` `presign_get` **requires** `filename` and `content_type` rather than accepting the object's own values, so §6.4's forced-download rule cannot be forgotten at a call site. `put_stream` takes `max_bytes` because a declared size is attacker-controlled and the cap must bind during the transfer. ### 7.4 `MalwareScanner` and `AiProvider` ```python class MalwareScanner(Protocol): def scan_stream(self, stream: Iterator[bytes]) -> "ScanVerdict": ... # ScanVerdict(status: clean|infected|unscannable, signature_name: str | None, # engine: str, engine_version: str, signature_version: str, # scanned_at: datetime) def engine_status(self) -> "EngineStatus": ... # signature freshness, drives fail-closed class AiProvider(Protocol): def complete_structured(self, *, prompt: str, output_schema: dict, model: str, max_tokens: int, timeout_s: float) -> "AiResult": ... def complete_stream(self, *, prompt: str, model: str, timeout_s: float) -> Iterator[str]: ... def embed(self, *, texts: Sequence[str], model: str) -> Sequence[Sequence[float]]: ... def model_info(self, model: str) -> "ModelInfo": ... # id, version, context, pricing ``` `AiProvider` is intentionally narrow and **not** the module boundary. Callers never touch it; they call `ai_orchestration.invoke(capability, context, actor)`, which writes the `AiRun` row **before** the result is usable, calls `iam.can()` with the **human** actor, and returns a suggestion object rather than a domain write (`_decisions.md` line 138). The circuit breaker, the token-bucket admission control, cost accounting and prompt-version pinning all live in `invoke()`, so no adapter and no caller can bypass them. --- ## 8. Cross-cutting operational requirements ### 8.1 Observability - **Structured JSON logs** with `request_id` (propagated from the API layer through the queue payload into every job — this is what makes a webhook receipt traceable to a score row), `job_id`, `idempotency_key`, `channel_id`, and the domain **public id**. Never a bigint PK, never a candidate name, never an email address, never a blob byte. - **Metrics** per integration: sync lag, messages ingested, attachments downloaded, scan verdicts, parse success rate by reason, OCR share, AI latency/tokens/cost/error rate by capability, queue depth and **time-to-start per queue**, DLQ depth and age, unresolved-intake age, unscored-application count, outbound bounce rate. - **The five alerts that matter most**, in priority order: (1) no successful mail sync in 30 minutes; (2) reconciliation found missed messages; (3) any intake unresolved for over 48 hours; (4) audit hash-chain verification failure; (5) retention purge failure. Each has a named owner and a runbook line. Everything else is a dashboard. - **Health endpoints** distinguishing liveness from readiness, plus `/healthz/integrations` reporting per-channel health, subscription expiry, scanner signature age and AI circuit-breaker state — one page a recruiter's admin can read before escalating. ### 8.2 Degradation ladder Stated explicitly so behaviour under partial failure is a design choice rather than an emergent accident (BRD NFR-7): | Failure | System behaviour | |---|---| | AI provider down | Intake, parsing (deterministic path), candidate creation, applications and all recruiter workflow continue. Scoring is **deferred**, never defaulted. Assistant is unavailable with an honest message. | | OCR unavailable | Text-layer PDFs and DOCX parse normally; scanned documents queue as `needs_ocr` and are recruiter-visible as pending. | | Malware scanner down | Intake continues; files stay quarantined and unviewable; a visible backlog counter with age. Never "assume clean". | | Object storage down | Intake **stops** — this is correct. Accepting a submission we cannot store the CV for would be data loss dressed up as success. The public endpoint returns `503` with `Retry-After` so the site can tell the applicant to try shortly. | | Mail provider down | Web tier unaffected; sync retries with backoff; delta cursor guarantees nothing is lost on recovery. | | Careers site down | ATS unaffected. Postings simply are not fetched. | | Queue/worker down | Web tier serves reads and writes; async work accumulates durably in Postgres and drains on recovery. Nothing is lost, because the queue **is** the database. This is the payoff for the `procrastinate` decision. | | Database down | Everything stops. Single point of failure, accepted deliberately by `_decisions.md` (line 235) for an internal business-hours tool, with PITR as the recovery path. It should be stated to the business rather than discovered. | ### 8.3 Work split for this document's scope Per `_decisions.md` (line 259). Every Ahmed item is independently demonstrable and has a Talha review checkpoint. | Owner | Items | |---|---| | **Talha** | Entra app registration and access-policy verification; the `MailProvider` port and Graph adapter; subscription lifecycle and reconciliation; `ai_orchestration` and the `AiProvider` port; the parsing/OCR pipeline and its resource sandboxing; scoring; queue configuration and locks; the `ObjectStore` port and adapters; storage IAM and encryption. | | **Ahmed** | The public submission endpoint with its DRF serializer + Zod pair (validation, an independently demonstrable slice); the idempotency-record implementation and its test matrix; rate-limit and CAPTCHA verification wiring including the fail-open-into-strict-mode path; the `FakeMailProvider` fixture library including the ugly cases (high-value, genuinely interesting, and it is what makes the whole integration testable); the intake triage queue UI and the failure-reason copy for every case in §4.3; the dead-letter operator screen with retry and abandon actions (a real state machine, not CRUD); the `/healthz/integrations` page; the orphan-blob sweep report; Playwright journeys for ingest → parsed → promoted. | --- ## 9. Inconsistencies, open questions and risks ### 9.1 Inconsistencies with `_decisions.md` found while writing this document > **The binding resolution for all six is in `_open-items.md`.** This table is the evidence and the > position this document took; the ruling is there, and the "Proposed resolution" column is > superseded wherever it differs. 1 → OPEN-04 (cloud platform, therefore object storage — owner: > business + Talha), 2 → RULING-02, 3 → RULING-03 (`_glossary.md`), 4 → **RULING-05** (the registry > is `app.stored_file` at migration `005`, not `files.stored_object`; the names describe the same > table and this row's proposal is superseded, not overruled — and the ruling adopts this row's > load-bearing point that scan status must exist in exactly one place), 5 → OPEN-07 (the minimal > Phase 1 `notifications` slice — owner: Talent Lead + Talha), 6 → C-08 (no divergence). > §9.2's Q1–Q4 → OPEN-06, OPEN-06, OPEN-03, OPEN-04. | # | Inconsistency | Severity | Proposed resolution | |---|---|---|---| | 1 | **Azure Blob Storage vs "S3-compatible".** `_decisions.md` line 233 recommends Azure Blob Storage; the assignment (§17) asks for an S3-compatible design and an AWS-S3-vs-MinIO evaluation. Azure Blob is not S3-API-compatible. | **Medium** | Resolved in §6.2 by treating "S3-compatible" as the port *contract* and providing three adapters. Production = the chosen cloud's managed store (Azure Blob under A7). The cloud decision must be confirmed before the storage adapter is written; it is cheap now and expensive after the retention and lifecycle rules exist. | | 2 | **Plain-SQL migrations vs Django.** Part 1 (line 91) chooses Django partly *because* migrations are built in; Part 2 (line 290) mandates ordered plain-SQL migrations with "the ORM never generates the schema". Part 2's own risk list (line 463) already flags this. | **High** — it is a daily workflow decision for both developers. | **Settled, not flagged.** The binding ruling is the canonical text in `02-system-architecture.md` §12.4 (ADR 0017), quoted verbatim immediately below this table. Two consequences that matter to *this* document: `procrastinate`'s vendor-managed migrations are ignore-listed **explicitly**, not by omission (§5); and the intake-side objects this document depends on — the `raw_intake` partial unique indexes, the idempotency-record unique index, the `CHECK` regexes on email addresses — are all declarable in `Meta.constraints`/`Meta.indexes`, so they stay inside the ORM gate rather than being handed to the SQL gate. | | 3 | **Intake entity names differ between parts.** Part 1 module 6 names `InboundSubmission`, `SubmissionAttachment`, `ProcessingAttempt`; Part 2 names `raw_intake`, `raw_intake_attachment`, `intake_parse_attempt`. | Low | Same design, two vocabularies. This document uses Part 2's physical names throughout. The data-model document should pick one set and state the mapping once. | | 4 | **Blob registry ownership — RESOLVED.** Part 1 module 3 gives `files` a `StoredFile(sha256, mime, size, storage_key, scan_status, retention_class)` table; Part 2 puts `object_store_key`, `sha256` and `virus_scan_status` directly on `raw_intake_attachment` and `candidate_document`. Taken literally, the same blob's scan status lives in two places. | Medium — a two-place scan status is exactly how "unscanned file becomes viewable" happens. | **Resolved as `app.stored_file`** — `03-database-design.md` §8.1 defines it and `03` §32.1 row 2 rules on it ("adopts Part 1's registry **and** keeps Part 2's columns as denormalised domain data"). Two corrections to this row's original wording: the table is **`app.stored_file`**, not `files.stored_object` — there is no `files` schema; `03` §1.1 lists five application schemas (`ref`, `app`, `ai`, `audit`, `staging`) plus the library-owned `queue`, and `files.*` elsewhere in this document names the Part 1 **module facade** (`files.store()`, `files.signed_url()`), not a schema. And this row's content-addressed keying **is** what was adopted, in its strong form: `uq_stored_file_sha256` means five candidates who send the same file share **one** row and **one** scan, with `03` §8.1 now stating explicitly what that implies for retention — a blob is deleted only when no non-purged row references it, tested by `NOT EXISTS` rather than a stored refcount. `stored_file.virus_scan_status` is the **only** scan status; the domain rows' copies are denormalised read conveniences and are never written independently. | | 5 | **`notifications` is Phase 2, but inbound email is Phase 1.** Part 1 module 5 phases `notifications` (including `OutboundMessage` and bounce handling) at Phase 2, while `integrations_inbound` is Phase 1. Yet Phase 1 needs at least "request an unprotected copy of your CV" replies for the §4.3 failure cases to be actionable, and NDR classification depends on the outbound row existing. | Medium | **Accepted and ruled — this row is now settled, not a recommendation.** The package had four different phasings (this row at 1, `07` §5.1 and `03` §5 / §33 at 2, `00` DEF-07 at 2–3, `00` OBD-21 at 3); `08` §7 finding 10 records the conflict and its resolution. The split: **Phase 1** takes the minimal slice — the `outbound_message` row, `Mail.Send` through the same `MailProvider` port as §2, the send idempotency guard, NDR classification, and one seeded transactional template — **4–6 developer-days, now `07` T-16b**. **Phase 2** takes the pipeline: template authoring UI, retry with backoff, complaint handling, digests, per-user preferences and the in-app notification centre (`07` T-29 / A-31 / A-36). The deciding argument for Phase 1 is not convenience: an NDR arriving with no `outbound_message` row to attach to **cannot be classified at all**, so deferring the row defers a Phase 1 correctness property, and without it every parse failure is a dead end a recruiter must resolve in Outlook by hand, outside the audit trail. The deciding argument against pulling more than that into Phase 1 is that the pipeline's expensive parts — sender domain configuration, template approval, bounce policy — are other people's work items and are separable from a single `Mail.Send` call. Schema effect: `03` §22 splits into migration `011a` (Phase 1 — seven of the eight tables, including `outbound_message` and its immutability grants) and `018` (Phase 2 — `notification_preference`); `03` §33.1 records why the Phase 1 half is numbered below `017b` instead of carrying a split phase label. Consent effect: `Mail.Send` must be requested in the **same** Entra admin-consent conversation as `Mail.Read` (§2.2, `07` T-07), because a second consent cycle at the point Phase 1 needs send is a weeks-long dependency (§9.3). | | 6 | **Redis is listed as cache/rate-limit/session only** (line 131), and this document uses it for the HMAC replay-nonce cache (§3.4). | None — noting for completeness | A replay nonce is a cache with a TTL, squarely inside the stated use. Volatility is acceptable: a lost nonce cache widens the replay window to 300 seconds, it does not break correctness. No divergence. | **Row 2, in full — reproduced verbatim from `02-system-architecture.md` §12.4 so no paraphrase can drift from it:** > **Migration authority ruling — canonical text (ADR 0017). Quote it; do not paraphrase it.** > > `db/migrations/NNN_*.sql` is the schema authority. Every Django migration is > `SeparateDatabaseAndState(database_operations=[RunSQL()], state_operations=[…])`, > so Django owns ordering and the applied-state ledger and authors no DDL. **Every model stays > `managed = True`; `managed = False` is used on no table**, because it would remove exactly the > tables that carry invariants from the one gate watching them. `makemigrations --check` compares > models against declared migration *state* — never against the live database — so it is kept as > the **model-vs-state** gate, and it is kept quiet not by a flag but by declaring every object > Django *can* model in `Meta.constraints` / `Meta.indexes` (`CheckConstraint`, > `UniqueConstraint(condition=…)`, `Index(Lower(…))`, `ExclusionConstraint`) and mirroring those > same declarations in `state_operations`. Objects Django cannot model at all — triggers, > column-level `GRANT`/`REVOKE`, `RANGE` partitions and their attach/detach, generated columns, > `DEFERRABLE INITIALLY DEFERRED` constraint triggers, and `procrastinate`'s vendor-managed > migrations — are named in an explicit, reviewed `db/schema-ignore.toml`, and are covered instead > by a **second, SQL-level gate**: CI builds a database by running every migration, captures > `pg_dump --schema-only --no-owner` plus a catalogue query for triggers and column privileges, > and diffs that against the committed expected dump; any difference fails the build, and updating > the expected dump is a reviewed part of the migration PR. Two gates, two failure modes, neither > one silently lying: the ORM gate catches a model that has drifted from state, the SQL gate > catches a database object that no migration created — or that a migration created and nobody > reviewed. Signed off as ADR 0017 **before migration `001` is written**; it restates the > mechanism already binding in `adr/0002-primary-relational-database.md` §3. ### 9.2 Open questions blocking or shaping this design | # | Question | Blocks | Owner | |---|---|---|---| | Q1 | Which mail provider, and does a dedicated careers **mailbox** exist? (A1, A2) | The entire §2 design; the choice between Graph and the §2.12 fallback. | Utopia Brands IT. **Ask in Phase 0** — `_decisions.md` line 269 already flags this as a weeks-long Phase 1 critical-path risk. | | Q2 | Will IT grant application-scope Graph permissions with an `ApplicationAccessPolicy`? (A3) | Least-privilege claim in §2.2; otherwise a full-mailbox credential must be accepted as a documented risk. | Talha + IT, Phase 0. | | Q3 | Model hosting: contracted API under a DPA, or self-hosted? (A5, BRD OQ-1) | Whether AI steps 7, 9, 11, 13, 17 can call an external provider at all; the whole `ai` queue sizing. | Legal + business. Already `_decisions.md`'s top-listed risk. | | Q4 | Cloud platform confirmation (A7). | The storage adapter, the emulator choice, encryption key management, and the immutable audit archive mechanism. | Business + Talha, before the first storage migration. | | Q5 | Careers-website ownership and stack (A6). | Who implements the site side of §3, and whether mTLS is available. | Marketing + Talha. | | Q6 | Backfill window for the existing careers mailbox (§2.4). | Day-one review-queue volume and the retention liability accepted on go-live. | Asfand Ahmed as Talent Lead. | | Q7 | Automatic candidate-creation policy (§4.2 step 16) — how much automation is acceptable versus a fully human triage queue? | Recruiter workload versus data-quality risk; the thresholds in the versioned matching config. | Talent Lead + Talha. Recommend starting **fully human** for the first 2-4 weeks of live intake and loosening on measured evidence, not on optimism. | | Q8 | Closed-posting grace window length (§3.6). | Candidate experience versus queue noise. | Talent Lead. | ### 9.3 Risks specific to integration and processing - **Graph tenant dependency is outside the team's control.** App registration, admin consent and mailbox creation are IT work items on someone else's backlog. Mitigation: request in Phase 0; build against `FakeMailProvider` so the intake pipeline is testable and demonstrable without a real mailbox, and the integration lands as an adapter swap rather than as a blocking dependency. - **The immutable-id mistake.** Using Graph's default (mutable) message id would make every message re-ingest after the processed-folder move. It would look like a duplicate-detection bug, not a header bug, and could burn days. Mitigation: `Prefer: IdType="ImmutableId"` set in the HTTP client's default headers, plus a specific test that moves a message and asserts the id is unchanged. - **Untrusted-file parsing runs in our own process in Phase 1.** `_decisions.md` line 271 already names this; §4.2 step 5 and §6.5 are where it bites. The Phase 1 mitigations (timeouts, memory caps, restricted user, structural gate before any parser, no outbound network from the parse step) are weaker than a sandbox. This is the split trigger most likely to fire, and the honest recommendation is to plan the Phase 2 `untrusted` queue as scheduled work rather than as a contingency. - **Parsing accuracy versus BRD §11's "no re-keying".** Step 12 deliberately leaves low-confidence fields empty. This is correct and it means recruiters **will** type into some fields. The business must accept a review step; measure and publish per-field fill rates from week one so the conversation is about numbers rather than impressions. - **A CAPTCHA verifier outage is a real availability risk on a public endpoint.** §3.5 fails open into a stricter mode by design. The tradeoff — a noisier review queue rather than blocked applicants — should be confirmed by the business, because the alternative failure mode is a careers page that silently rejects every application while the ATS looks healthy. - **Duplicate rejection emails are the worst outbound failure this system can produce.** The guard is a single `sent_at IS NULL` check inside the send job's transaction plus the idempotency key. It deserves a dedicated test that runs the same job twice concurrently and asserts one send. - **The reconciliation sweep can mask a broken webhook path indefinitely.** If nobody watches the missed-message counter, mail keeps arriving via the hourly sweep and latency quietly degrades from seconds to an hour. Mitigation: a non-zero count is an **alert**, not a metric, and it is in the top-five list in §8.1. - **Storage/database restore skew.** A blob backup and a database PITR restored to different points produce `candidate_document` rows with dangling keys, or orphans the sweep will eventually delete. Mitigation: the restore runbook restores both to the same timestamp, the orphan sweep's 30-day grace covers the gap, and the runbook is rehearsed once per phase. - **AI cost has no natural ceiling.** Steps 9, 11, 13 and 17 call a paid provider per document and per rescore, and `score.rescore_batch` fans out across every application on a requisition version. A config change on a 400-application requisition is 400 model calls. Mitigation: `input_fingerprint` short-circuits unchanged rescores (`_decisions.md`), per-capability daily cost budgets enforced in `ai_orchestration.invoke()` with a hard stop, and a cost line on the AI Studio surface so it is visible before it is a finance conversation.