HR-ATS-Portal/docs/architecture/04-integrations-and-processing.md

04 — Integrations and Asynchronous Processing

Status / Scope of this document

Design document, implementation-ready. Covers assignment §13 (careers email), §14 (careers website), §15 (CV and AI processing pipeline), §16 (background processing) and §17 (file storage), plus the external-provider abstraction that keeps email, AI, storage and malware scanning swappable.

Binding inputs: _repo-findings.md (verified repository evidence) and _decisions.md (binding architecture and database decisions). Where this document names a table, it uses the physical names from _decisions.md Part 2 (raw_intake, raw_intake_attachment, intake_parse_attempt, intake_resolution, candidate_document, ats_result).

What is not in scope here: the relational schema in detail (owned by the data-model document), authorization internals (identity/iam.can()), the scoring algorithm itself, and outbound job-board publishing beyond its integration contract — _decisions.md puts integrations_outbound in Phase 4.

Two facts constrain everything below and are stated once:

1. The repository contains no integration code of any kind. No .env or .env.example, no HTTP client, no email library, no queue, no storage client, no server-side code at all (findings §B). grep for fetch(, XMLHttpRequest, axios, WebSocket and EventSource across js/ and index.html returns nothing (findings §C). Every provider named in this document is therefore a recommendation with a labelled assumption, not a discovered fact. The prototype's inbox array (js/data.js:284-300) is generated in-browser by a seeded PRNG (js/data.js:8-10); it is a mock, not a mail integration.
2. There is no meeting transcript and no recruitment document beyond the BRD this project produced (findings §B). Requirements come from the assignment prompt and the BRD.

Confidence labels used throughout: High (follows from repository evidence or a binding decision), Medium (standard practice, low risk of being wrong), Low (depends on an unanswered question, named inline).

Vocabulary pinned once — actor and decision enums

Two closed sets appear in guards, CHECK constraints and audit rows throughout this document. They are written from one source so that a service-layer guard and the database constraint it mirrors can never compare against a value the other side cannot hold. Both are text plus a CHECK (not Postgres enum types) per the _decisions.md enum-strategy decision.

Set	Column(s)	Permitted values	Notes
Actor kind	actor_kind on audit.audit_event and every *_history table; created_by_actor_kind on outbound_message	user, system, integration, ai_agent	_decisions.md audit decision; Part 2 §9.5 and §28.1. user is the only "a real person did this" value — it is the only one that carries an actor_user_id. There is no human value. Every "AI must never auto-reject" guard is therefore written <> 'user', never != 'human'. ai_agent is the chatbot acting on_behalf_of_user_id; integration is a service principal such as the careers-form endpoint; system is a timer or an unattributed trigger write (actor_unknown = true).
Decision mode	intake_resolution.decision_mode	human, automatic	A different set with a genuinely different meaning — it records whether a triage decision was taken by a person or by the auto-create policy (§4.2 step 16). human here is correct and must not be renamed to user; it labels a mode, not an actor identity.

Reading these as one set is the mistake to avoid: decision_mode = 'human' and actor_kind = 'user' both mean "a person did it", in two different columns, deliberately.

---

1. Provider evidence and the assumption register

1.1 What the repository says about providers: nothing

Question	Repository evidence	Verdict
Which mail provider hosts the careers mailbox?	No .env, no .env.example, no mail client, no OAuth config, no SDK (findings §B). .gitignore anticipates .env files but none exist and its Python section postdates devserver.py, which findings §H explicitly warns must not be read as stack intent.	Unknown from the repository. Must be confirmed with Utopia Brands IT.
Which AI provider?	No AI code. The prototype's aiScore is int(52,98) (js/data.js:123) and js/aiassistant.js renders static markup.	Unknown. _decisions.md flags this as BRD OQ-1, unresolved, and the single decision most likely to change the design.
Which object store?	No storage client, no bucket name, no key convention. Attachments do not exist; the prototype has no file handling.	Unknown.
Which malware scanner?	Nothing.	Unknown.

I will not manufacture evidence for a provider. What follows is the recommendation and the reasoning, with the assumption labelled at every point where it is load-bearing.

1.2 Assumption register

Every assumption in this document, in one place, so a reviewer can attack them individually.

ID	Assumption	Basis	If wrong
A1	Utopia Brands runs Microsoft 365 for corporate mail, and the careers mailbox is an Exchange Online mailbox in the corporate tenant.	BRD §8.1 names Outlook as inbound channel #1; _decisions.md already assumes M365 for Entra ID SSO and recommends Azure for that reason (line 233).	Fall back to the IMAP/SMTP design in §2.12. Google Workspace would need a Gmail API adapter — same port, ~4-6 days.
A2	A dedicated shared mailbox (e.g. careers@utopiabrands.com) exists or can be created, distinct from any individual's mailbox.	Corporate norm for a careers address; required for a service-principal access policy.	If it is a distribution list, there is no mailbox to read. A distribution list forwarding into a shared mailbox is acceptable; a list with no mailbox is a blocker.
A3	Corporate IT will grant an Entra ID app registration with application (not delegated) Graph mail permissions, admin-consented, restricted to that one mailbox.	Standard M365 admin operation.	Delegated-only access forces a resource-owner flow with a service account password — worse security and a refresh-token expiry problem. _decisions.md already flags this as a Phase 1 critical-path risk (line 269) and says to start the request in Phase 0.
A4	Volume: 200-600 documents/day at peak, 20k-60k applications/year.	Carried forward from _decisions.md (line 126) so sizing is consistent across the package.	Bulk job-board feeds would raise this 1-2 orders of magnitude (_decisions.md risk, line 462) and both queue and storage sizing must be re-derived.
A5	AI model access is a contracted API provider under a data-processing agreement, called only from the worker plus one streaming endpoint.	Binding assumption in _decisions.md (line 138), pending BRD OQ-1.	Self-hosting adds GPU infrastructure and an MLOps burden two developers cannot absorb; the phase ranges break.
A6	The careers website is a separate application (marketing-owned CMS or Next.js site) that does not and will not share the ATS database.	Assignment §14 forbids shared-database access; no careers site exists in this repository.	If the careers site is later built inside this repository as a public Django app, §3.3 still applies — the public surface calls /api/v1/ like any other client and gets its own credentials and its own rate-limit bucket.
A7	Cloud is Azure (Container Apps, PostgreSQL Flexible Server, Blob Storage, Key Vault, Entra ID).	_decisions.md line 233, labelled an assumption there too.	Drives the storage decision in §6 and is the one place this document must diverge from a literal reading of the task; see §6.2 and §9.1.

---

2. Careers email — Microsoft 365 and Microsoft Graph

2.1 Recommendation and why

Recommended: Microsoft 365 / Exchange Online, accessed through Microsoft Graph v1.0 with an Entra ID application registration, client-credentials flow, restricted to a single mailbox by an Exchange ApplicationAccessPolicy. Confidence: Medium (the design is sound; the premise A1/A2/A3 is unconfirmed).

Reasoning, decisively:

• Delta queries plus change notifications remove the polling/latency tradeoff. Graph gives a server-maintained cursor (@odata.deltaLink) and a push subscription. IMAP gives UID/MODSEQ and IDLE, which work but require the client to maintain state that Graph maintains for us and re-derive it after every reconnect.
• Least privilege is actually achievable. With IMAP the credential is a mailbox password or an OAuth token for the whole mailbox; there is no way to say "this app may only touch this one mailbox". Graph application permissions are tenant-wide by default — which is worse — but New-ApplicationAccessPolicy narrows them back to a mail-enabled security group containing exactly the careers mailbox. That combination is the only configuration in this space that is genuinely least-privilege, and it is why Graph wins.
• One tenant for identity, mail and hosting. A7 puts Entra ID SSO, the Graph app registration and the container platform in the same tenant, which is the largest free reduction in integration risk available (_decisions.md line 235).
• Python SDK exists and mail volume is I/O-light at A4 volumes (_decisions.md line 91).

Rejected: Gmail API (wrong tenant under A1 — revisit only if A1 is false). A third-party inbound-mail SaaS such as Mailgun Routes or SendGrid Inbound Parse (candidate CVs would transit a processor outside controlled infrastructure; BRD §7.4 forbids it, and it also means the careers address stops being a real mailbox a recruiter can open). Reading the mailbox from a recruiter's Outlook desktop client via a plugin (no server-side durability, no audit). IMAP/SMTP as the Phase 1 default — designed in §2.12 as the fallback, not the default, because of the least-privilege point above.

2.2 App registration and least-privilege scopes

Item	Value	Rationale
Registration	Single-tenant app, name utopia-ats-mail-<env>. One registration per environment (staging registration points at a separate test mailbox, per _decisions.md line 233).	An environment boundary that shares a client id will eventually read production candidate mail from staging.
Credential	Certificate or Entra workload-identity federation, not a client secret. Stored in Key Vault; the container's managed identity reads it at boot.	Secrets rotate on a calendar; certificates and federated credentials remove the "secret expired at 2am" incident class. Never in a repository file — findings §B confirms no .env exists today and none should be added with real values.
Graph permission: Mail.Read	Application. Read messages and attachments.	Required for attachment bytes; Mail.ReadBasic.All cannot read body or attachments.
Graph permission: Mail.ReadWrite	Application. Move processed messages to folders, set categories, set isRead.	The processed/quarantine folder moves in §2.9 need write. If the business rejects any mutation of the mailbox, drop this and use a database-side cursor only — stated tradeoff: recruiters then cannot see processing state in Outlook.
Graph permission: Mail.Send	Application. Send replies as the careers mailbox.	§2.11.
Explicitly not requested	Mail.Read.All beyond the policy scope, User.Read.All, Directory.Read.All, Files.*, Sites.*, Calendars.*.	Interview calendar integration is Phase 2 (_decisions.md module 13) and gets its own scope request then, not a pre-emptive grant now.
Mailbox scoping	Exchange Online PowerShell: New-ApplicationAccessPolicy -AppId <id> -PolicyScopeGroupId ats-mail-scope@utopiabrands.com -AccessRight RestrictAccess, where that group contains only the careers mailbox. Verified with Test-ApplicationAccessPolicy against both an in-scope and an out-of-scope mailbox, and that verification is a documented go-live check.	Without this, Mail.Read application permission reads every mailbox in the tenant. This single line is the difference between a defensible and an indefensible integration.
Network	Egress to graph.microsoft.com and login.microsoftonline.com only, from the web/worker egress identity. Graph webhook ingress restricted to Microsoft's published notification source ranges at the platform WAF in addition to the token checks in §2.5.	Defence in depth; the ranges change, so the token checks remain the primary control.
Token handling	Acquired by MSAL with in-process caching, refreshed on expiry. Never logged, never persisted to the database. ChannelConnection.credential_ref (_decisions.md module 22) stores a Key Vault reference, not a credential.	Findings §J: no secret values in the repository.

Graph service limits must be verified against current Microsoft documentation at implementation time — they change, and my knowledge has a cutoff. Design to the throttling behaviour rather than to a number: honour Retry-After on 429 and 503 absolutely, cap concurrency per mailbox at a small fixed number (start at 2), use a single-flight lock per ChannelConnection so two workers never sync one mailbox concurrently, and treat sustained throttling as a monitored condition rather than an error to retry blindly.

2.3 Sync architecture: three independent mechanisms, deliberately overlapping

The requirement "never rely on webhooks alone" is met by running three mechanisms that each alone would eventually get every message, and whose combination has no single point of loss.

#	Mechanism	Cadence	Role
1	Change notification (webhook)	Push, seconds	Latency. Notifications are treated as hints only — the payload is never trusted as data.
2	Delta query	On every hint, plus a periodic tick every 5 minutes	The authoritative incremental fetch. Delta is the only thing that ever advances the cursor.
3	Reconciliation sweep	Every 60 minutes, plus a nightly deep pass	Catches everything the other two missed: dropped notifications, expired subscriptions, dead-lettered items, messages present in the mailbox with no raw_intake row.

Design rule, stated once and applied everywhere: the webhook is a doorbell, not a delivery. A notification body may be truncated, out of order, duplicated or spoofed. The handler's only job is to validate, record, and enqueue a delta run.

2.4 Initial mailbox sync (backfill)

The careers mailbox will already contain history on day one. Backfill is a distinct job from steady-state sync because it has different failure and volume characteristics.

1. Bounded by policy, not by "everything". Operator supplies a backfill_from date at connection setup. Default recommendation: 90 days, because older applications are stale for active requisitions and each one carries retention obligations the moment it is ingested (_decisions.md retention decision). Ingesting five years of CVs on day one creates a retention liability and a review queue nobody will clear. Confidence: Medium — this is a business call; surface it as a setup choice, do not hardcode.
2. Page GET /users/{mailbox}/mailFolders/{folderId}/messages filtered on receivedDateTime ge {backfill_from}, $select restricted to the fields actually used, $top=50, following @odata.nextLink. Attachments are not expanded inline — a separate per-message attachment job (§2.9) keeps page responses small and makes each attachment independently retryable.
3. Each page is processed in its own transaction: insert raw_intake rows and enqueue attachment-download jobs, then persist the page cursor. Postgres-backed queueing (procrastinate, _decisions.md line 131) makes the insert and the enqueue commit together, so a crash mid-backfill cannot produce an intake row with no job or a job with no row.
4. Backfill runs on the mail queue at low concurrency (1) and yields to steady-state sync: if a delta run is pending, the backfill job re-enqueues itself with a short delay. New applications must never sit behind a five-year backfill.
5. On completion, call .../messages/delta once to obtain the initial deltaLink, store it on ChannelConnection.delta_cursor, and only then create the webhook subscription. Order matters: a cursor established after the subscription can miss messages that arrive in the gap; establishing the cursor first means the first delta run picks them up.
6. Backfill state is resumable and idempotent — a re-run inserts nothing new because of the uniqueness rules in §2.6.

2.5 Incremental sync via delta queries

• Cursor: ChannelConnection.delta_cursor holds the opaque @odata.deltaLink. It is never parsed and never reconstructed by hand.
• A delta run: GET {delta_cursor} → for each value entry, dispatch by shape (@removed → mark the raw_intake row's provider state as removed-at-source; keep the intake row, because a deleted mail does not un-apply a candidate); follow nextLink pages; on the final page persist the new deltaLink in the same transaction as the last page's rows.
• Cursor invalidation. Graph can reject a stale or too-old delta token (410 Gone, syncStateNotFound / resyncRequired). Handling: clear the cursor, run a bounded re-sync over receivedDateTime ge now() - 7 days, obtain a fresh deltaLink, and raise a channel_resync operational event. Bounded, not full — a full re-sync of a five-year mailbox on a token expiry is a self-inflicted incident, and the reconciliation sweep in §2.7 will find anything older.
• Single-flight. A procrastinate queueing lock keyed on channel:{id} guarantees one delta run per connection at a time. Without it, two concurrent runs both advance the cursor and one of them loses the messages the other saw.
• Cursor is advanced only on a fully-persisted page. If any row in a page fails to insert for a non-idempotency reason, the transaction rolls back, the cursor is unchanged, and the page is retried. Losing a page is a correctness bug; re-reading a page is free (§2.6).

2.6 Idempotency and duplicate prevention

Four layers. Layers 1-2 are database facts, not code paths.

Layer	Mechanism	Catches
1	UNIQUE (channel_id, external_message_id) on raw_intake, where external_message_id = the Graph immutable message id (Prefer: IdType="ImmutableId" on every request) and channel_id identifies the mailbox.	Webhook redelivery, delta re-read, reconciliation overlap, backfill re-run. Insert uses ON CONFLICT (channel_id, external_message_id) DO NOTHING RETURNING id; an empty return means "already have it, nothing to do".
2	UNIQUE (channel_id, payload_sha256) on raw_intake (_decisions.md intake decision).	The same envelope arriving with a different provider id — e.g. after a mailbox migration or an IMAP fallback switch where ids are not stable.
3	raw_intake_attachment.sha256 + content-addressed storage keys (§6.1).	The same CV attached twice in one mail, or re-sent in a follow-up mail. The blob is stored once; a second raw_intake_attachment row points at the same object and is marked duplicate_of_attachment_id.
4	duplicate_candidate_pair + the global partial unique index on candidate_email.address_normalised (_decisions.md).	The same person applying twice through different channels. This is identity dedupe, not message dedupe, and it is deliberately a separate layer with a human reviewer.

Why the immutable id specifically. The default Graph message id changes when a message is moved between folders. Since §2.9 moves processed messages into a Processed folder, using the mutable id would make every processed message look like a brand-new message on the next reconciliation sweep. Prefer: IdType="ImmutableId" is therefore mandatory on every mail request, and it belongs in the HTTP client's default headers so it cannot be forgotten. This is the single most likely implementation mistake in the whole email integration.

2.7 Webhook subscriptions, renewal, failure recovery and reconciliation

Subscription lifecycle

Concern	Design
Create	POST /subscriptions with changeType: created, resource: users/{mailbox}/mailFolders/{inboxId}/messages, notificationUrl: https://<host>/api/v1/webhooks/graph/mail, lifecycleNotificationUrl pointing at the same host, a per-subscription random clientState (stored hashed in Key Vault, compared in constant time), and expirationDateTime.
Validation handshake	Graph immediately POSTs a validationToken query parameter; respond 200 with the raw token as text/plain within the provider's short timeout. This endpoint must be reachable before the subscription is created — a common first-deploy failure.
Expiry	Outlook message subscriptions have a maximum lifetime measured in days, not weeks (at time of writing ~4230 minutes for Outlook resources; verify against current Graph documentation). Treat the maximum as unknown-and-short: read expirationDateTime from the create response and drive renewal from it rather than from a hardcoded constant.
Renewal	A periodic maintenance-queue job every 15 minutes renews any subscription whose expirationDateTime is inside the next 12 hours, by PATCH /subscriptions/{id} with a new expiry. Renewing at half-life with a 15-minute check means a single failed renewal window is recoverable and does not lose the subscription.
Renewal failure	3 attempts with backoff; on continued failure delete and recreate the subscription, then force a delta run. Recreation is safe precisely because the notification is only a hint and the delta cursor is authoritative. Raise channel_subscription_recreated.
Lifecycle notifications	Handle all three: reauthorizationRequired → refresh token and PATCH to reauthorize; subscriptionRemoved → recreate then force delta; missed → force delta immediately and log a notifications_missed counter. missed is Graph telling us it dropped notifications; ignoring it is how mail silently stops arriving.
Secret rotation	clientState is regenerated on every recreate. A notification whose clientState matches neither the current nor the immediately-previous value is dropped and counted as webhook_rejected.

Webhook endpoint contract

• Path: POST /api/v1/webhooks/graph/mail. Unauthenticated by design (Graph cannot present our credentials), therefore hardened: clientState constant-time comparison, subscription id must exist and be active, body size cap, per-source rate limit, and a request-id in every log line. Optionally validate the JWT in the Authorization header when includeResourceData is used — not used here, because we deliberately do not accept resource data in the notification.
• Behaviour: validate → insert a webhook_receipt row (subscription id, provider change id, received_at, raw body, outcome) → enqueue one delta run per affected connection with a debounce (coalesce to at most one queued delta run per connection) → return 202 in under 1 second. No mail fetching, no parsing, no candidate work in the request path.
• The endpoint returns 202 even when the enqueue is coalesced. Returning 5xx to Graph triggers provider-side retries and eventually subscription removal; we would rather record the receipt and let reconciliation cover us.

Reconciliation sweep — the actual safety net

Two tiers, both on the mail queue:

Tier	Cadence	Method
Shallow	Every 60 minutes	List message ids in the Inbox and Processed folders for receivedDateTime ge now() - 48h, $select=id,receivedDateTime,internetMessageId only. Left-anti-join against raw_intake.external_message_id. Any provider id with no intake row is ingested. Any intake row still in received/parsing for more than 30 minutes is re-enqueued.
Deep	Nightly, off-hours	Same comparison over now() - 30 days, plus: intake rows whose attachments have no blob, blobs with no virus_scan_status, intake_parse_attempt rows stuck in a non-terminal state, and dead-letter entries older than 24 hours with no operator decision. Emits a single reconciliation_report row with counts per anomaly class.

The nightly report having a non-zero missed-message count is an alert, not a statistic. If reconciliation is routinely finding messages, the webhook or delta path is broken and the sweep is masking it.

Diagram — webhook failure and reconciliation recovery

sequenceDiagram
    autonumber
    participant GR as Microsoft Graph
    participant WEB as ATS web process
    participant DB as PostgreSQL
    participant WK as ATS worker
    participant OPS as On-call dashboard

    Note over GR,WEB: Normal path
    GR->>WEB: POST webhook notification with clientState
    WEB->>DB: insert webhook_receipt, enqueue delta run (debounced)
    WEB-->>GR: 202 Accepted
    WK->>DB: claim delta job, take the per-channel single-flight lock
    WK->>GR: GET deltaLink
    GR-->>WK: messages page plus new deltaLink
    WK->>DB: insert raw_intake rows and advance cursor in one transaction

    Note over GR,WEB: Failure path A - notifications dropped or endpoint down
    GR--xWEB: notification lost (deploy restart or WAF block)
    Note over WEB: no receipt row is written, so nothing is enqueued

    Note over WK,DB: Hourly shallow reconciliation
    WK->>GR: list message ids received in the last 48 hours
    GR-->>WK: id list
    WK->>DB: left-anti-join against raw_intake.external_message_id
    DB-->>WK: 3 provider ids with no intake row
    WK->>DB: ingest the 3 missing messages, increment missed_by_webhook
    WK->>OPS: alert - reconciliation found missed messages

    Note over GR,WK: Failure path B - subscription expired or removed
    GR->>WEB: lifecycle notification subscriptionRemoved
    WEB->>DB: enqueue subscription repair and a forced delta run
    WK->>GR: DELETE old subscription then POST new subscription
    GR-->>WK: new subscription id and expirationDateTime
    WK->>DB: store subscription, clientState hash and expiry
    WK->>GR: GET deltaLink (cursor was never lost)
    GR-->>WK: everything that arrived while the subscription was dead

    Note over WK,GR: Failure path C - delta cursor rejected
    WK->>GR: GET deltaLink
    GR-->>WK: 410 Gone resyncRequired
    WK->>DB: clear cursor, raise channel_resync
    WK->>GR: bounded re-sync over the last 7 days
    GR-->>WK: pages plus a fresh deltaLink
    WK->>DB: upsert on (channel_id, external_message_id) - duplicates are no-ops

2.8 Thread handling

Email threads are not applications, and conflating them is a real failure mode: a candidate replying "here is the updated CV" would otherwise create a second application.

Signal	Use
conversationId (Graph)	Stored on raw_intake.payload. Groups intakes into a conversation for recruiter display.
internetMessageId, inReplyTo, references (RFC 5322)	Stored in the payload; used as the thread key on the IMAP fallback path where conversationId does not exist.
ATS-owned token	Every outbound message from the careers mailbox carries X-Utopia-Thread: <outbound_message.public_id> and a Reply-To of careers+t.<token>@utopiabrands.com (sub-addressing) where the mailbox supports it. On inbound, the token is the primary thread key because it survives clients that strip references.

Resolution rule, in priority order: ATS token → conversationId → references chain → normalised subject plus sender within 30 days. Each inbound message still gets its own raw_intake row (arrival is a fact and must be recorded), but the resolution step (§4 step 16) proposes attach_to_existing_candidate rather than create_candidate when a thread link exists, and attaches new attachments as a new candidate_document revision rather than a new candidate. A recruiter can override in the triage UI; the override is recorded on intake_resolution with actor and reason.

Auto-reply and vacation-message suppression: messages with Auto-Submitted: auto-replied, X-Auto-Response-Suppress, or a precedence: bulk header are ingested (arrival is still a fact) but resolved automatically to reject_unusable with reason auto_reply, and never enter the review queue. Without this, every out-of-office reply to a rejection email becomes a triage item.

2.9 Attachment handling

Attachments are downloaded by a separate job per attachment, never inline with the message fetch. Rationale: a 40 MB attachment must not fail a page of 50 messages, and each attachment needs independent retry, scanning and quarantine state.

Case	Handling
Enumerate	GET /messages/{id}/attachments?$select=id,name,contentType,size,isInline — metadata first, bytes second.
Download	GET /messages/{id}/attachments/{aid}/$value streamed directly to object storage, computing sha256 during the stream. Never buffered fully in worker memory. Hard cap enforced during the stream, not from the declared size (a declared size is attacker-controlled).
Inline images	isInline: true with a contentId referenced by the HTML body → signature logos, not CVs. Recorded as attachments with is_inline = true and excluded from parse candidacy.
itemAttachment (an embedded email or contact)	Recorded; not parsed in Phase 1. If it contains a nested file attachment, one level of extraction is attempted; deeper nesting is quarantined. Rationale: unbounded nesting is a zip-bomb-shaped attack.
referenceAttachment (OneDrive/SharePoint link)	Recorded with the link; not followed in Phase 1 — following it needs Files.Read.All, which §2.2 deliberately does not request. Resolution is needs_review with a recruiter-visible reason "CV is a cloud link; download manually or ask the candidate to attach the file". Honest limitation, not a silent drop.
Multiple attachments	Every attachment gets its own raw_intake_attachment row and its own parse pipeline. Classification (§4 step 9) decides which are CVs.
Multiple CVs in one email	Fully supported and expected (agency submissions). Each CV-classified attachment produces its own intake_parse_attempt and its own proposed candidate. The intake stays in needs_review with n proposals; the recruiter confirms each independently. Never auto-create n candidates from one email — an agency PDF pack that is really one CV plus three reference letters would silently create three ghost identities. Confidence: High (this is the exact failure the raw-intake layer exists to prevent).
No CV in the email	Not an error. The intake is recorded, body text is retained, classification finds no CV, and resolution proposes needs_review with reason no_cv_attachment. A recruiter can still create a candidate from the body text (a plain-text application) or reject it. General enquiries are rejected with reason not_an_application.
Duplicate attachment	sha256 match against raw_intake_attachment in the same intake → second row marked duplicate_of_attachment_id, parsed once. sha256 match against an existing candidate_document → strong duplicate-identity signal fed to §4 step 15 and surfaced to the reviewer as "identical file already on file for candidate X".
Password-protected / corrupted / unsupported	See the failure matrix in §4.3. All three end in a terminal, recruiter-visible state with a specific reason and a "request a new file" action — never a silent discard.
Post-processing mailbox move	On terminal resolution, PATCH the message's categories (ATS/Processed, ATS/NeedsReview, ATS/Quarantined) and move it to the matching folder. Best-effort and idempotent: a failed move never fails the intake, and the reconciliation sweep scans the Processed folder too, so a message in the wrong folder is not lost.

2.10 Diagram — inbound email to intake to candidate and application

sequenceDiagram
    autonumber
    participant CA as Candidate mail client
    participant EX as Exchange Online careers mailbox
    participant GR as Microsoft Graph
    participant WEB as ATS web
    participant WK as ATS worker
    participant OS as Object storage
    participant DB as PostgreSQL
    participant REC as Recruiter

    CA->>EX: email with CV attachment
    EX->>GR: message available
    GR->>WEB: change notification (hint only)
    WEB->>DB: insert webhook_receipt and enqueue delta run
    WEB-->>GR: 202

    WK->>GR: GET deltaLink with ImmutableId preference
    GR-->>WK: message envelope and headers
    WK->>DB: insert raw_intake ON CONFLICT DO NOTHING
    Note over WK,DB: unique on channel plus external_message_id, and on payload hash
    WK->>DB: insert raw_intake_attachment rows (metadata only)
    WK->>DB: enqueue one attachment_download job per attachment

    WK->>GR: GET attachment value (streamed)
    WK->>OS: PUT object at sha256-addressed key
    WK->>DB: set object_store_key, sha256, byte_size
    WK->>DB: enqueue malware_scan

    WK->>OS: fetch bytes for scanning
    WK->>DB: virus_scan_status = clean, enqueue parse
    WK->>DB: insert intake_parse_attempt (append-only)
    Note over WK: 17-step pipeline of section 4 runs here
    WK->>DB: parse attempt succeeded, parsed jsonb plus per-field confidence
    WK->>DB: identity resolution and duplicate scan write proposals
    WK->>DB: raw_intake.state = needs_review

    REC->>WEB: open triage queue, review parsed fields and the CV side by side
    REC->>WEB: confirm - create candidate and apply to JOB-1042
    WEB->>DB: BEGIN
    WEB->>DB: insert intake_resolution (decision_mode = human, decided_by_user_id)
    WEB->>DB: insert candidate with created_from_raw_intake_id NOT NULL
    WEB->>DB: insert candidate_email (CHECK on normalised address)
    WEB->>DB: insert job_application with raw_intake_id and source_channel_id
    WEB->>DB: insert candidate_document from the attachment
    WEB->>DB: enqueue ats_scoring for the application
    WEB->>DB: COMMIT
    Note over DB: deferrable contactability trigger fires at COMMIT
    WEB->>DB: audit_event rows written by trigger using SET LOCAL actor
    WEB-->>REC: candidate CAN-5219 created, application APP-30412 created

    WK->>DB: score against pinned job_version and scoring_config_version
    WK->>DB: insert ats_result with is_current true plus criterion rows

2.11 Sending replies through the careers mailbox

Concern	Design
Transport	POST /users/{careers-mailbox}/sendMail with saveToSentItems: true, so every outbound message is visible in the mailbox a recruiter can open. For a genuine threaded reply, POST /messages/{id}/reply preserves references and conversationId; use it whenever the outbound message answers a specific inbound message.
Record of intent	Every send is an OutboundMessage row (_decisions.md module 5) written before the provider call: recipient, template version, rendered subject and body, application_id/candidate_id, thread_token, status, provider_ref, attempt_no. The row is the record; the provider call is an attempt against it.
Transactional enqueue	The row insert and the send job enqueue commit together (Postgres queue). No dual-write, so "we recorded a rejection email that was never sent" and "we sent an email we have no record of" are both impossible.
Idempotency	Idempotency key outbound:{OutboundMessage.public_id}, plus a sent_at IS NULL guard checked inside the job's transaction. A retried job that finds provider_ref already set is a no-op. Graph sendMail is not idempotent server-side, so this guard is the only thing preventing a duplicate rejection email — the single most reputationally damaging duplicate this system can produce.
Human gate	Candidate-facing rejection and offer communications require a human action to enter queued. _decisions.md makes AI structurally incapable of writing domain state and forbids terminal-negative transitions by a non-human actor; this document adds the matching rule on the delivery side: no AI-originated or rule-originated job may enqueue a candidate-facing rejection. Enforced by the send service refusing an OutboundMessage whose category = rejection and created_by_actor_kind <> 'user'. 'user', not 'human' — actor_kind is one closed set repository-wide, CHECK in ('user', 'system', 'integration', 'ai_agent') (Part 2 §9.5/§28.1, _decisions.md audit and enum-strategy decisions), and 'user' is the only member that carries an actor_user_id identifying a real person. A guard written against 'human' would compare against a value the CHECK cannot hold, so it would either always refuse or always pass depending on the comparison direction — see the vocabulary note in Status / Scope.
Suppression list	Checked before every send: unsubscribed, hard-bounced, retention-pseudonymised, and legal-hold candidates. A pseudonymised candidate has no deliverable address by construction, and the send fails closed rather than mailing a token.
Rate and batch	Per-mailbox send concurrency 1, with a modest per-minute cap. Bulk sends (e.g. 200 rejections after a requisition closes) are individual queued jobs, never one loop — so one bad address cannot abort the batch, and the throttle is enforced by the queue rather than by sleep.
Templates	config.render_template() with a pinned TemplateVersion recorded on the row, so "what exactly did we tell this candidate in March" is answerable.
Attachments outbound	Offer letters go out as generated PDFs from files, referenced by StoredFile id on the outbound row. Never a signed URL to storage in an email to a candidate — candidate-facing document access goes through candidate_access_token (_decisions.md), which has expiry and revocation.

Delivery status — what is actually knowable, stated honestly

Graph sendMail returning 202 means Exchange accepted the message for delivery. It does not mean delivered, and there is no Graph webhook for delivery. Overclaiming here would mislead recruiters into believing a candidate received a rejection they never got. The state machine therefore names its own uncertainty:

State	Meaning	Set by
draft	Row exists, not yet approved for sending	Application
queued	Approved; job enqueued	Human action (or a rule, for non-candidate-facing mail)
accepted_by_provider	Graph returned success	Send job
delivery_unconfirmed	Accepted, no DSN after 24 hours	Timer job. This is the normal terminal state for a successful send and the UI must say "sent" rather than "delivered".
bounced_hard	NDR classified as permanent (5.x.x)	Inbound NDR classifier
bounced_soft	NDR classified as transient (4.x.x)	Inbound NDR classifier; one automatic re-send after 6 hours, then failed
rejected_by_provider	4xx/5xx from Graph after max attempts	Send job
suppressed	Blocked by the suppression list before sending	Send service

Bounce detection is the same inbound pipeline, reused. NDRs arrive in the careers mailbox as ordinary messages. The inbound classifier detects multipart/report; report-type=delivery-status, extracts the DSN status code and the original recipient plus our X-Utopia-Thread token, links back to the OutboundMessage, and sets the bounce state. This is the reason §2.8's ATS-owned thread token exists: without it, matching an NDR back to the right outbound row is guesswork. NDR intakes are resolved to reject_unusable with reason bounce_report so they never enter the recruiter triage queue.

For operational investigation only, Exchange Online message trace (Reporting API) can confirm delivery to the recipient's server. It is a manual diagnostic, not a per-message pipeline — polling it for every send is disproportionate at A4 volumes. Open-tracking pixels and click-tracking on candidate mail are rejected: they are candidate surveillance without a lawful basis, and they degrade deliverability.

Audit: every state transition writes an audit_event with actor_kind (user for a recruiter action, system for a timer or classifier transition — the closed set is in Status / Scope), entity_table = 'outbound_message', before/after, and the request_id. Body content of candidate-facing mail is classified personal, so the audit payload stores the template version and a body hash, not the rendered body — the rendered body already lives on the outbound row where the retention purge can reach it (_decisions.md audit PII rule).

Diagram — outbound candidate communication with delivery tracking

sequenceDiagram
    autonumber
    participant REC as Recruiter
    participant WEB as ATS web
    participant DB as PostgreSQL
    participant WK as ATS worker
    participant GR as Microsoft Graph
    participant EX as Exchange Online
    participant CA as Candidate

    REC->>WEB: send "interview invitation" for APP-30412
    WEB->>DB: BEGIN
    WEB->>DB: check suppression list (unsubscribed, bounced, pseudonymised)
    WEB->>DB: insert outbound_message status queued, pinned template_version, thread_token
    WEB->>DB: enqueue send_email keyed on the outbound message public id
    WEB->>DB: COMMIT
    WEB-->>REC: queued for sending

    WK->>DB: claim job, re-check provider_ref IS NULL
    WK->>GR: POST sendMail with X-Utopia-Thread header and saveToSentItems
    GR-->>WK: 202 Accepted with request id
    WK->>DB: status accepted_by_provider, provider_ref, sent_at
    WK->>DB: audit_event (actor_kind user, template version, body hash)
    GR->>EX: place in Sent Items
    EX->>CA: deliver

    Note over WK,DB: 24h timer with no DSN
    WK->>DB: status delivery_unconfirmed
    Note over REC: UI shows "Sent" - never "Delivered"

    Note over EX,CA: Failure path - mailbox does not exist
    EX->>EX: generate NDR 5.1.1 to the careers mailbox
    EX->>GR: NDR message available
    GR->>WEB: change notification
    WK->>GR: delta fetch
    WK->>DB: insert raw_intake for the NDR
    WK->>DB: classify as delivery-status report, extract thread token and DSN code
    WK->>DB: outbound_message status bounced_hard, add address to suppression list
    WK->>DB: resolve intake as reject_unusable reason bounce_report
    WK->>DB: create worklist task "invitation to CAN-5219 bounced - obtain a valid address"
    WK-->>REC: in-app notification

2.12 IMAP/SMTP fallback design

Required if A1 or A3 fails (non-M365 provider, or IT refuses application permissions). Same ports (§7), a different adapter. Confidence: High that this works; Medium on effort — budget 5-8 developer-days for the adapter plus its own reconciliation tests.

Concern	Graph	IMAP/SMTP fallback
Auth	Entra app + certificate, mailbox-scoped policy	OAuth 2.0 XOAUTH2 where the provider supports it. Basic auth with a stored mailbox password is a last resort and must be escalated as an accepted risk, not adopted quietly — it is a full-mailbox credential with no scoping.
Push	Change notifications	IDLE on a long-lived connection, one connection per mailbox, with a 29-minute re-IDLE cycle and reconnect-with-backoff. Same doorbell semantics: IDLE wakes a fetch, it does not deliver data.
Cursor	Opaque deltaLink	UIDVALIDITY + UIDNEXT per folder; CONDSTORE/HIGHESTMODSEQ when the server advertises it. A UIDVALIDITY change invalidates all stored UIDs and forces a bounded re-sync — the exact analogue of a 410 resyncRequired, and the fallback's sharpest edge.
Stable id	Immutable id	Message-ID header, normalised. Weaker: it is client-generated, occasionally missing and occasionally duplicated. Compensated by making UNIQUE (channel_id, payload_sha256) (dedupe layer 2) the primary defence rather than the backstop, with a synthetic id `sha256(Message-ID
Fetch	REST, $select	FETCH (BODY.PEEK[]) — PEEK is mandatory so reading does not set \Seen and change what a recruiter sees in the mailbox. Parse MIME with Python email / mailparser.
Attachments	Per-attachment endpoint	Extracted from the MIME tree in one pass. Hard limits on part count and nesting depth (recommend 50 parts, depth 5) as an anti-zip-bomb measure — no equivalent server-side guard exists.
Threading	conversationId	ATS thread token → In-Reply-To/References → subject+sender heuristic. Materially worse; the ATS-owned token carries most of the weight.
Processed state	Folder move + categories	UID MOVE into ATS/Processed (UID COPY + \Deleted + EXPUNGE on servers without MOVE). EXPUNGE is the only destructive operation in the whole integration and needs an explicit config flag, default off.
Send	Graph sendMail	SMTP submission on 587 with STARTTLS, one connection per batch, DKIM/SPF/DMARC alignment verified for the careers domain before first production send. Same OutboundMessage state machine, same idempotency guard.
Reconciliation	§2.7	Unchanged in shape and more important: SEARCH SINCE <date> for the id set, anti-joined against raw_intake. IMAP loses notifications more often than Graph does.

The fallback's design cost is concentrated in exactly two places — id stability and cursor invalidation — and both are already covered by mechanisms the Graph path uses anyway (payload-hash uniqueness, bounded re-sync, reconciliation sweep). That is the payoff for building those as first-class layers rather than as Graph-specific workarounds.

---

3. Careers website integration

3.1 Shape of the integration

The careers website is a client of a versioned REST API, exactly like the recruiter SPA. Three interaction directions:

Direction	Mechanism	Why
Job listings (ATS → site)	Pull. GET /api/v1/public/postings with ETag/If-None-Match, 60-second site-side cache.	Pull survives an ATS outage — the site serves its cache and stays up. Push would make a failed webhook silently freeze the listings, and reconciling that requires a pull anyway.
Cache invalidation (ATS → site)	Optional POST to a site-supplied webhook on publish/update/close, with HMAC signature. Best-effort.	Reduces publish latency from 60s to ~1s. It is an optimisation; correctness never depends on it.
Application submission (site → ATS)	POST /api/v1/public/postings/{posting_public_id}/applications, multipart, service-authenticated, idempotency-keyed.	The only write the site is permitted to make.

3.2 API versioning and contract

• Namespace /api/v1/, per _decisions.md (line 202). Public endpoints sit under /api/v1/public/ with their own DRF authentication class, throttle scope, serializers and OpenAPI tag — so a change to internal serializers cannot accidentally widen the public payload. This separation is the whole point: the public contract is a deliberate, reviewed, narrow projection.
• Breaking-change policy: additive within v1; a breaking change mints /api/v2/ and v1 is supported for 90 days with a deprecation date in a Sunset header. Confidence: High — there is exactly one external consumer under A6, so this is cheap.
• Schema published by drf-spectacular; the careers site's client is generated from it, not hand-written.
• One error envelope across the whole API: { "error": { "code", "message", "field_errors", "request_id" } }. request_id is the same value in the ATS logs, which turns "the site got an error" into a single log query.

3.3 Shared database access between the public site and the ATS is forbidden

Rule: the careers website has no database credential, no connection string, no read replica and no direct table access — now or in any later phase. Its only interface is /api/v1/public/.

Six reasons, in order of severity:

1. Every invariant in _decisions.md Part 2 lives in the application service layer or in database triggers that need transaction-local context. History triggers read current_setting('app.actor_user_id'), set by ATS middleware. A direct-writing careers site would write history rows attributed to system with actor_unknown = true, or would insert a candidate row without going through the resolution flow. The raw-intake-before-candidate ordering, the deferrable contactability trigger, the audit chain and the reapplication cooling-off trigger are all premised on writes arriving through one application role from one codebase. Shared access does not weaken these guarantees — it voids them.
2. The public site is the most likely component to be compromised. It is internet-facing, CMS-shaped, and owned by a different team under A6. A database credential there is a credential to the entire candidate PII estate — candidate_email, candidate_document extracted text, offer_version compensation, the lot. Through the API, a compromise yields only what the public endpoints expose: posting reads and application writes.
3. Column-level GRANTs are the enforcement mechanism for append-only ats_result and audit_event (_decisions.md). Those grants are attached to one application role. A second role means either duplicating and maintaining the grant matrix in two places, or — what actually happens — granting the site broader privileges "for now".
4. Schema coupling freezes migrations. The site's queries become an undocumented consumer of table shapes. Splitting candidate from application, versioning requirements, and the merge re-pointing model all change table shapes; each one would then require a coordinated deploy with a team that does not report to this project. Two developers cannot absorb that.
5. No chokepoint for authorization, validation, rate limiting, file scanning or audit. All five exist in the API layer. SQL bypasses all five simultaneously.
6. Data-protection scope. Under A6 the careers site is operated by a different team, possibly on different infrastructure. Direct database access puts that infrastructure inside the processing boundary for all candidate data, not just submissions — a materially larger DPIA scope for zero engineering benefit.

Also forbidden, for the same reasons: a read-only replica for the site ("just for job listings" — job listings are a cached JSON endpoint, which is cheaper and faster anyway); a shared object-storage bucket with site write access (uploads go through the API so they are scanned before they exist as a candidate document); and a nightly SQL dump to the site's database (a stale unencrypted copy of candidate PII outside the retention machinery).

3.4 Service-to-service authentication

Layer	Mechanism
Primary	mTLS between the careers site and the ATS ingress where the platform supports it, with the client certificate pinned to a named service identity.
Credential	A per-environment service principal with a single scoped permission, public_application:create, and no user identity. Presented as a short-lived bearer token from a client-credentials exchange — not a static API key. If the platform makes token exchange awkward, the fallback is a static key with a documented 90-day rotation and two-key overlap, recorded as an accepted risk.
Request integrity	HMAC-SHA256 over timestamp + method + path + sha256(body) in an X-Utopia-Signature header, timestamp window ±300 seconds, nonce cached in Redis for the window to block replay. Belt and braces with mTLS, and the only protection if the fallback static key is in use.
Authorization	The service principal can create applications and read public postings. It cannot read candidates, cannot read applications back, cannot list submissions and cannot read scores. The submission response contains only the external reference and status. This is the important part: a compromised site cannot enumerate the candidate base.
Audit	Every public submission writes an audit_event with actor_kind = 'integration' and the service principal id.

3.5 Submission endpoint: validation, scanning, limits, idempotency

POST /api/v1/public/postings/{posting_public_id}/applications, multipart/form-data.

Control	Specification
Input validation	DRF serializer, allowlist fields only, unknown fields rejected (not ignored) so a site-side bug surfaces immediately. Name 1-200 chars; email against the same regex as the candidate_email CHECK (_decisions.md) so the API and the database agree — a mismatch means the API accepts what the database will refuse; phone normalised to E.164 or rejected; free-text fields length-capped and stored raw. No sanitisation on write — _decisions.md requires the original be preserved, and findings §E requires escaping at the rendering layer. Both are true simultaneously and neither substitutes for the other.
File constraints	Max 5 files, each ≤ 10 MB, total ≤ 25 MB. Accepted true types: PDF, DOCX, DOC, RTF, ODT, TXT. True type from magic bytes (libmagic), never from the extension or the client Content-Type; a mismatch between declared and detected type is logged as a suspicious-upload signal. Filenames are stored raw for display and never used to build a storage key (§6.1).
Streaming	Files stream to a quarantine prefix in object storage with a size cap enforced during the stream. Nothing is written to the worker filesystem.
File scanning	Malware scan is synchronous-blocking on the response only for a fast verdict path with a 5-second budget; on timeout the submission is accepted and the file stays quarantined until the async scan completes. The candidate is never made to wait on a slow scanner, and an unscanned file is never readable by a recruiter — quarantine-prefix objects are not signable (§6.4).
Rate limiting	Redis token buckets (_decisions.md: Redis is cache and rate-limit only), layered: per source IP 5/hour and 20/day; per normalised email 3/day per posting and 10/day overall; per service principal 300/hour as a circuit breaker against a looping site bug; global public-endpoint ceiling. Exceeded → 429 with Retry-After.
CAPTCHA / abuse	A privacy-preserving challenge on the site, verified server-side by the ATS (Cloudflare Turnstile recommended: no cookies, no behavioural profiling, and siteverify is a single server call). The ATS trusts nothing the site asserts about the challenge — it verifies the token itself against the provider, checks the action and hostname claims, and single-uses the token. Verifier-unavailable policy: fail open into a stricter mode (rate limits divided by 5, submissions flagged abuse_check_unavailable and forced to needs_review) rather than fail closed. A CAPTCHA provider outage must not stop real people applying for jobs; the honest tradeoff is a slightly noisier review queue for the duration.
Honeypot + timing	A hidden field that must stay empty, and a minimum form-fill time of 3 seconds. Cheap, no privacy cost, catches naive bots before the CAPTCHA quota is spent.
Idempotency	Mandatory Idempotency-Key header (site-generated UUID per submission attempt). Stored in public_submission_idempotency (key, service_principal_id, request_fingerprint, response_status, response_body, raw_intake_id, created_at) with UNIQUE (service_principal_id, key). A replay with the same key and the same fingerprint replays the stored response; the same key with a different fingerprint returns 409 Conflict. Retention 30 days. This is what makes the site's own retry-on-timeout safe.
Duplicate submissions	Three distinct cases, three different answers. (a) Network retry — same idempotency key → replayed response, one intake. (b) Impatient double-click, new key — caught by raw_intake UNIQUE (channel_id, payload_sha256); the second insert conflicts and the first intake's reference is returned, so the candidate sees success both times. (c) Genuine reapplication — passes to the reapplication rule (_decisions.md: one live application per candidate+job, cooling-off, attempt_no), which may return 409 with a plain-language reason such as "you already have an active application for this role".
Response	202 Accepted with { external_application_id, posting_public_id, received_at, status: "received" }. 202, not 201 — nothing durable has been created except a raw_intake row, and promising a created application would be a lie: parsing may fail and a recruiter may reject the submission.
Source attribution	source_channel_id → ref.source_channel value careers_website. raw_intake.payload captures referrer, UTM parameters, posting id, site locale and a coarse user-agent class. On resolution, job_application.raw_intake_id carries the attribution forward, so source reporting is a single FK hop (_decisions.md). No IP address is stored in payload beyond what the abuse controls need, and that copy expires with the idempotency record.
External application id	external_application_id = the raw_intake.public_id (UUIDv7), returned to the site and shown to the candidate. Deliberately not job_application.reference_code (APP-30412), which _decisions.md classifies as internal-only and enumerable, and which does not exist yet at 202 time.

3.6 Job publish, update and closure flows

Flow	Mechanism	Failure handling
Publish	HR-admin publishes a job_version → job_posting row created for the careers_website platform with state published and a pinned job_version_id. It becomes visible in GET /api/v1/public/postings on the next cache cycle; the optional invalidation webhook makes it near-immediate.	Webhook failure is logged, not retried past 3 attempts — the 60-second pull covers it. Publishing is never blocked by the site being unreachable.
Update	A content change mints a new job_version (immutable version rows, _decisions.md) and re-points the posting. ETag changes; the site refetches.	The posting always resolves to exactly one job_version, so an application submitted at 10:00 pins the text the applicant actually read, even if the description changed at 10:01. This is the "defensible versus indefensible rejection" property.
Closure	Posting state → closed, closed_at set. Removed from the list endpoint. GET /api/v1/public/postings/{id} returns 410 Gone with a body naming the closure reason class (filled / withdrawn / expired) so the site can render a proper page rather than a 404.	Submissions to a closed posting are accepted, not rejected, when they arrive within a 10-minute grace window — the candidate was filling the form when it closed and the fault is not theirs. The intake is flagged posting_closed_at_submission and routed to needs_review for talent-pool consideration. After the grace window, 409 with a plain-language message and a link to open roles. Confidence: Medium — the grace window length is a business preference; make it configurable.
File upload	Part of the same multipart submission, not a separate pre-signed upload.	A pre-signed direct-to-storage upload was considered and rejected for Phase 1: it hands an internet-facing site a storage write capability, and it makes the "scan before it is ever a candidate document" guarantee racy. At 10 MB and A4 volumes the proxied upload costs nothing. Revisit only if uploads exceed ~50 MB or volume exceeds ~10k/day.
Listing payload	Only fields intended for the public: title, department, business unit, location, employment type, description, posted date, closing date, and the salary range only where job_version marks it publishable. Never job_requirement weights, never internal grade, never scoring_config binding.	Weighted requirements are scoring inputs. Publishing them tells applicants exactly what to keyword-stuff, which corrupts the score.

3.7 Diagram — website application submission

sequenceDiagram
    autonumber
    participant AP as Applicant browser
    participant SITE as Careers website
    participant CAP as CAPTCHA verifier
    participant API as ATS public API
    participant RD as Redis
    participant OS as Object storage
    participant DB as PostgreSQL
    participant WK as ATS worker

    AP->>SITE: GET job listing
    SITE-->>AP: page rendered from the 60s cache of /api/v1/public/postings
    AP->>SITE: submit form with CV file and challenge token
    SITE->>API: POST /api/v1/public/postings/{id}/applications
    Note over SITE,API: mTLS, bearer from client credentials, HMAC signature, Idempotency-Key

    API->>RD: check nonce and rate-limit buckets (IP, email, principal)
    RD-->>API: within limits
    API->>CAP: verify challenge token server-side
    CAP-->>API: valid for action apply and the expected hostname
    API->>DB: SELECT posting - is it published or inside the grace window
    DB-->>API: published, pinned job_version 7
    API->>API: validate fields, detect true file type from magic bytes
    API->>OS: stream file to the quarantine prefix, hashing while streaming
    OS-->>API: object key and sha256

    API->>DB: BEGIN
    API->>DB: insert idempotency record
    API->>DB: insert raw_intake ON CONFLICT (channel_id, payload_sha256) DO NOTHING
    API->>DB: insert raw_intake_attachment with quarantine key and sha256
    API->>DB: enqueue malware_scan
    API->>DB: insert audit_event actor_kind integration
    API->>DB: COMMIT
    API-->>SITE: 202 with external_application_id (raw_intake public_id)
    SITE-->>AP: confirmation page showing the reference

    WK->>OS: fetch quarantined bytes
    WK->>WK: malware scan
    alt clean
        WK->>OS: move object to the active prefix
        WK->>DB: virus_scan_status clean, enqueue parse
        Note over WK,DB: the 17-step pipeline of section 4 runs here
    else infected
        WK->>OS: move object to the infected prefix (never deleted in Phase 1)
        WK->>DB: virus_scan_status infected, raw_intake.state quarantined
        WK->>DB: security audit_event, notify security owner
        Note over WK: terminal - no candidate is ever created, applicant is not told why
    end

Note on that last point: the applicant receives the same neutral confirmation whether the file was clean or infected. Telling a submitter "your file was flagged as malware" is a free oracle for tuning an evasion attempt.

---

4. CV and AI processing pipeline (§15)

4.1 Structure

One asynchronous pipeline, 17 steps, from an attachment arriving to a scored application. Each step is a separately queued, separately retryable job that reads and writes explicit rows — not a single long function. Rationale: a 17-stage in-process function fails as a unit, so an OCR timeout would discard the successful download, scan and extraction that preceded it.

Every step, without exception, obeys four rules:

1. Idempotent. Re-running a step with the same inputs produces the same row or no new row.
2. Append-only where it matters. intake_parse_attempt is append-only (_decisions.md), so a re-parse with a better parser is a new attempt, never an overwrite. The original payload and blob are never mutated.
3. Fails to a named, recruiter-visible state. No silent drops. Findings §F establishes that the prototype has no representation for "arrived but cannot become a candidate"; that state is the point of this design.
4. Records its own version. parser_name, parser_version, ai_model_version, prompt_template_version on the attempt, so any output is attributable and replayable.

Column key: DET = deterministic (same input, same output, no model). AI = model-based. HYB = deterministic path with a model fallback, explicitly labelled per field.

4.2 The 17 steps

#	Step	Type	Input	Output	Queue	Notes
1	Attachment enumeration	DET	raw_intake	one raw_intake_attachment per part, inline/item/reference classified	ingest	Part count and nesting depth capped. Inline signature images excluded from parse candidacy.
2	Blob persist, hash, content dedupe	DET	provider attachment stream	object in storage, sha256, byte_size, object_store_key; duplicate_of_attachment_id on a hash hit	ingest	Streamed, size-capped mid-stream, never buffered whole. A hash hit skips steps 3-13 entirely and reuses the existing intake_parse_attempt.
3	Malware scan	DET	blob	virus_scan_status ∈ pending/clean/infected/unscannable; infected → quarantined (terminal)	ingest	Scanner unavailable → pending, retried; the blob stays in the quarantine prefix and is not signable. Never "assume clean on scanner failure".
4	True-type detection and structural safety gate	DET	blob	detected_mime, is_encrypted, page_count, has_macros, declared_vs_detected_mismatch	parse	Magic bytes, not extension. Rejects at the gate: encrypted, over the page cap (recommend 100), macro-bearing DOCM, or a type not on the allowlist. Runs before any parser touches the file, because the parser is the attack surface.
5	Native text and layout extraction	DET	blob + detected type	extracted_text, layout_metadata jsonb, char_count, text_layer_ratio	parse (Phase 2: untrusted)	PDF via pdfplumber/PyMuPDF; DOCX via python-docx/mammoth; RTF/ODT/TXT via their own readers. Hard wall-clock and memory caps per document. _decisions.md line 271 flags this as the most likely early process-split trigger — parsers over attacker-supplied files are an RCE surface.
6	Text sufficiency evaluation	DET	step 5 output	needs_ocr boolean plus the reason	parse	Thresholds, not a model: char_count < 300, or text_layer_ratio < 0.1, or a page count with near-zero extractable characters. Held in versioned config, not hardcoded, so tuning is attributable.
7	OCR fallback	AI (local OCR model, versioned)	rasterised pages	extracted_text, ocr_confidence per page, ocr_engine_version	parse	Only when step 6 says so. Page cap and per-page timeout. Runs entirely in our infrastructure — CV images do not go to a third party for OCR. Low mean confidence marks the whole attempt partial, which forces human review rather than trusting a guess.
8	Normalisation and sectioning	DET (AI fallback)	text	normalised text, section offsets (experience, education, skills, contact)	parse	Unicode NFKC, de-hyphenation, header/footer stripping, column-order repair from layout_metadata. Rule-based first because it is free and explainable; the AI section-labelling fallback fires only when the rules find fewer than two sections, and that fact is recorded on the attempt.
9	Document-type classification	AI	normalised text + filename + section signals	document_kind ∈ cv/cover_letter/certificate/portfolio/id_document/not_a_cv, with confidence	ai	This is what makes "multiple attachments, one of which is a CV" work, and what stops a reference letter becoming a candidate. Cheap deterministic pre-filters (a 40-page slide deck, a 2 KB text file) short-circuit before the model call. Low confidence → needs_review, never a guess.
10	Deterministic entity extraction	DET	normalised text	emails, phones, URLs, LinkedIn handles, dates, postal locations — each with a character offset	parse	Regex and libphonenumber. Runs before the model and its results win on conflict (step 12), because a regex-matched email address is more reliable than a generated one and cannot hallucinate.
11	Structured field extraction	AI	normalised text + section offsets	schema-constrained JSON: name, contacts, employment history, education, skills, total experience — each field with a confidence and a source offset	ai	Strict output schema (JSON-schema-constrained decoding or a validated retry). Every invocation writes an AiRun with capability, model id and version, prompt template version, input reference, raw output, tokens, cost, latency (_decisions.md AI boundary). Output is a suggestion, never a domain write.
12	Reconciliation, confidence, provenance	DET	steps 10 + 11	merged field set, per-field value, confidence, source ∈ regex/model/layout, char_offset	parse	Precedence: deterministic > model. Disagreement is recorded, not resolved silently. Fields below the confidence floor are left empty rather than guessed — _decisions.md line 270 flags the gap between BRD §11's "no re-keying" and real-world parsing accuracy, and this is where that honesty is implemented. An empty field a recruiter fills is recoverable; a confidently wrong employer name is not.
13	Skill canonicalisation	HYB	extracted skill labels	candidate_skill proposals with skill_id or raw_label, confidence, source	parse	Exact → alias (ref.skill_alias) → trigram similarity above threshold → AI suggestion for unmapped labels only, proposed as a new alias for admin approval, never auto-added to the vocabulary. raw_label is always retained (_decisions.md: parser output that maps to nothing must still be storable).
14	Identity resolution proposal	DET	reconciled fields	proposal ∈ new_candidate / existing_candidate(id) / ambiguous, with per-signal evidence	ingest	Normalised email exact → E.164 phone exact → normalised LinkedIn URL → candidate_document.sha256 identical → name+employer trigram. Deterministic and fully explainable; the reviewer sees which signal fired. confirmed_distinct pairs are skipped (_decisions.md).
15	Duplicate pair detection	DET	proposal + candidate corpus	duplicate_candidate_pair rows with match_score, per-signal signals jsonb, detector_version, matching_config_version_id	ingest	Trigram GIN + rapidfuzz. Never merges. candidate_merge.performed_by_user_id is NOT NULL (_decisions.md), so an automatic merge is a database impossibility, not a policy.
16	Intake resolution	DET	everything above	intake_resolution row, then candidate + job_application + candidate_document, or a terminal non-candidate state	ingest / web	Default is needs_review with a human decision. Automatic create_candidate is permitted only with auto_create_evidence populated (a database CHECK, _decisions.md) — recommended Phase 1 policy: automatic creation only when the email matches no existing candidate, document_kind = cv with high confidence, name and email both above the confidence floor, no duplicate pair above the review threshold, and a job title resolvable to exactly one open posting. Everything else is human. Confidence: Medium on the thresholds; they belong in a versioned matching config and will be tuned.
17	ATS scoring	HYB	job_application + pinned job_version + scoring_config_version + candidate_document + parse_attempt	ats_result with is_current, plus ats_result_criterion rows carrying weight_applied, contribution, matched_evidence	score (+ai)	Aggregation is deterministic arithmetic over criterion scores; per-criterion evidence extraction for unstructured requirements is AI, with its AiRun id recorded. All five version identities are pinned on the row (_decisions.md), so a displayed historical score can never drift. input_fingerprint makes "has anything changed" an index lookup, preventing pointless rescore churn. No scoring job may set any status to rejected — review_outcome requires reviewed_by_user_id, enforced by CHECK.

Where a job title is missing (assignment's "CV with no job title" case): steps 16-17 need a posting to score against. Resolution order — an ATS thread token or posting id in the email body or subject → exact/fuzzy match of a stated title against open job_posting titles → a single open posting in the mailbox's configured default department → otherwise the candidate is created (if the identity signals are strong) as a general application with no job_application and therefore no score. General applications are a first-class outcome, not an error: the candidate enters the talent pool, appears in rematch runs (_decisions.md module 16), and a recruiter can attach them to a requisition later, which creates the application and triggers scoring then. A speculative application to a guessed requisition would produce a low score that misrepresents the candidate — which is worse than no score.

Repeat applicants are handled entirely by the model rather than by this pipeline: step 14 resolves to the existing candidate, a new candidate_document revision is added, and the reapplication rule (_decisions.md: one live application per candidate+job, attempt_no, cooling-off with an audited override) decides whether a new application is permitted. The recruiter sees the prior attempts and their outcomes in the triage view before deciding.

4.3 Failure and edge-case matrix

Every row ends in a named terminal or reviewable state. Nothing is discarded.

Case	Detected at	Behaviour	Retry	Terminal state	Recruiter sees
Password-protected PDF	Step 4	No decryption attempted — no password guessing, ever	No	parse_failed reason encrypted_document	"This CV is password protected. Request an unprotected copy." One-click templated reply.
Image-only PDF (scan)	Step 6	OCR path, steps 7-8	Yes, 2×	Normal flow; partial if OCR confidence is low	Attempt badged "OCR — verify fields" and low-confidence fields left empty
OCR yields nothing usable	Step 7/8	Attempt failed	No	parse_failed reason no_extractable_text	"Could not read this document. Ask for a text-based PDF or DOCX."
Corrupted file	Step 4/5	Parser exception caught and classified	1 retry (transient I/O)	parse_failed reason corrupt_file	"File appears damaged. Request a re-send."
Unsupported type (.pages, .zip, .heic, .jfif)	Step 4	Rejected at the gate; blob retained	No	parse_failed reason unsupported_format, with the detected type named	"We cannot read .pages files. Ask for PDF or DOCX." Recruiter may convert manually and re-upload, which creates a new attempt against the same intake.
Declared/detected type mismatch	Step 4	Processed by detected type; mismatch logged as a suspicious-upload signal	n/a	Continues if the detected type is allowed	Nothing (an ops signal, not recruiter noise)
Very large file	Step 2 (bytes) / step 4 (pages)	Rejected during the stream at the byte cap; page cap at the gate	No	parse_failed reason file_too_large / too_many_pages	"File exceeds 10 MB / 100 pages."
Zip-bomb / decompression bomb	Step 4/5	Output-size ratio cap plus wall-clock cap during extraction	No	parse_failed reason resource_limit_exceeded; blob quarantined	"File rejected for safety." Security event raised.
Multiple CVs in one email	Step 9	n independent attempts, n proposals, one intake	n/a	needs_review with n proposals	A list to confirm or reject individually
Non-CV attachment	Step 9	Classified and retained; not a candidate source	n/a	Attachment not_parsed_as_cv	Shown as a supporting document, attachable to the candidate
Duplicate attachment	Step 2	Blob stored once, second row linked	n/a	duplicate_of_attachment_id set	"Identical file already received"
No text at all after all paths	Step 8	Attempt failed	No	parse_failed reason no_extractable_text	As above
AI provider failure (5xx, auth, malformed output)	Step 9/11/17	Circuit breaker per capability (_decisions.md BRD NFR-7). Open → AI steps are skipped, not failed	3 attempts, exponential backoff + jitter	Attempt partial reason ai_unavailable; deterministic fields from step 10 are kept	"Parsed with basic extraction only — AI enrichment unavailable. Retry?" The application still exists and is still reviewable. Scoring is deferred, not faked.
AI timeout	Step 9/11/17	Per-call deadline (recommend 30 s extraction, 60 s scoring), hard-cancelled	2 attempts with a longer deadline	As above	As above
AI rate limit (429)	Step 9/11/17	Honour Retry-After; token-bucket admission control in front of every provider call so we self-throttle before the provider does	Backoff until the window clears, max 6 attempts	partial if still limited	"AI enrichment queued" — a delay, not an error
AI returns schema-invalid JSON	Step 11	One constrained retry with the validation error fed back; then abandon the AI path	1	partial reason ai_schema_violation; the raw output is retained on the AiRun for debugging	As ai_unavailable
AI output contradicts a regex match	Step 12	Deterministic wins; disagreement recorded	n/a	Continues	Field badged "AI and pattern extraction disagreed"
Scanner unavailable	Step 3	pending, blob stays quarantined and unsignable	Every 5 min, 12 attempts	scan_unavailable → dead-letter, ops alert	"File awaiting security scan" — deliberately not viewable
Worker crash mid-step	any	Job lease expires, job is re-claimed; steps are idempotent	Per the step's policy	Unchanged	Nothing
Deploy during processing	any	In-flight jobs finish or lease-expire and are retried on the new revision	n/a	Unchanged	Nothing

4.4 Diagram — CV parsing and ATS scoring pipeline

sequenceDiagram
    autonumber
    participant Q as Queue procrastinate on Postgres
    participant WK as Worker
    participant OS as Object storage
    participant SC as Malware scanner
    participant PX as Parser libraries
    participant OCR as Local OCR engine
    participant AIO as ai_orchestration
    participant PV as AI provider
    participant DB as PostgreSQL

    Q->>WK: attachment_download job
    WK->>OS: stream bytes to the quarantine prefix, hash while streaming
    WK->>DB: object_store_key, sha256, byte_size
    WK->>DB: sha256 already known?
    DB-->>WK: no
    WK->>Q: enqueue malware_scan

    Q->>WK: malware_scan
    WK->>SC: scan
    SC-->>WK: clean
    WK->>OS: move to the active prefix
    WK->>DB: virus_scan_status clean
    WK->>Q: enqueue parse_document

    Q->>WK: parse_document
    WK->>DB: insert intake_parse_attempt (parser_name, parser_version) status running
    WK->>PX: detect true type, check encryption, page count, macros
    PX-->>WK: application/pdf, not encrypted, 3 pages
    WK->>PX: extract text and layout
    PX-->>WK: 180 characters, text_layer_ratio 0.02
    Note over WK: step 6 thresholds say this is a scan - needs_ocr
    WK->>OCR: rasterise and OCR 3 pages
    OCR-->>WK: 4200 characters, mean confidence 0.86
    WK->>WK: normalise, section, deterministic entity extraction (steps 8 and 10)

    WK->>AIO: invoke document_classification, actor_kind = system, allows_system_actor = true
    AIO->>DB: insert AiRun (actor_user_id NULL, trigger_kind batch, model id and version, prompt version, input ref)
    AIO->>PV: classify
    PV-->>AIO: cv, confidence 0.94
    AIO->>DB: AiRun output, tokens, cost, latency
    AIO-->>WK: cv

    WK->>AIO: invoke cv_field_extraction, actor_kind = system, strict output schema
    AIO->>PV: extract
    PV--xAIO: 429 Too Many Requests with Retry-After 20
    Note over AIO: honour Retry-After, token bucket throttles admission
    AIO->>PV: retry after the window
    PV-->>AIO: schema-valid JSON with per-field confidence
    AIO->>DB: AiRun row
    AIO-->>WK: suggested fields

    WK->>WK: reconcile - regex wins over model, drop fields below the confidence floor
    WK->>DB: intake_parse_attempt succeeded, parsed jsonb, confidence, versions
    WK->>DB: canonicalise skills against ref.skill and ref.skill_alias
    WK->>DB: identity resolution proposal plus duplicate_candidate_pair rows
    WK->>DB: raw_intake.state = needs_review

    Note over DB: a human resolves the intake - candidate and application are created
    Q->>WK: ats_scoring job for the new application
    WK->>DB: load pinned job_version, job_requirement rows, scoring_config_version, parse attempt
    WK->>DB: compute input_fingerprint - changed since the last score?
    DB-->>WK: no current score exists
    WK->>AIO: invoke requirement_evidence for unstructured criteria only
    AIO->>PV: evidence extraction
    PV-->>AIO: per-criterion evidence with text offsets
    AIO->>DB: AiRun row
    AIO-->>WK: evidence set
    WK->>WK: deterministic weighted aggregation using weight_applied per criterion
    WK->>DB: BEGIN
    WK->>DB: insert ats_result (pins job_version, config version, document, parse attempt, model version)
    WK->>DB: insert ats_result_criterion rows with weight_applied, contribution, matched_evidence
    WK->>DB: flip any prior current row to superseded
    WK->>DB: audit_event carrying ai_run_id
    WK->>DB: COMMIT
    Note over WK,DB: no status is ever set to rejected here - review_outcome needs reviewed_by_user_id

actor_kind = 'system' on these three invocations, and why that does not weaken the chatbot guarantee. The parse of an emailed CV has no human present — the mail poller found it, the worker parsed it, and no app_user caused any of it. 03-database-design.md §27.2 originally made ai.ai_model_invocation.actor_user_id NOT NULL with actor_kind restricted to (user, ai_agent) on the grounds that "there is no service account and no system principal", which would have made these runs unable to write a ledger row at all — i.e. document_classification and cv_field_extraction, the two capabilities the Phase 1 vertical slice depends on, could not run. The two documents now agree, and the boundary is expressed in the schema rather than in prose:

Rule	Enforced by (03 §27.2)
A system run has no actor_user_id — it cannot borrow or impersonate a user	ck_run_actor_pairing
A system run can never be interactive; only batch, scheduled or webhook	ck_run_system_trigger
A system run is permitted only on a capability with ai_capability.allows_system_actor = true, which is false for every assistant, answering or ranking-for-display capability	deferred constraint trigger
Where a human did cause the work — a manual CV upload through the Import screen, a publish-triggered rescore, a recruiter-requested match — the pipeline propagates that user and must not fall back to system	service rule, asserted by a test

So a chatbot answer still cannot be produced by a principal with no permissions: a chatbot capability can never carry actor_kind = 'system'. The absolute wording in _decisions.md's AI-boundary decision is amended to match — there is no system principal for any capability that returns data to a user, which is the property the constraint was protecting.

---

5. Background processing (§16)

5.1 Queue technology

Postgres-backed durable queue via procrastinate. This is a binding decision in _decisions.md (line 131) and this document does not diverge. Restating the two properties this section depends on:

• Transactional enqueue. Insert-and-enqueue commit together, so "an intake row with no parse job" and "a parse job for a rolled-back row" are both impossible with no outbox table. Every job in the table below relies on this.
• Queueing locks serialise per-key work declaratively: one delta run per mailbox, one dedupe scan per candidate, one rescore per requisition version. This is where a naive queue produces races.

Queues, per _decisions.md: ingest, parse, score, ai, mail, maintenance, plus untrusted from Phase 2 (parsing under a restricted OS user with no outbound network). Redis is cache, rate limiting and sessions only — never a broker.

Conventions applied to every job below: exponential backoff with full jitter (bare exponential backoff synchronises retries into a thundering herd); an explicit max_attempts; a failed terminal state that surfaces in the UI rather than a silent drop; an idempotent body keyed on a domain row; a per-job wall-clock timeout; and structured logs carrying request_id, job_id, idempotency_key and the domain public id.

Dead-letter handling is one shared mechanism, not per-job code. On final failure a job writes dead_letter (queue, task_name, idempotency_key, payload jsonb, first_failed_at, last_error, attempt_count, subject_table, subject_pk, state ∈ open/retried/abandoned, triaged_by, triaged_at, note) and sets the owning domain row to its named failure state. The dead-letter queue is an operator surface with actions (retry one, retry a filtered batch, abandon with a reason), reviewed as part of the nightly reconciliation report. A dead-letter row is never auto-retried forever — auto-retry is the retry policy's job, and a DLQ that silently retries is just a slower infinite loop.

5.2 Async job catalogue

Job	Trigger	Input	Output	Status writes	Idempotency key	Retry / max attempts / backoff	Failure state	Dead-letter	User-visible error	Monitoring
mail.sync_delta	Webhook hint (debounced) + 5-min tick	channel_id, delta_cursor	raw_intake rows; advanced cursor	ChannelConnection.health, last_sync_at, IngestionRun	channel:{id} (queueing lock, single-flight)	5 attempts; 30 s → 8 min; honour Retry-After	ChannelConnection.health = degraded	Yes, per run	Admin banner "Careers mailbox sync degraded — last successful sync 14:20"	Alert if no successful sync in 30 min. Metrics: lag, messages/run, 429 rate, cursor-invalidation count
mail.backfill_page	Connection setup	channel_id, page cursor, backfill_from	raw_intake rows; page cursor	IngestionRun progress	channel:{id}:backfill:{page_cursor}	5; 30 s → 8 min	IngestionRun.failed	Yes	Setup progress bar with a resume action	Alert if stalled > 2 h
mail.renew_subscriptions	Every 15 min	active subscriptions	renewed or recreated subscriptions	subscription.expires_at	subscription:{id}:renew	3, then delete-and-recreate; 1 → 4 min	subscription.state = lost → forced delta	Yes	Admin "Mail push notifications unavailable — falling back to polling"	Alert on any subscription within 2 h of expiry, and on every recreate
mail.reconcile	Hourly (shallow) / nightly (deep)	window, channel_id	ingested missed messages; anomaly report	reconciliation_report	channel:{id}:reconcile:{window}	3; 5 → 20 min	report partial	Yes	None (ops)	Alert on any non-zero missed-message count — this is the webhook-broken signal
ingest.download_attachment	After raw_intake insert	attachment_id	blob in storage, sha256, key	raw_intake_attachment.download_state	attachment:{id}:download	5; 10 s → 5 min	download_failed	Yes	"Attachment could not be retrieved — retry"	Alert if p95 > 60 s or failure rate > 1 %
ingest.malware_scan	After download	attachment_id	verdict; blob moved to active/infected prefix	virus_scan_status	attachment:{id}:scan	12; 30 s → 5 min (scanner-down tolerant)	unscannable → intake quarantined	Yes	"Awaiting security scan" / "File blocked by security scan"	Alert on any infected. Alert if scanner unavailable > 15 min. Metric: unscanned backlog age
parse.extract_text	After clean scan	attachment_id	intake_parse_attempt with text and layout	attempt status	attachment:{id}:parse:{parser_version}	2; 1 → 5 min	parse_failed + reason	Yes	Specific per §4.3	p95 duration, failure rate by reason, memory/CPU per document
parse.ocr_fallback	Step 6 says needs_ocr	attachment_id, page range	OCR text + per-page confidence	attempt partial/succeeded	attachment:{id}:ocr:{engine_version}	2; 2 → 10 min	parse_failed reason no_extractable_text	Yes	"Could not read this scanned document"	Alert if OCR share of documents > 40 % (mailbox quality regression) or p95 > 5 min
parse.cv_fields	After sufficient text	attempt_id	parsed jsonb, per-field confidence, AiRun	attempt status; AiRun	attempt:{id}:fields:{prompt_version}	3; 20 s → 4 min; honour Retry-After	partial reason ai_unavailable	Yes	"Parsed with basic extraction only — AI unavailable"	Alert on circuit-breaker open. Metrics: token cost/day, p95 latency, schema-violation rate
ingest.resolve_identity	After parse	attempt_id	resolution proposal + evidence	raw_intake.state = needs_review	intake:{id}:resolve	3; 30 s → 5 min	resolution_failed	Yes	"Could not match this application — review manually"	Queue depth of needs_review; alert if any intake sits > 48 h
ingest.detect_duplicates	New/updated candidate; nightly rescan	candidate_id or window	duplicate_candidate_pair rows	pair state = open	candidate:{id}:dedupe:{detector_version} (lock)	3; 1 → 10 min	detector_failed	Yes	None (queue simply lacks new items)	Open-pair count trend; alert on a step change after a detector or threshold version bump
score.ats_score	Application created; requisition version published; scoring config activated; manual rescore	application_id, config version	ats_result + criterion rows	ats_result.is_current	application:{id}:score:{fingerprint}	3; 30 s → 6 min	score_failed	Yes	"Score unavailable — retry" (never a fake score, never a default of 0)	Unscored-application count; p95 score latency; alert if any application is unscored > 4 h
score.rescore_batch	Requisition version publish / config activation	job_version_id or config_version_id	fan-out of per-application score jobs	batch progress row	rescore:{job_version_id}:{config_version_id} (lock)	3; 1 → 15 min	batch_partial with a per-application failure list	Per child job	"Rescoring 240 applications — 12 failed, retry"	Batch duration; failure share
outbound.publish_posting	Posting published/updated/closed	posting_id, target platform	external id, posting state	PublishAttempt	posting:{id}:publish:{platform}:{version}	5; 1 → 30 min	publish_failed	Yes	"Not yet live on the careers site — retry"	Alert if a posting is published in the ATS but not confirmed live within 15 min
notify.send_email	Human action, or a non-candidate-facing rule	outbound_message_id	provider accept, provider_ref	OutboundMessage.status	outbound:{public_id} + sent_at IS NULL guard	5; 1 → 30 min; honour Retry-After	rejected_by_provider	Yes	"Message could not be sent — recruiter action required"	Alert on any candidate-facing send failure. Metrics: hard-bounce rate, queue age
notify.classify_bounce	Inbound NDR intake	raw_intake_id	linked OutboundMessage bounce state; suppression entry	OutboundMessage.status	intake:{id}:ndr	3; 1 → 10 min	ndr_unmatched → needs_review	Yes	"Delivery failure could not be matched to a sent message"	Hard-bounce rate; alert if unmatched NDRs > 5 % (the thread token is broken)
notify.reminders	Scheduled (interviews, offer expiry, SLA, stale intake)	window	notifications + worklist tasks	Notification, Task	reminder:{kind}:{subject_pk}:{due_bucket}	3; 1 → 10 min	reminder_failed	Yes	None (in-app)	Alert if a scheduled tick is missed — a silent scheduler is the worst failure here
analytics.refresh_read_models	Nightly + on demand	—	refreshed materialised views	ReportRun	analytics:refresh:{date} (lock)	2; 10 → 40 min	refresh_failed; last good data retained	Yes	"Dashboard data as at 02:00 — refresh in progress"	Refresh duration; alert if data staleness > 26 h
ai.generate_embeddings (Phase 2)	New/updated candidate_document	document_id, model version	candidate_embedding row	embedding_state	document:{id}:embed:{model_version}	3; 1 → 15 min	embedding_failed	Yes	None (semantic search silently narrows)	Coverage % of candidates with a current-model embedding; cost/day
ai.index_assistant_corpus (Phase 2)	Content change + nightly	changed refs	refreshed retrieval index entries	index_state	corpus:{ref}:{index_version}	3; 1 → 15 min	index_stale	Yes	Assistant states "my index was last updated at 02:00"	Index lag; alert if lag > 24 h
maintenance.retention_purge	Nightly	retention_due_on <= today, minus retention_hold	pseudonymised rows, deleted blobs, retention_action rows	retention_action	retention:{policy}:{date} (lock)	2; 1 h	purge_partial	Yes	None	Alert on any failure — this is a legal obligation. Metrics: rows purged, blobs deleted, holds skipped
maintenance.orphan_blob_sweep	Weekly	storage listing vs DB keys	orphan report; delete after a 30-day grace	orphan_candidate rows	orphan_sweep:{week}	2; 1 h	sweep_partial	Yes	None	Orphan count trend; alert on any deletion of an object with a live DB reference (never expected)
maintenance.audit_partition_maintain	Monthly	—	new partition; detached/archived old partition + final row_hash	partition_registry	audit_partition:{yyyymm}	3; 1 h	partition_failed	Yes	None	Alert if next month's partition does not exist by the 25th
maintenance.audit_hash_verify	Nightly	partition	verification result	verification_run	audit_verify:{partition}:{date}	2; 1 h	verification_failed	Yes	None	Page immediately on a hash-chain mismatch — this is tamper evidence
maintenance.dead_letter_report	Nightly	open DLQ rows	digest to admins	—	dlq_report:{date}	2; 30 min	—	No	Admin digest	Alert if open DLQ rows > 20 or any row is older than 72 h
maintenance.session_cleanup	Nightly	expired sessions/tokens	rows deleted	—	session_cleanup:{date}	2; 30 min	—	No	None	Row count

Two policy notes that apply across the table:

• User-visible errors name the object and the next action. "Processing failed" is useless to a recruiter. "Could not read Ahmed_CV.pdf — the file is password protected. Request an unprotected copy?" is actionable. Copy for every failure reason in §4.3 is a deliverable of the intake UI, not an afterthought.
• Interactive-path monitoring differs from batch. _decisions.md sets a soft process-split trigger at "p95 time-to-start for interactive AI tasks > 10 s while web p95 < 300 ms". That requires time-to-start (enqueue → claim) as a first-class metric per queue, not just duration. Instrument it in Phase 1 or the trigger cannot be evaluated when it matters.

---

6. File storage (§17)

6.1 Key convention, metadata and hashing

Content-addressed keys, environment- and class-prefixed, with no user-controlled component:

{env}/{class}/{yyyy}/{mm}/{sha256[0:2]}/{sha256[2:4]}/{sha256}

Examples:

prod/quarantine/2026/07/9f/3c/9f3c…e1  (freshly received, unscanned)
prod/active/2026/07/9f/3c/9f3ca7…e1    (scanned clean, referenced by a domain row)
prod/infected/2026/07/9f/3c/9f3ca7…e1  (quarantined, restricted ACL, never signable)
prod/generated/2026/07/…                (offer-letter PDFs, exports)

Decisions embedded in that convention, each for a reason:

Choice	Why
Content-addressed on sha256	Storage-level dedupe for free (the same CV sent five times is one object), and a key that cannot collide.
No filename in the key, ever	Candidate filenames are attacker-controlled: path traversal, unicode direction overrides, .pdf.html, 400-character names. The original filename is stored as a database column for display and set as a Content-Disposition filename only after sanitisation. This is the single most common file-storage vulnerability and the convention removes it structurally.
Two-level hash-prefix fan-out	Keeps listing and any future prefix-partitioned operation sane at millions of objects. Costs nothing.
Date segment	Makes lifecycle rules, cost attribution and "everything received in July" trivially expressible.
Explicit quarantine / active / infected / generated classes	The scan state is expressed in the key, so "unscanned files are unsignable" is enforced by a prefix check in one function rather than by remembering to check a column.
Environment prefix plus a separate bucket/container per environment	Belt and braces; see §6.7.

Metadata, split deliberately between store and database:

• Object metadata (immutable, set at write): sha256, byte_size, detected_mime, received_at, source_channel, raw_intake_public_id, retention_class, original_filename_sanitised. Enough to reconstruct provenance if the database were lost — and enough for a storage-side lifecycle rule to act on retention_class without a database lookup.
• Database of record: app.stored_file (03 §8.1) is the registry, keyed uniquely on sha256, and it owns storage_key, byte_size, mime_type, virus_scan_status and retention_class. raw_intake_attachment / candidate_document hold stored_file_id plus their own per-occurrence data — original_filename (raw, for display), attachment_index, deleted_at — and a denormalised sha256/byte_size copy for local checks. Scan status exists in exactly one place, which is the load-bearing half of §9.1 row 4's resolution. One stored_file row can be shared by N occurrences (content-addressed), so the retention sweep deletes a blob only when no non-purged row still references it (03 §8.1). files.store() / files.signed_url() in this document name the Part 1 module facade, not a schema — there is no files schema.
• No PII in keys or object metadata. No candidate name, no email. Keys appear in logs, storage access logs and CDN traces, all of which have different retention rules from candidate data.

Hashing: sha256 computed during the upload stream, never by re-reading the object. Verified on first read after write; a mismatch is a storage_integrity_error, pages ops, and blocks the parse rather than parsing a possibly-truncated file.

6.2 AWS S3 vs MinIO — evaluated against the actual deployment environment

The assignment asks for an S3-compatible design and an S3-vs-MinIO evaluation. The honest answer requires naming a conflict first: _decisions.md (line 233) recommends Azure (Container Apps, PostgreSQL Flexible Server, Blob Storage, Key Vault, Entra ID), for the strong reason that Outlook is inbound channel #1 so mail, identity and hosting land in one tenant. Azure Blob Storage is not S3-compatible. So "S3-compatible" cannot be both the literal implementation and consistent with the binding deployment decision.

Resolution, stated as a decision rather than a fudge: "S3-compatible" is adopted as the design contract, not as a vendor commitment. The files module exposes an object-store port (§7.3) whose semantics are the S3 subset every provider supports — put, get, presign-get, presign-put, head, copy, delete, list-prefix, plus server-side encryption and object versioning. Three adapters implement it, and the platform decision selects one:

Option	Fit against A7 (Azure)	Verdict
Azure Blob Storage	Same tenant, same identity model (managed identity, no keys), same portal, same Key Vault, Private Endpoint to the container platform, no egress cost between tiers, immutability policies for the audit archive, versioning and soft delete built in, lifecycle rules for retention tiering. Not S3-API.	Recommended production store if A7 holds. The adapter is ~200 lines and everything above it is provider-agnostic.
AWS S3	Best-in-class API, Object Lock (which _decisions.md names for the audit archive), mature presigning, s3:ObjectCreated events. But under A7 it means a second cloud account, a second IAM model, cross-cloud credentials in Key Vault, cross-cloud egress on every attachment read, and two consoles for two developers with no ops staff.	Recommended production store only if the cloud decision moves to AWS. Then it is the default and MinIO is local-only.
MinIO (self-hosted)	S3-API-compatible, runs anywhere, no vendor lock-in. But in production it is a stateful distributed system that this team would own: disk provisioning, erasure coding, upgrades, TLS certificates, capacity planning, and its own backup story for candidate CVs. _decisions.md rejects self-managed Postgres precisely because "backup, patching and failover become the senior developer's unpaid second job" (line 237); the identical argument applies here, and CV blobs are the least replaceable data in the system.	Rejected for production. Adopted for local development and CI, where its value is real: one docker compose service, no cloud credential in a developer's environment, and deterministic tests.

Decision. Production: the managed object store of whichever cloud is chosen — Azure Blob Storage under A7. Local and CI: Azurite when production is Azure Blob (same API, same adapter, true parity), or MinIO when production is S3. Choosing the emulator that matches the production adapter matters more than choosing the S3 API for its own sake: a MinIO-local plus Azure-production pairing means the local environment exercises an adapter that never runs in production, which is how "works locally" stops meaning anything.

Explicitly rejected: AWS Glue, and every ETL/catalogue service like it. There is no data lake, no cross-source ETL and no schema-discovery problem here. Analytics is read-only SQL views inside the one PostgreSQL database (_decisions.md module 24); adding a catalogued ETL layer would be a second data model, a second cloud dependency and a second thing to secure, for zero Phase 1 requirement.

6.3 Encryption

Layer	Design
In transit	TLS 1.2+ enforced at the store; HTTPS-only bucket/container policy; Private Endpoint or VPC endpoint so attachment traffic never traverses the public internet.
At rest	Provider-managed server-side encryption (SSE) on by default at the container level, so an object cannot be written unencrypted. Phase 1 uses provider-managed keys.
Customer-managed keys	Phase 2 upgrade: a Key Vault / KMS key with rotation, so key destruction becomes an additional erasure lever. Recommended, not Phase 1 — it adds a key-availability failure mode, and Phase 1 already has enough new moving parts. Confidence: Medium; escalate if legal requires CMK for the six jurisdictions.
Client-side / envelope encryption	Rejected for Phase 1. It would break server-side scanning, provider-side lifecycle rules and any storage-side preview, and it puts key management on two developers. Revisit only if a jurisdiction mandates it.
Application-level	None on the blob. Field-level protection is a database concern (pii_classification, masking_strategy); double-encrypting the blob buys nothing and complicates the retention purge.

6.4 Signed URLs and authorization

Rule: the authorization decision happens in the application, before a URL is minted. The signed URL is the transport, never the permission.

Control	Specification
Mint path	files.signed_url(actor, document_ref, purpose) → calls iam.can(actor, 'read', owning_domain_object) → refuses if the object is in the quarantine or infected prefix or virus_scan_status != 'clean' → mints. Access is derived from the owning domain object (_decisions.md module 3), never from possession of a key.
TTL	5 minutes for read, 15 minutes for a direct upload (not used in Phase 1, §3.6). Short enough that a leaked URL in a chat log or a support ticket is dead on arrival.
Response headers	Forced Content-Disposition: attachment; filename="<sanitised>" and Content-Type from our detected type, never the uploader's declared type. Prevents an HTML or SVG "CV" executing script in a recruiter session — which matters enormously given findings §E establishes there is no escaping anywhere in the existing frontend.
Serving origin	CV blobs are served from a separate hostname from the application. Even with Content-Disposition: attachment, a same-origin file-serving path is one misconfiguration away from a stored-XSS vector against a recruiter session.
Audit	Every mint writes an audit_event with action = 'document.access', actor, document ref and purpose. _decisions.md requires explicit application writes for access events because no trigger can observe a read; a CV download is exactly such an event and is the compliance-relevant one.
Candidate-facing access	Never a signed URL in an email. A candidate_access_token (hashed, scoped, expiring, revocable, _decisions.md) is exchanged server-side for a short-lived stream. Entity ids are identifiers, not capabilities.
Bucket policy	Public access blocked at the account and container level, not merely absent. Anonymous listing and anonymous read denied explicitly, and a CI check asserts it per environment.
Enumeration	Keys are sha256-derived and unlisted; a signed URL grants access to one key only; list is not exposed through the port to any caller except the orphan sweep, which runs as a distinct storage identity.

6.5 Malware scanning

Concern	Design
Placement	Between download/upload and any parse. A blob in the quarantine prefix is unsignable and unparseable — enforced by a prefix check in files.signed_url() and in the parse job's precondition, not by convention.
Engine	ClamAV in the worker image (or a sidecar) for Phase 1: no per-file cost, no candidate CV leaving controlled infrastructure (BRD §7.4), signature updates by scheduled job. Stated limitation honestly: ClamAV's detection rate on targeted or novel malware is materially below a commercial multi-engine service. It is a hygiene control, not a guarantee.
Upgrade path	A cloud-native scanner (Microsoft Defender for Storage under A7) or a multi-engine service, behind the same MalwareScanner port (§7.4). Trigger to upgrade: any confirmed infected submission, or a legal/insurance requirement.
Signature freshness	Scanner signature version recorded on every scan. Signatures older than 48 h raise an alert; older than 7 days, scanning fails closed to unscannable rather than issuing a verdict a stale engine cannot support.
Rescan on signature update	Weekly rescan of blobs received in the last 30 days (a cheap window, catches same-day-zero-day). A newly-infected verdict on an already-active blob moves the object to the infected prefix, revokes access, alerts security, and creates a worklist item — it does not delete the candidate.
Infected handling	Object moved to the infected prefix with a restricted ACL. Retained, not deleted, in Phase 1: it is evidence, and BRD-side incident handling may need it. Retention 90 days, then purge. raw_intake.state = quarantined (terminal, no candidate). Applicant receives the same neutral confirmation as everyone else (§3.7).
Beyond signatures	Structural gate at step 4 (macros, embedded JavaScript in PDFs, embedded executables, decompression ratio caps) catches a class ClamAV does not. Parsing runs with a hard timeout, memory cap, restricted OS user and — from Phase 2 — no outbound network (_decisions.md line 110). _decisions.md line 271 is right to name this the most likely early process-split trigger.

6.6 Versioning, backups, retention, deletion and orphans

Concern	Design
Object versioning	On for active and generated; not needed for quarantine. Content-addressed keys mean a "new version" of a CV is a new object anyway, so versioning here is protection against accidental overwrite or deletion, not a revision mechanism. CV revisions are modelled as candidate_document rows (_decisions.md), which is where they belong.
Soft delete (storage-level)	Provider soft-delete / delete-retention 14 days, matching the database PITR window (_decisions.md line 233) so a point-in-time restore always has its blobs.
Soft delete (application-level)	deleted_at on candidate_document; the blob is untouched. Excluded from the live views (v_candidate_live) and unsignable. Soft delete is not erasure and the UI must never call it "deleted permanently" — _decisions.md is explicit on this.
Permanent delete	Only two paths: (a) the retention purge, whose action is pseudonymisation of database columns plus real deletion of CV blobs, recording blob_keys_deleted on retention_action; (b) an explicit legal erasure request, executed by the same code path with a distinct policy key. Both are audited, both skip retention_hold subjects, and both require the blob-delete to succeed or the whole subject's purge is marked partial and re-queued — a pseudonymised database row whose CV blob still exists is a false claim of erasure.
Backups	Geo-redundant storage (GRS/GZRS or cross-region replication) with a documented RPO 15 min / RTO 4 h for blobs. Blob backup and database PITR must be restorable to the same point, or a restore produces candidate_document rows with dangling keys. Restore is a rehearsed runbook, tested once per phase; an untested restore is not a backup.
Retention classes	raw_attachment (retain with the intake), candidate_document, generated_document (offer letters — longer, contractual), audit_archive (immutable, WORM, _decisions.md layer 4). Class is on the object metadata and the database row, so a provider lifecycle rule and the application purge agree.
Immutability for audit archive	Azure immutable blob policy / S3 Object Lock in compliance mode on the audit_archive container, with the closed partition's final row_hash recorded alongside. This is the only genuinely independent tamper check (_decisions.md is explicit that hash chaining is evidence, not prevention).
Orphan cleanup	Weekly maintenance.orphan_blob_sweep: list the store by prefix, anti-join against object_store_key across raw_intake_attachment, candidate_document and generated-document tables. An object with no reference and older than 30 days is reported, then deleted on the following run if still orphaned. Two-pass with a grace period is deliberate — a single-pass delete races an in-flight upload whose database row has not yet committed, and the failure mode is destroying a candidate's only CV. The reverse anti-join (a database row pointing at a missing object) is a storage_integrity_error and pages immediately; that direction is data loss, not garbage.

6.7 Environment separation

Rule	Detail
Separate storage accounts/buckets per environment	utopia-ats-prod, utopia-ats-staging, utopia-ats-dev. Not prefixes in one bucket. A prefix is one IAM-policy typo away from staging writing to production; separate accounts make cross-environment access require a credential that does not exist.
No production credential outside production	Distinct managed identities; production storage grants access only to the production container revisions. Developers have no production data credential; production diagnosis is by log and metric, or by an audited break-glass grant.
Non-production data	Never a copy of production blobs. Staging gets synthetic CVs plus a pseudonymised extract produced by the anonymisation script that reads pii_classification (_decisions.md). Copying real CVs into staging silently widens the processing boundary and defeats the retention machinery.
Local development	Azurite (or MinIO, per §6.2) in docker compose, seeded with synthetic fixtures. No cloud credential on a laptop.
Test mailbox	The staging Graph app registration points at a dedicated test mailbox (_decisions.md line 233). A staging deploy must be structurally incapable of reading production candidate mail — separate app registration, separate access policy, separate mailbox.
CI	Ephemeral MinIO/Azurite service container plus the real Postgres service container _decisions.md mandates. No external provider is contacted from CI; email, AI, storage and scanner ports are all faked at the port boundary (§7).

---

7. External provider abstraction

7.1 Rule

Every external dependency sits behind a port — a typing.Protocol in the owning platform module — with adapters selected by configuration. Only the owning module may import a provider SDK; an import-linter forbidden-contract makes an SDK import anywhere else a failed build (_decisions.md line 19 already enforces boundaries this way, so this is the same mechanism, not a new one).

Four ports, four owners:

Port	Owner module	Adapters	Selected by
MailProvider	integrations_inbound (+ notifications for send)	GraphMailProvider, ImapSmtpMailProvider, FakeMailProvider	ChannelConnection.type
AiProvider	ai_orchestration — the only package permitted a model client (_decisions.md line 138)	HostedApiAiProvider, LocalOcrProvider, FakeAiProvider	ModelConfigVersion
ObjectStore	files	AzureBlobObjectStore, S3ObjectStore (AWS or MinIO), FakeObjectStore	env config
MalwareScanner	files	ClamAvScanner, DefenderForStorageScanner, FakeScanner	env config

Three properties the ports must have, or they are decoration:

1. Provider-neutral vocabulary. No messageId, deltaLink, Bucket or blobClient in a signature. Cursors are opaque str; ids are provider_message_id: str.
2. Provider-neutral errors. Every adapter maps its SDK's exceptions onto one taxonomy: Transient, RateLimited(retry_after), AuthFailure, NotFound, CursorInvalid, Permanent. Retry policy is written once against that taxonomy. Without this, retry logic leaks provider knowledge into every call site — which is how a "swappable" provider turns out not to be.
3. The fake is a first-class adapter, shipped in the main package and used by CI and local development. Its fixtures include the ugly cases: a 410 cursor invalidation, a 429 with Retry-After, a message with three attachments and one CV, an encrypted PDF, an image-only PDF, an NDR, and a schema-violating AI response. A fake that only does the happy path tests nothing worth testing.

7.2 MailProvider

from typing import Protocol, Iterator, Sequence
from dataclasses import dataclass
from datetime import datetime

@dataclass(frozen=True)
class MailAttachmentRef:
    provider_attachment_id: str
    filename: str
    declared_mime: str
    declared_size: int
    is_inline: bool
    kind: str  # "file" | "item" | "reference"

@dataclass(frozen=True)
class MailMessage:
    provider_message_id: str        # stable across folder moves (Graph immutable id)
    thread_key: str | None          # conversationId, or the ATS thread token
    internet_message_id: str | None
    in_reply_to: str | None
    references: Sequence[str]
    received_at: datetime           # tz-aware, UTC
    sender_address: str
    sender_display_name: str | None
    to: Sequence[str]
    cc: Sequence[str]
    subject: str
    body_text: str | None
    body_html: str | None           # stored raw; escaped at render (findings section E)
    headers: dict[str, str]
    attachments: Sequence[MailAttachmentRef]
    raw_envelope: dict              # -> raw_intake.payload

@dataclass(frozen=True)
class MailPage:
    messages: Sequence[MailMessage]
    next_cursor: str | None         # opaque; None means the page set is exhausted
    delta_cursor: str | None        # opaque; set only on the final page

@dataclass(frozen=True)
class SubscriptionInfo:
    provider_subscription_id: str
    expires_at: datetime
    client_state_hash: bytes

@dataclass(frozen=True)
class SendResult:
    provider_ref: str
    accepted_at: datetime

class MailProvider(Protocol):
    def list_since(self, folder: str, since: datetime, cursor: str | None) -> MailPage: ...
    def delta(self, cursor: str) -> MailPage: ...                      # raises CursorInvalid
    def initial_delta_cursor(self, folder: str) -> str: ...
    def fetch_attachment(self, provider_message_id: str,
                         provider_attachment_id: str) -> Iterator[bytes]: ...
    def list_message_ids(self, folder: str,
                         since: datetime) -> Iterator[tuple[str, datetime]]: ...  # reconciliation
    def move_message(self, provider_message_id: str, folder: str) -> str: ...
    def tag_message(self, provider_message_id: str, categories: Sequence[str]) -> None: ...
    def send(self, *, to: Sequence[str], subject: str, body_html: str,
             body_text: str, headers: dict[str, str],
             attachments: Sequence[tuple[str, str, Iterator[bytes]]],
             reply_to_provider_message_id: str | None) -> SendResult: ...
    def create_subscription(self, folder: str, notification_url: str,
                            lifecycle_url: str, client_state: str) -> SubscriptionInfo: ...
    def renew_subscription(self, provider_subscription_id: str) -> SubscriptionInfo: ...
    def delete_subscription(self, provider_subscription_id: str) -> None: ...
    def capabilities(self) -> frozenset[str]: ...

capabilities() is how the IMAP adapter declares honestly what it cannot do — e.g. {"push_idle", "server_threading_absent", "no_immutable_ids"} — so the sync orchestrator can lean harder on payload-hash dedupe and the ATS thread token instead of silently producing worse results. list_message_ids exists purely for reconciliation and is deliberately cheap (ids and timestamps only) so the hourly sweep costs almost nothing.

7.3 ObjectStore

class ObjectStore(Protocol):
    def put_stream(self, key: str, stream: Iterator[bytes], *, content_type: str,
                   metadata: dict[str, str], max_bytes: int) -> "PutResult": ...
                   # computes sha256 while streaming; raises SizeLimitExceeded mid-stream
    def get_stream(self, key: str) -> Iterator[bytes]: ...
    def head(self, key: str) -> "ObjectInfo": ...
    def copy(self, src_key: str, dst_key: str) -> None: ...
    def delete(self, key: str, *, permanent: bool = False) -> None: ...
    def presign_get(self, key: str, *, ttl_seconds: int, filename: str,
                    content_type: str) -> str: ...
    def presign_put(self, key: str, *, ttl_seconds: int, max_bytes: int) -> str: ...
    def list_prefix(self, prefix: str, *,
                    since: datetime | None = None) -> Iterator["ObjectInfo"]: ...

presign_get requires filename and content_type rather than accepting the object's own values, so §6.4's forced-download rule cannot be forgotten at a call site. put_stream takes max_bytes because a declared size is attacker-controlled and the cap must bind during the transfer.

7.4 MalwareScanner and AiProvider

class MalwareScanner(Protocol):
    def scan_stream(self, stream: Iterator[bytes]) -> "ScanVerdict": ...
    # ScanVerdict(status: clean|infected|unscannable, signature_name: str | None,
    #             engine: str, engine_version: str, signature_version: str,
    #             scanned_at: datetime)
    def engine_status(self) -> "EngineStatus": ...   # signature freshness, drives fail-closed

class AiProvider(Protocol):
    def complete_structured(self, *, prompt: str, output_schema: dict,
                            model: str, max_tokens: int,
                            timeout_s: float) -> "AiResult": ...
    def complete_stream(self, *, prompt: str, model: str,
                        timeout_s: float) -> Iterator[str]: ...
    def embed(self, *, texts: Sequence[str], model: str) -> Sequence[Sequence[float]]: ...
    def model_info(self, model: str) -> "ModelInfo": ...   # id, version, context, pricing

AiProvider is intentionally narrow and not the module boundary. Callers never touch it; they call ai_orchestration.invoke(capability, context, actor), which writes the AiRun row before the result is usable, calls iam.can() with the human actor, and returns a suggestion object rather than a domain write (_decisions.md line 138). The circuit breaker, the token-bucket admission control, cost accounting and prompt-version pinning all live in invoke(), so no adapter and no caller can bypass them.

---

8. Cross-cutting operational requirements

8.1 Observability

• Structured JSON logs with request_id (propagated from the API layer through the queue payload into every job — this is what makes a webhook receipt traceable to a score row), job_id, idempotency_key, channel_id, and the domain public id. Never a bigint PK, never a candidate name, never an email address, never a blob byte.
• Metrics per integration: sync lag, messages ingested, attachments downloaded, scan verdicts, parse success rate by reason, OCR share, AI latency/tokens/cost/error rate by capability, queue depth and time-to-start per queue, DLQ depth and age, unresolved-intake age, unscored-application count, outbound bounce rate.
• The five alerts that matter most, in priority order: (1) no successful mail sync in 30 minutes; (2) reconciliation found missed messages; (3) any intake unresolved for over 48 hours; (4) audit hash-chain verification failure; (5) retention purge failure. Each has a named owner and a runbook line. Everything else is a dashboard.
• Health endpoints distinguishing liveness from readiness, plus /healthz/integrations reporting per-channel health, subscription expiry, scanner signature age and AI circuit-breaker state — one page a recruiter's admin can read before escalating.

8.2 Degradation ladder

Stated explicitly so behaviour under partial failure is a design choice rather than an emergent accident (BRD NFR-7):

Failure	System behaviour
AI provider down	Intake, parsing (deterministic path), candidate creation, applications and all recruiter workflow continue. Scoring is deferred, never defaulted. Assistant is unavailable with an honest message.
OCR unavailable	Text-layer PDFs and DOCX parse normally; scanned documents queue as needs_ocr and are recruiter-visible as pending.
Malware scanner down	Intake continues; files stay quarantined and unviewable; a visible backlog counter with age. Never "assume clean".
Object storage down	Intake stops — this is correct. Accepting a submission we cannot store the CV for would be data loss dressed up as success. The public endpoint returns 503 with Retry-After so the site can tell the applicant to try shortly.
Mail provider down	Web tier unaffected; sync retries with backoff; delta cursor guarantees nothing is lost on recovery.
Careers site down	ATS unaffected. Postings simply are not fetched.
Queue/worker down	Web tier serves reads and writes; async work accumulates durably in Postgres and drains on recovery. Nothing is lost, because the queue is the database. This is the payoff for the procrastinate decision.
Database down	Everything stops. Single point of failure, accepted deliberately by _decisions.md (line 235) for an internal business-hours tool, with PITR as the recovery path. It should be stated to the business rather than discovered.

8.3 Work split for this document's scope

Per _decisions.md (line 259). Every Ahmed item is independently demonstrable and has a Talha review checkpoint.

Owner	Items
Talha	Entra app registration and access-policy verification; the MailProvider port and Graph adapter; subscription lifecycle and reconciliation; ai_orchestration and the AiProvider port; the parsing/OCR pipeline and its resource sandboxing; scoring; queue configuration and locks; the ObjectStore port and adapters; storage IAM and encryption.
Ahmed	The public submission endpoint with its DRF serializer + Zod pair (validation, an independently demonstrable slice); the idempotency-record implementation and its test matrix; rate-limit and CAPTCHA verification wiring including the fail-open-into-strict-mode path; the FakeMailProvider fixture library including the ugly cases (high-value, genuinely interesting, and it is what makes the whole integration testable); the intake triage queue UI and the failure-reason copy for every case in §4.3; the dead-letter operator screen with retry and abandon actions (a real state machine, not CRUD); the /healthz/integrations page; the orphan-blob sweep report; Playwright journeys for ingest → parsed → promoted.

---

9. Inconsistencies, open questions and risks

9.1 Inconsistencies with _decisions.md found while writing this document

The binding resolution for all six is in _open-items.md. This table is the evidence and the position this document took; the ruling is there, and the "Proposed resolution" column is superseded wherever it differs. 1 → OPEN-04 (cloud platform, therefore object storage — owner: business + Talha), 2 → RULING-02, 3 → RULING-03 (_glossary.md), 4 → RULING-05 (the registry is app.stored_file at migration 005, not files.stored_object; the names describe the same table and this row's proposal is superseded, not overruled — and the ruling adopts this row's load-bearing point that scan status must exist in exactly one place), 5 → OPEN-07 (the minimal Phase 1 notifications slice — owner: Talent Lead + Talha), 6 → C-08 (no divergence). §9.2's Q1–Q4 → OPEN-06, OPEN-06, OPEN-03, OPEN-04.

#	Inconsistency	Severity	Proposed resolution
1	Azure Blob Storage vs "S3-compatible". _decisions.md line 233 recommends Azure Blob Storage; the assignment (§17) asks for an S3-compatible design and an AWS-S3-vs-MinIO evaluation. Azure Blob is not S3-API-compatible.	Medium	Resolved in §6.2 by treating "S3-compatible" as the port contract and providing three adapters. Production = the chosen cloud's managed store (Azure Blob under A7). The cloud decision must be confirmed before the storage adapter is written; it is cheap now and expensive after the retention and lifecycle rules exist.
2	Plain-SQL migrations vs Django. Part 1 (line 91) chooses Django partly because migrations are built in; Part 2 (line 290) mandates ordered plain-SQL migrations with "the ORM never generates the schema". Part 2's own risk list (line 463) already flags this.	High — it is a daily workflow decision for both developers.	Settled, not flagged. The binding ruling is the canonical text in 02-system-architecture.md §12.4 (ADR 0017), quoted verbatim immediately below this table. Two consequences that matter to this document: procrastinate's vendor-managed migrations are ignore-listed explicitly, not by omission (§5); and the intake-side objects this document depends on — the raw_intake partial unique indexes, the idempotency-record unique index, the CHECK regexes on email addresses — are all declarable in Meta.constraints/Meta.indexes, so they stay inside the ORM gate rather than being handed to the SQL gate.
3	Intake entity names differ between parts. Part 1 module 6 names InboundSubmission, SubmissionAttachment, ProcessingAttempt; Part 2 names raw_intake, raw_intake_attachment, intake_parse_attempt.	Low	Same design, two vocabularies. This document uses Part 2's physical names throughout. The data-model document should pick one set and state the mapping once.
4	Blob registry ownership — RESOLVED. Part 1 module 3 gives files a StoredFile(sha256, mime, size, storage_key, scan_status, retention_class) table; Part 2 puts object_store_key, sha256 and virus_scan_status directly on raw_intake_attachment and candidate_document. Taken literally, the same blob's scan status lives in two places.	Medium — a two-place scan status is exactly how "unscanned file becomes viewable" happens.	Resolved as app.stored_file — 03-database-design.md §8.1 defines it and 03 §32.1 row 2 rules on it ("adopts Part 1's registry and keeps Part 2's columns as denormalised domain data"). Two corrections to this row's original wording: the table is app.stored_file, not files.stored_object — there is no files schema; 03 §1.1 lists five application schemas (ref, app, ai, audit, staging) plus the library-owned queue, and files.* elsewhere in this document names the Part 1 module facade (files.store(), files.signed_url()), not a schema. And this row's content-addressed keying is what was adopted, in its strong form: uq_stored_file_sha256 means five candidates who send the same file share one row and one scan, with 03 §8.1 now stating explicitly what that implies for retention — a blob is deleted only when no non-purged row references it, tested by NOT EXISTS rather than a stored refcount. stored_file.virus_scan_status is the only scan status; the domain rows' copies are denormalised read conveniences and are never written independently.
5	notifications is Phase 2, but inbound email is Phase 1. Part 1 module 5 phases notifications (including OutboundMessage and bounce handling) at Phase 2, while integrations_inbound is Phase 1. Yet Phase 1 needs at least "request an unprotected copy of your CV" replies for the §4.3 failure cases to be actionable, and NDR classification depends on the outbound row existing.	Medium	Accepted and ruled — this row is now settled, not a recommendation. The package had four different phasings (this row at 1, 07 §5.1 and 03 §5 / §33 at 2, 00 DEF-07 at 2–3, 00 OBD-21 at 3); 08 §7 finding 10 records the conflict and its resolution. The split: Phase 1 takes the minimal slice — the outbound_message row, Mail.Send through the same MailProvider port as §2, the send idempotency guard, NDR classification, and one seeded transactional template — 4–6 developer-days, now 07 T-16b. Phase 2 takes the pipeline: template authoring UI, retry with backoff, complaint handling, digests, per-user preferences and the in-app notification centre (07 T-29 / A-31 / A-36). The deciding argument for Phase 1 is not convenience: an NDR arriving with no outbound_message row to attach to cannot be classified at all, so deferring the row defers a Phase 1 correctness property, and without it every parse failure is a dead end a recruiter must resolve in Outlook by hand, outside the audit trail. The deciding argument against pulling more than that into Phase 1 is that the pipeline's expensive parts — sender domain configuration, template approval, bounce policy — are other people's work items and are separable from a single Mail.Send call. Schema effect: 03 §22 splits into migration 011a (Phase 1 — seven of the eight tables, including outbound_message and its immutability grants) and 018 (Phase 2 — notification_preference); 03 §33.1 records why the Phase 1 half is numbered below 017b instead of carrying a split phase label. Consent effect: Mail.Send must be requested in the same Entra admin-consent conversation as Mail.Read (§2.2, 07 T-07), because a second consent cycle at the point Phase 1 needs send is a weeks-long dependency (§9.3).
6	Redis is listed as cache/rate-limit/session only (line 131), and this document uses it for the HMAC replay-nonce cache (§3.4).	None — noting for completeness	A replay nonce is a cache with a TTL, squarely inside the stated use. Volatility is acceptable: a lost nonce cache widens the replay window to 300 seconds, it does not break correctness. No divergence.

Row 2, in full — reproduced verbatim from 02-system-architecture.md §12.4 so no paraphrase can drift from it:

Migration authority ruling — canonical text (ADR 0017). Quote it; do not paraphrase it.

db/migrations/NNN_*.sql is the schema authority. Every Django migration is SeparateDatabaseAndState(database_operations=[RunSQL(<that file>)], state_operations=[…]), so Django owns ordering and the applied-state ledger and authors no DDL. Every model stays managed = True; managed = False is used on no table, because it would remove exactly the tables that carry invariants from the one gate watching them. makemigrations --check compares models against declared migration state — never against the live database — so it is kept as the model-vs-state gate, and it is kept quiet not by a flag but by declaring every object Django can model in Meta.constraints / Meta.indexes (CheckConstraint, UniqueConstraint(condition=…), Index(Lower(…)), ExclusionConstraint) and mirroring those same declarations in state_operations. Objects Django cannot model at all — triggers, column-level GRANT/REVOKE, RANGE partitions and their attach/detach, generated columns, DEFERRABLE INITIALLY DEFERRED constraint triggers, and procrastinate's vendor-managed migrations — are named in an explicit, reviewed db/schema-ignore.toml, and are covered instead by a second, SQL-level gate: CI builds a database by running every migration, captures pg_dump --schema-only --no-owner plus a catalogue query for triggers and column privileges, and diffs that against the committed expected dump; any difference fails the build, and updating the expected dump is a reviewed part of the migration PR. Two gates, two failure modes, neither one silently lying: the ORM gate catches a model that has drifted from state, the SQL gate catches a database object that no migration created — or that a migration created and nobody reviewed. Signed off as ADR 0017 before migration 001 is written; it restates the mechanism already binding in adr/0002-primary-relational-database.md §3.

9.2 Open questions blocking or shaping this design

#	Question	Blocks	Owner
Q1	Which mail provider, and does a dedicated careers mailbox exist? (A1, A2)	The entire §2 design; the choice between Graph and the §2.12 fallback.	Utopia Brands IT. Ask in Phase 0 — _decisions.md line 269 already flags this as a weeks-long Phase 1 critical-path risk.
Q2	Will IT grant application-scope Graph permissions with an ApplicationAccessPolicy? (A3)	Least-privilege claim in §2.2; otherwise a full-mailbox credential must be accepted as a documented risk.	Talha + IT, Phase 0.
Q3	Model hosting: contracted API under a DPA, or self-hosted? (A5, BRD OQ-1)	Whether AI steps 7, 9, 11, 13, 17 can call an external provider at all; the whole ai queue sizing.	Legal + business. Already _decisions.md's top-listed risk.
Q4	Cloud platform confirmation (A7).	The storage adapter, the emulator choice, encryption key management, and the immutable audit archive mechanism.	Business + Talha, before the first storage migration.
Q5	Careers-website ownership and stack (A6).	Who implements the site side of §3, and whether mTLS is available.	Marketing + Talha.
Q6	Backfill window for the existing careers mailbox (§2.4).	Day-one review-queue volume and the retention liability accepted on go-live.	Asfand Ahmed as Talent Lead.
Q7	Automatic candidate-creation policy (§4.2 step 16) — how much automation is acceptable versus a fully human triage queue?	Recruiter workload versus data-quality risk; the thresholds in the versioned matching config.	Talent Lead + Talha. Recommend starting fully human for the first 2-4 weeks of live intake and loosening on measured evidence, not on optimism.
Q8	Closed-posting grace window length (§3.6).	Candidate experience versus queue noise.	Talent Lead.

9.3 Risks specific to integration and processing

• Graph tenant dependency is outside the team's control. App registration, admin consent and mailbox creation are IT work items on someone else's backlog. Mitigation: request in Phase 0; build against FakeMailProvider so the intake pipeline is testable and demonstrable without a real mailbox, and the integration lands as an adapter swap rather than as a blocking dependency.
• The immutable-id mistake. Using Graph's default (mutable) message id would make every message re-ingest after the processed-folder move. It would look like a duplicate-detection bug, not a header bug, and could burn days. Mitigation: Prefer: IdType="ImmutableId" set in the HTTP client's default headers, plus a specific test that moves a message and asserts the id is unchanged.
• Untrusted-file parsing runs in our own process in Phase 1. _decisions.md line 271 already names this; §4.2 step 5 and §6.5 are where it bites. The Phase 1 mitigations (timeouts, memory caps, restricted user, structural gate before any parser, no outbound network from the parse step) are weaker than a sandbox. This is the split trigger most likely to fire, and the honest recommendation is to plan the Phase 2 untrusted queue as scheduled work rather than as a contingency.
• Parsing accuracy versus BRD §11's "no re-keying". Step 12 deliberately leaves low-confidence fields empty. This is correct and it means recruiters will type into some fields. The business must accept a review step; measure and publish per-field fill rates from week one so the conversation is about numbers rather than impressions.
• A CAPTCHA verifier outage is a real availability risk on a public endpoint. §3.5 fails open into a stricter mode by design. The tradeoff — a noisier review queue rather than blocked applicants — should be confirmed by the business, because the alternative failure mode is a careers page that silently rejects every application while the ATS looks healthy.
• Duplicate rejection emails are the worst outbound failure this system can produce. The guard is a single sent_at IS NULL check inside the send job's transaction plus the idempotency key. It deserves a dedicated test that runs the same job twice concurrently and asserts one send.
• The reconciliation sweep can mask a broken webhook path indefinitely. If nobody watches the missed-message counter, mail keeps arriving via the hourly sweep and latency quietly degrades from seconds to an hour. Mitigation: a non-zero count is an alert, not a metric, and it is in the top-five list in §8.1.
• Storage/database restore skew. A blob backup and a database PITR restored to different points produce candidate_document rows with dangling keys, or orphans the sweep will eventually delete. Mitigation: the restore runbook restores both to the same timestamp, the orphan sweep's 30-day grace covers the gap, and the runbook is rehearsed once per phase.
• AI cost has no natural ceiling. Steps 9, 11, 13 and 17 call a paid provider per document and per rescore, and score.rescore_batch fans out across every application on a requisition version. A config change on a 400-application requisition is 400 model calls. Mitigation: input_fingerprint short-circuits unchanged rescores (_decisions.md), per-capability daily cost budgets enforced in ai_orchestration.invoke() with a hard stop, and a cost line on the AI Studio surface so it is visible before it is a finance conversation.