24 KiB

Raw Permalink Blame History

ADR 0005 — Inbound email integration method for the careers mailbox

Status: Proposed — 2026-07-29. Assumption-dependent: the design is settled, the premise is not. Ratification gate: written confirmation from Utopia Brands IT of assumptions A1, A2 and A3 below. Until that confirmation lands this ADR is Proposed; on confirmation it becomes Accepted with no design change, and if A1 or A3 is refused, Option B (IMAP/SMTP) is promoted without reopening the surrounding intake design.

Deciders: Talha Ahmed (owner), Ahmed Mujtaba (implements the reconciliation sweep and the dead-letter surface).

Context

Inbound email is Phase 1 intake channel #1 (BRD §8.1, which names Outlook). It is also one of the two attacker-supplied data sources in the system, the other being CV files. Everything below is constrained by what the repository actually contains.

Fact	Evidence	Consequence for this decision
The repository makes zero network calls of any kind — no `fetch`, `XMLHttpRequest`, `axios`, `WebSocket` or `EventSource` anywhere in `js/` or `index.html`	`_repo-findings.md` §C	There is no incumbent mail client, no HTTP layer and no retry/backoff code to preserve. This is a greenfield integration.
There is no `.env`, `.env.example`, mail SDK, OAuth config or provider name anywhere	`_repo-findings.md` §B	The provider is genuinely unknown from the repository. It must be assumed and labelled, never inferred from `.gitignore` (§H warns explicitly against reading `.gitignore` as stack intent).
The prototype's inbox is a pre-resolved array: each row already carries name, email, `jobId`, `atsScore` and recruiter	`js/data.js:284-300`	There is no representable state for "a message arrived and cannot become a candidate". That state is mandatory, so the integration must land raw and resolve later.
No backend, no queue, no database	`_repo-findings.md` §B	Durability, idempotency and retry semantics all have to be designed here rather than adapted.
Two developers, one senior	`_repo-findings.md` §I	The integration must be operable by one person on a bad day. Three overlapping sync mechanisms are acceptable; three provider integrations are not.

Binding decisions this ADR must not contradict, all from _decisions.md:

raw_intake is an immutable landing row and is created before any candidate work. candidate.created_from_raw_intake_id and job_application.raw_intake_id are both NOT NULL against non-deferrable foreign keys, so no candidate can physically exist without a prior intake row.
Idempotency is a database fact: UNIQUE (channel_id, external_message_id) and UNIQUE (channel_id, payload_sha256) on raw_intake.
The queue is Postgres-backed (procrastinate), so an intake insert and its follow-on job enqueue commit or roll back together. The dual-write bug class does not exist here.
Attachments live in object storage keyed by sha256, never as bytea in-row.
raw_intake.state includes the terminal, candidate-less states rejected_unusable and quarantined.

Load-bearing assumptions, restated from 04-integrations-and-processing.md §1.2 so this ADR can be attacked on its own terms. Each is an assumption, not a finding.

ID	Assumption	If wrong
A1	Utopia Brands runs Microsoft 365 and the careers mailbox is an Exchange Online mailbox in the corporate tenant	Option B (IMAP/SMTP), or a Gmail API adapter behind the same port at roughly 4-6 developer-days
A2	A dedicated shared mailbox (`careers@utopiabrands.com`) exists or can be created, distinct from any individual's mailbox	A distribution list forwarding into a shared mailbox is acceptable; a list with no mailbox is a blocker, because there is nothing to read
A3	IT will grant an Entra ID app registration with application Graph mail permissions, admin-consented and narrowed by an Exchange `ApplicationAccessPolicy`	Delegated-only access forces a service-account password and a refresh-token expiry problem — strictly worse security. Option B becomes the decision
A4	Volume of 200-600 documents/day at peak, 20k-60k applications/year	Bulk job-board feeds would raise this by 1-2 orders of magnitude and both queue and storage sizing re-derive

Options considered

#	Option	Pros (stated at their strongest)	Cons
A	Microsoft Graph v1.0, Entra single-tenant app registration, client-credentials flow with a certificate or workload-identity federation, narrowed to one mailbox by `New-ApplicationAccessPolicy`. Delta query as the authoritative cursor, change notifications as latency hints, plus a reconciliation sweep	Least privilege is actually achievable — Graph application permissions are tenant-wide by default, which is worse than IMAP, but the Exchange access policy narrows them back to a mail-enabled group containing exactly one mailbox. Nothing else in this space is genuinely least-privilege. Server-maintained opaque cursor (`@odata.deltaLink`) removes client cursor state. Push subscriptions remove the polling-latency tradeoff. Immutable message ids survive folder moves. One tenant for identity, mail and hosting under A1/A7. Maintained Python SDK; mail is I/O-light at A4 volumes	Hard dependency on a corporate IT grant we do not control (A3) — a Phase 1 critical-path risk. Subscription lifetime is short (days, not weeks) so renewal is standing operational work. Requires an unauthenticated public webhook endpoint. `Mail.ReadWrite` mutates a mailbox recruiters can see. Throttling limits change and cannot be designed to a fixed number. Provider lock-in for the adapter, though not for the intake model
B	IMAP/SMTP with `XOAUTH2` where supported, `IDLE` for push, `UIDVALIDITY`/`UIDNEXT` (plus `CONDSTORE`/`HIGHESTMODSEQ`) as the cursor, MIME parsed in-process	Genuinely provider-agnostic — works for M365, Google Workspace and anything else, so it de-risks A1 entirely. No app registration, no admin consent, no tenant dependency, therefore no external blocker on the Phase 1 critical path. No public webhook endpoint needed. Well-understood protocol with mature Python libraries	The credential is a whole-mailbox credential; there is no way to express "this app may touch only this mailbox". Basic auth is a full-mailbox password and would have to be escalated as an accepted risk. `Message-ID` is client-generated, occasionally missing and occasionally duplicated, so the stable-id guarantee is weaker. A `UIDVALIDITY` change invalidates every stored UID and forces a bounded re-sync. Long-lived `IDLE` connections need reconnect-with-backoff supervision. `EXPUNGE` on servers without `MOVE` is the only destructive operation in the whole integration. Threading degrades to `In-Reply-To`/`References` plus heuristics. Budget 5-8 developer-days for the adapter and its own reconciliation tests
C	Third-party inbound-mail SaaS (Mailgun Routes, SendGrid Inbound Parse, Postmark inbound) posting parsed messages to a webhook	Least code by a wide margin: no polling, no cursor, no subscription renewal, no MIME parsing, retries and signed webhooks provided. Fastest route to a working channel — days, not weeks	Every candidate CV and every piece of candidate PII transits a processor outside controlled infrastructure, which BRD §7.4 forbids and which would need its own DPA and a jurisdiction review across six markets. The careers address stops being a real mailbox a recruiter can open, so the human fallback disappears. Inbound-only: replies still need a separate send path. Rejected on data-protection grounds, not on engineering quality
D	Exchange transport rule forwarding to an ATS-owned inbound SMTP receiver we operate	No Graph permission, no app registration, tenant-agnostic, push by nature with no polling and no cursor at all. Conceptually the simplest possible ingest	We would run and harden an internet-facing MTA — spam, relay abuse, TLS, size limits — which two developers should not take on. Forwarding breaks SPF/DKIM alignment, so authenticity signals on inbound mail are degraded exactly where we care about attacker-supplied content. No read-back: the mailbox cannot be re-listed, so there is no reconciliation source of truth and a message dropped in transit is gone with no way to detect it. That single point is disqualifying
E	No integration in Phase 1 — recruiters forward or drag-drop CVs into an upload screen	Zero integration risk, zero external dependency, unblocks the rest of Phase 1 immediately. Honest about what a two-person team can ship	Fails BRD §8.1's primary channel. Loses the envelope, headers and provider id, so `raw_intake.payload` becomes a partial record and `UNIQUE (channel_id, external_message_id)` has nothing to key on. Recruiters become the idempotency mechanism, which means duplicates. Retained not as the decision but as the degradation mode: manual upload on the `manual_ui` intake channel exists anyway and is what the business falls back to during an outage

Decision

Option A. Microsoft Graph v1.0 as the Phase 1 mail provider, behind a MailProvider port with Option B pre-designed as a second adapter. Concretely:

Access. Single-tenant Entra app registration, one per environment, credentialed by certificate or workload-identity federation held in Key Vault and read by the container's managed identity. Never a client secret in a file. Permissions: Mail.Read, Mail.ReadWrite, Mail.Send — application, admin-consented. Explicitly not requested: Mail.Read.All beyond the policy scope, User.Read.All, Directory.Read.All, Files.*, Sites.*, Calendars.*.
Mailbox scoping is a go-live gate, not a configuration detail. New-ApplicationAccessPolicy -AccessRight RestrictAccess against a mail-enabled group containing only the careers mailbox, verified with Test-ApplicationAccessPolicy against both an in-scope and an out-of-scope mailbox, with the output recorded. Without this, Mail.Read application permission reads every mailbox in the tenant. This is the single line that separates a defensible integration from an indefensible one.

Three overlapping mechanisms, because the requirement is that no message is ever silently lost:

Mechanism	Cadence	Role
Change notification (webhook)	push, seconds	latency only
Delta query	on every hint, plus a 5-minute tick	the authoritative fetch; the only thing that advances the cursor
Reconciliation sweep	shallow hourly over 48h, deep nightly over 30 days	the actual safety net

The webhook is a doorbell, not a delivery. Notification bodies are never trusted as data. The handler validates clientState in constant time, records a webhook_receipt row, enqueues a debounced delta run, and returns 202 in under a second. includeResourceData is not used.
Prefer: IdType="ImmutableId" on every mail request, set in the HTTP client's default headers so it cannot be forgotten. The default Graph message id changes on folder move, and we move processed messages, so the mutable id would make every processed message look new on the next sweep. This is the most likely implementation mistake in the whole integration.
Cursor discipline. ChannelConnection.delta_cursor holds the opaque deltaLink; it is never parsed or reconstructed. The new deltaLink is persisted in the same transaction as the final page's rows. On backfill, the cursor is established before the subscription is created.
Nothing is parsed in the request path. Attachments download as separate per-message jobs on the mail queue, so each is independently retryable (ADR 0004), and CV parsing is a worker concern (ADR 0001). A 410 resyncRequired triggers a bounded re-sync, not a full re-ingest.
Design to throttling behaviour, not to a number. Honour Retry-After on 429 and 503 absolutely, cap per-mailbox concurrency at 2, hold a single-flight lock per ChannelConnection so two workers never sync one mailbox, and treat sustained throttling as a monitored condition. Graph service limits change and must be re-verified against current Microsoft documentation at implementation time.

sequenceDiagram
  participant G as Microsoft Graph
  participant W as web webhook endpoint
  participant Q as procrastinate queue
  participant K as worker mail task
  participant D as PostgreSQL
  G->>W: notification, doorbell only
  W->>D: insert webhook_receipt
  W->>Q: enqueue debounced delta run
  W-->>G: 202 under one second
  Q->>K: delta run
  K->>G: GET stored deltaLink
  G-->>K: message pages
  K->>D: insert raw_intake rows and enqueue attachment jobs
  Note over K,D: new deltaLink persists in the same transaction as the last page
  K->>Q: enqueue parse jobs

Justification

The deciding factor is least privilege, not latency or code volume. Latency is solved on every option — Graph notifications, IMAP IDLE, SaaS webhooks and SMTP forwarding all deliver in seconds, and at A4 volumes a plain 5-minute poll would also be acceptable to the business. What differs irreducibly is the shape of the credential. Option B's credential is the mailbox itself; Option D gives up the ability to re-read the mailbox at all; Option C moves candidate PII outside controlled infrastructure. Option A is the only configuration where the blast radius of a leaked credential is one mailbox that contains only what candidates sent us voluntarily.

The second factor is that Graph maintains state we would otherwise own. An opaque deltaLink is a cursor we cannot corrupt, mis-parse or reconstruct wrongly. Option B requires us to hold UIDVALIDITY, UIDNEXT and HIGHESTMODSEQ per folder and to re-derive correctness after every reconnect — a correctness burden landing on a two-person team.

Why three mechanisms instead of one. Push-only loses mail whenever a subscription expires, a notification is dropped or the endpoint is briefly down; poll-only is either slow or wasteful. The combination has no single point of loss, and the marginal cost is small because the reconciliation sweep is a left-anti-join we want regardless as a data-quality probe. The nightly report's missed-message count is an alert, not a statistic: if reconciliation routinely finds messages, the fast path is broken and the sweep is masking it.

Why the port abstraction is cheap here. Option B's two hard edges — id stability and cursor invalidation — are already handled by mechanisms the Graph path uses anyway: UNIQUE (channel_id, payload_sha256) and a bounded re-sync driven by the reconciliation sweep. Building those as first-class layers rather than Graph-specific workarounds is what makes the fallback a 5-8 day adapter instead of a re-architecture. That is the payoff for taking dedupe layer 2 seriously.

Tradeoff accepted explicitly. Option A puts an external organisation on the Phase 1 critical path. We buy that down by (a) filing the app registration request in Phase 0, before any code depends on it, (b) building against a developer tenant test mailbox in the interim, and (c) keeping Option B designed rather than merely mentioned. If A3 is refused, we lose 5-8 days, not a phase.

Consequences

Positive

A leaked mail credential exposes exactly one mailbox, and the exposure is provable via Test-ApplicationAccessPolicy output recorded at go-live.
Message loss is detectable rather than theoretical: the hourly and nightly sweeps produce a countable anomaly figure per class, so "did we lose mail" is a query.
Redelivery, delta re-read, sweep overlap and backfill re-run are all absorbed by ON CONFLICT (channel_id, external_message_id) DO NOTHING — no application-side "have I seen this" logic, therefore no place for that logic to be wrong.
Because enqueue is transactional with the intake insert, a crash mid-backfill cannot leave an intake row with no job or a job with no row.
The mailbox stays a real mailbox. A recruiter can open Outlook and see what the system saw, which is the cheapest possible debugging and business-continuity story.
Terminal rejected_unusable and quarantined states mean an unparseable message becomes visible work in a queue rather than a silent drop — the exact state the prototype cannot represent.

Negative — costs we are accepting

An external dependency we do not control. If IT declines or delays the app registration, this channel does not exist. Mitigation is scheduling, not architecture.
Standing operational work. Subscription renewal every 15 minutes against a short expiry, lifecycle notification handling (reauthorizationRequired, subscriptionRemoved, missed), and clientState rotation on every recreate. This is code that exists purely to keep a subscription alive and delivers no user-visible feature.
A public unauthenticated endpoint. Graph cannot present our credentials, so POST /api/v1/webhooks/graph/mail is unauthenticated by design and hardened instead (constant-time clientState comparison, active-subscription check, body cap, per-source rate limit, request id in every log line). This is the only such endpoint in the system and it is a permanent review obligation.
Mailbox mutation. Mail.ReadWrite moves processed messages and sets categories, so the system changes what recruiters see. If the business objects, we drop the scope and rely on a database-side cursor only, and lose processing visibility in Outlook.
Three mechanisms is more code than one. Backfill, delta, notifications, renewal, lifecycle, shallow sweep, deep sweep. Estimate 12-18 developer-days for the Graph path including tests, on top of the intake schema. A naive poller would be 3-4.
Provider lock-in at the adapter layer, deliberately confined there. The intake schema, dedupe layers and reconciliation shape are provider-neutral.
Preserved-original strings reach the renderer. raw_intake.payload and intake_parse_attempt.parsed are stored unsanitised by design so the original is recoverable. Combined with the 34 unescaped innerHTML sites (_repo-findings.md §E), the first real mailbox connection turns every screen into a stored-XSS sink. The Phase 0 escaping patch is a hard prerequisite for connecting a real mailbox, not a parallel task.

Risks

#	Risk	Severity	Mitigation
R1	A3 refused or delayed; no application permissions granted	High	Request filed in Phase 0; developer-tenant mailbox for build; Option B designed, 5-8 days
R2	A2 false — the careers address is a distribution list with no mailbox	High	Confirm with IT before any code; a list forwarding into a shared mailbox is acceptable, a list alone is a blocker
R3	Mutable message id used by mistake, so every processed message re-ingests forever	High	`Prefer: IdType="ImmutableId"` in default client headers; a test asserting the header is present on every mail request; dedupe layer 2 (`payload_sha256`) catches the consequence even if the header is missed
R4	Mailbox scoping misconfigured, giving tenant-wide mail read	Critical	Two-sided `Test-ApplicationAccessPolicy` verification as a recorded go-live check, repeated after any tenant change
R5	Graph throttling or service-limit changes invalidate hardcoded assumptions	Medium	No hardcoded limits; `Retry-After` honoured absolutely; concurrency capped at 2; sustained throttling monitored
R6	Silent subscription death — renewal fails and nobody notices	Medium	Renew at half-life on a 15-minute check; delete-and-recreate after 3 failures; `missed` lifecycle notification forces a delta run; the hourly sweep bounds worst-case latency to about one hour
R7	Webhook endpoint abused as an unauthenticated enqueue amplifier	Medium	Debounce coalesces to at most one queued delta run per connection, so request volume does not translate into job volume; rate limit; body cap
R8	Attacker-supplied content — zip bombs, deep MIME nesting, malicious PDFs	High	Attachments to object storage with `virus_scan_status` as a separate stage; parsing in the Phase 2 restricted worker queue with no outbound network and hard CPU/wall/memory caps; part-count and nesting-depth limits (relevant mainly to Option B, where no server-side guard exists)
R9	Backfill floods the review queue and creates retention liability on day one	Medium	Operator-supplied `backfill_from`, default 90 days, surfaced as a setup choice not a constant; backfill runs at concurrency 1 and yields to steady-state sync
R10	Real mailbox connected before the Phase 0 escaping patch ships	High	Sequencing is enforced, not advised: the mailbox credential is not provisioned until the CSP header and the CI escaping gate are both merged

Revisit conditions

Any one of these reopens this ADR. All are measurable from data the system already records.

Trigger	Threshold	Reopens
Assumption A1 falsified	Careers mail is not Exchange Online	Provider choice; Gmail adapter or Option B
Assumption A3 refused	No admin consent for application permissions within 30 days of the Phase 0 request	Promote Option B
Assumption A2 falsified	No mailbox object exists behind the careers address	Blocker; escalate to the business before any build
Reconciliation is doing the real work	Nightly missed-message count non-zero on 2 consecutive runs, or shallow-sweep ingests exceeding 1% of daily message volume for 7 days	The fast path (webhook plus delta) is broken; fix before adding features
Subscription instability	More than 3 subscription recreations in any 7-day window	Renewal design, or drop to poll-plus-sweep only and accept minutes of latency
Cursor instability	More than one `410 resyncRequired` per week	Delta usage pattern and re-sync bounds
Throttling	429 responses exceeding 2% of Graph calls over 24 hours after concurrency is already at 1	Batching strategy, and whether per-mailbox serialisation is sufficient
Volume beyond A4	Sustained above 2,000 documents/day, or above 5,000 inbound messages/day	Queue sizing, worker count, and whether the 5-minute delta tick is still appropriate
Latency regression	p95 mail-arrival to `raw_intake` row exceeding 5 minutes for a week, or p95 arrival to `parsed` exceeding 30 minutes	Worker sizing before provider choice
Channel count grows	A third or fourth mailbox is added, for example per-region careers addresses	Per-connection single-flight locking and concurrency caps; the model is per-connection already, so this is sizing, not redesign
A second provider becomes permanent	Both Graph and IMAP adapters run in production for more than one quarter	Whether the `MailProvider` port is the right abstraction or whether the two paths should diverge

_decisions.md — raw intake to candidate resolution model; invariants that stop a malformed email creating a candidate; queue technology; process split.
04-integrations-and-processing.md §2 — the full implementation design this ADR ratifies (scopes, backfill, delta, subscription lifecycle, threading, attachments, reply sending, §2.12 fallback).
ADR 0001 — the two-process split that keeps parsing off the request path.
ADR 0003 — object storage for attachments, keyed by sha256.
ADR 0004 — the Postgres-backed queue whose transactional enqueue makes intake loss-free.
ADR 0008 — duplicate resolution. Message-level dedupe (layers 1-3) and identity-level dedupe (layer 4) are deliberately different mechanisms; this ADR owns only the former.

24 KiB Raw Permalink Blame History