258 lines
24 KiB
Markdown
258 lines
24 KiB
Markdown
# ADR 0005 — Inbound email integration method for the careers mailbox
|
|
|
|
**Status:** Proposed — 2026-07-29. Assumption-dependent: the *design* is settled, the *premise* is not.
|
|
Ratification gate: written confirmation from Utopia Brands IT of assumptions A1, A2 and A3 below.
|
|
Until that confirmation lands this ADR is Proposed; on confirmation it becomes Accepted with no
|
|
design change, and if A1 or A3 is refused, Option B (IMAP/SMTP) is promoted without reopening
|
|
the surrounding intake design.
|
|
|
|
**Deciders:** Talha Ahmed (owner), Ahmed Mujtaba (implements the reconciliation sweep and the
|
|
dead-letter surface).
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
Inbound email is Phase 1 intake channel #1 (BRD §8.1, which names Outlook). It is also one of the
|
|
two attacker-supplied data sources in the system, the other being CV files. Everything below is
|
|
constrained by what the repository actually contains.
|
|
|
|
| Fact | Evidence | Consequence for this decision |
|
|
|---|---|---|
|
|
| The repository makes **zero network calls** of any kind — no `fetch`, `XMLHttpRequest`, `axios`, `WebSocket` or `EventSource` anywhere in `js/` or `index.html` | `_repo-findings.md` §C | There is no incumbent mail client, no HTTP layer and no retry/backoff code to preserve. This is a greenfield integration. |
|
|
| There is **no** `.env`, `.env.example`, mail SDK, OAuth config or provider name anywhere | `_repo-findings.md` §B | The provider is genuinely unknown from the repository. It must be assumed and labelled, never inferred from `.gitignore` (§H warns explicitly against reading `.gitignore` as stack intent). |
|
|
| The prototype's inbox is a **pre-resolved** array: each row already carries name, email, `jobId`, `atsScore` and recruiter | `js/data.js:284-300` | There is no representable state for "a message arrived and cannot become a candidate". That state is mandatory, so the integration must land raw and resolve later. |
|
|
| No backend, no queue, no database | `_repo-findings.md` §B | Durability, idempotency and retry semantics all have to be designed here rather than adapted. |
|
|
| Two developers, one senior | `_repo-findings.md` §I | The integration must be operable by one person on a bad day. Three overlapping sync mechanisms are acceptable; three provider integrations are not. |
|
|
|
|
Binding decisions this ADR must not contradict, all from `_decisions.md`:
|
|
|
|
- `raw_intake` is an **immutable landing row** and is created before any candidate work.
|
|
`candidate.created_from_raw_intake_id` and `job_application.raw_intake_id` are both `NOT NULL`
|
|
against non-deferrable foreign keys, so no candidate can physically exist without a prior intake row.
|
|
- Idempotency is a **database fact**: `UNIQUE (channel_id, external_message_id)` and
|
|
`UNIQUE (channel_id, payload_sha256)` on `raw_intake`.
|
|
- The queue is **Postgres-backed** (`procrastinate`), so an intake insert and its follow-on job
|
|
enqueue commit or roll back together. The dual-write bug class does not exist here.
|
|
- Attachments live in object storage keyed by `sha256`, never as `bytea` in-row.
|
|
- `raw_intake.state` includes the terminal, candidate-less states `rejected_unusable` and `quarantined`.
|
|
|
|
Load-bearing assumptions, restated from `04-integrations-and-processing.md` §1.2 so this ADR can be
|
|
attacked on its own terms. **Each is an assumption, not a finding.**
|
|
|
|
| ID | Assumption | If wrong |
|
|
|---|---|---|
|
|
| A1 | Utopia Brands runs Microsoft 365 and the careers mailbox is an Exchange Online mailbox in the corporate tenant | Option B (IMAP/SMTP), or a Gmail API adapter behind the same port at roughly 4-6 developer-days |
|
|
| A2 | A dedicated **shared mailbox** (`careers@utopiabrands.com`) exists or can be created, distinct from any individual's mailbox | A distribution list forwarding into a shared mailbox is acceptable; a list with **no** mailbox is a blocker, because there is nothing to read |
|
|
| A3 | IT will grant an Entra ID app registration with **application** Graph mail permissions, admin-consented and narrowed by an Exchange `ApplicationAccessPolicy` | Delegated-only access forces a service-account password and a refresh-token expiry problem — strictly worse security. Option B becomes the decision |
|
|
| A4 | Volume of 200-600 documents/day at peak, 20k-60k applications/year | Bulk job-board feeds would raise this by 1-2 orders of magnitude and both queue and storage sizing re-derive |
|
|
|
|
---
|
|
|
|
## Options considered
|
|
|
|
| # | Option | Pros (stated at their strongest) | Cons |
|
|
|---|---|---|---|
|
|
| **A** | **Microsoft Graph v1.0**, Entra single-tenant app registration, client-credentials flow with a certificate or workload-identity federation, narrowed to one mailbox by `New-ApplicationAccessPolicy`. Delta query as the authoritative cursor, change notifications as latency hints, plus a reconciliation sweep | Least privilege is **actually achievable** — Graph application permissions are tenant-wide by default, which is worse than IMAP, but the Exchange access policy narrows them back to a mail-enabled group containing exactly one mailbox. Nothing else in this space is genuinely least-privilege. Server-maintained opaque cursor (`@odata.deltaLink`) removes client cursor state. Push subscriptions remove the polling-latency tradeoff. Immutable message ids survive folder moves. One tenant for identity, mail and hosting under A1/A7. Maintained Python SDK; mail is I/O-light at A4 volumes | Hard dependency on a corporate IT grant we do not control (A3) — a Phase 1 critical-path risk. Subscription lifetime is short (days, not weeks) so renewal is standing operational work. Requires an **unauthenticated** public webhook endpoint. `Mail.ReadWrite` mutates a mailbox recruiters can see. Throttling limits change and cannot be designed to a fixed number. Provider lock-in for the adapter, though not for the intake model |
|
|
| **B** | **IMAP/SMTP** with `XOAUTH2` where supported, `IDLE` for push, `UIDVALIDITY`/`UIDNEXT` (plus `CONDSTORE`/`HIGHESTMODSEQ`) as the cursor, MIME parsed in-process | Genuinely provider-agnostic — works for M365, Google Workspace and anything else, so it de-risks A1 entirely. No app registration, no admin consent, no tenant dependency, therefore no external blocker on the Phase 1 critical path. No public webhook endpoint needed. Well-understood protocol with mature Python libraries | The credential is a **whole-mailbox** credential; there is no way to express "this app may touch only this mailbox". Basic auth is a full-mailbox password and would have to be escalated as an accepted risk. `Message-ID` is client-generated, occasionally missing and occasionally duplicated, so the stable-id guarantee is weaker. A `UIDVALIDITY` change invalidates every stored UID and forces a bounded re-sync. Long-lived `IDLE` connections need reconnect-with-backoff supervision. `EXPUNGE` on servers without `MOVE` is the only destructive operation in the whole integration. Threading degrades to `In-Reply-To`/`References` plus heuristics. Budget 5-8 developer-days for the adapter and its own reconciliation tests |
|
|
| **C** | **Third-party inbound-mail SaaS** (Mailgun Routes, SendGrid Inbound Parse, Postmark inbound) posting parsed messages to a webhook | Least code by a wide margin: no polling, no cursor, no subscription renewal, no MIME parsing, retries and signed webhooks provided. Fastest route to a working channel — days, not weeks | Every candidate CV and every piece of candidate PII transits a processor outside controlled infrastructure, which BRD §7.4 forbids and which would need its own DPA and a jurisdiction review across six markets. The careers address stops being a real mailbox a recruiter can open, so the human fallback disappears. Inbound-only: replies still need a separate send path. Rejected on data-protection grounds, not on engineering quality |
|
|
| **D** | **Exchange transport rule forwarding** to an ATS-owned inbound SMTP receiver we operate | No Graph permission, no app registration, tenant-agnostic, push by nature with no polling and no cursor at all. Conceptually the simplest possible ingest | We would run and harden an internet-facing MTA — spam, relay abuse, TLS, size limits — which two developers should not take on. Forwarding breaks SPF/DKIM alignment, so authenticity signals on inbound mail are degraded exactly where we care about attacker-supplied content. No read-back: the mailbox cannot be re-listed, so **there is no reconciliation source of truth** and a message dropped in transit is gone with no way to detect it. That single point is disqualifying |
|
|
| **E** | **No integration in Phase 1** — recruiters forward or drag-drop CVs into an upload screen | Zero integration risk, zero external dependency, unblocks the rest of Phase 1 immediately. Honest about what a two-person team can ship | Fails BRD §8.1's primary channel. Loses the envelope, headers and provider id, so `raw_intake.payload` becomes a partial record and `UNIQUE (channel_id, external_message_id)` has nothing to key on. Recruiters become the idempotency mechanism, which means duplicates. Retained not as the decision but as the **degradation mode**: manual upload on the `manual_ui` intake channel exists anyway and is what the business falls back to during an outage |
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
**Option A. Microsoft Graph v1.0 as the Phase 1 mail provider, behind a `MailProvider` port with
|
|
Option B pre-designed as a second adapter.** Concretely:
|
|
|
|
1. **Access.** Single-tenant Entra app registration, one per environment, credentialed by
|
|
certificate or workload-identity federation held in Key Vault and read by the container's
|
|
managed identity. Never a client secret in a file. Permissions: `Mail.Read`, `Mail.ReadWrite`,
|
|
`Mail.Send` — application, admin-consented. Explicitly not requested: `Mail.Read.All` beyond the
|
|
policy scope, `User.Read.All`, `Directory.Read.All`, `Files.*`, `Sites.*`, `Calendars.*`.
|
|
2. **Mailbox scoping is a go-live gate, not a configuration detail.**
|
|
`New-ApplicationAccessPolicy -AccessRight RestrictAccess` against a mail-enabled group containing
|
|
only the careers mailbox, **verified with `Test-ApplicationAccessPolicy` against both an
|
|
in-scope and an out-of-scope mailbox**, with the output recorded. Without this, `Mail.Read`
|
|
application permission reads every mailbox in the tenant. This is the single line that separates
|
|
a defensible integration from an indefensible one.
|
|
3. **Three overlapping mechanisms**, because the requirement is that no message is ever silently lost:
|
|
|
|
| Mechanism | Cadence | Role |
|
|
|---|---|---|
|
|
| Change notification (webhook) | push, seconds | latency only |
|
|
| Delta query | on every hint, plus a 5-minute tick | the authoritative fetch; the **only** thing that advances the cursor |
|
|
| Reconciliation sweep | shallow hourly over 48h, deep nightly over 30 days | the actual safety net |
|
|
|
|
4. **The webhook is a doorbell, not a delivery.** Notification bodies are never trusted as data.
|
|
The handler validates `clientState` in constant time, records a `webhook_receipt` row, enqueues a
|
|
debounced delta run, and returns `202` in under a second. `includeResourceData` is not used.
|
|
5. **`Prefer: IdType="ImmutableId"` on every mail request**, set in the HTTP client's default
|
|
headers so it cannot be forgotten. The default Graph message id changes on folder move, and we
|
|
move processed messages, so the mutable id would make every processed message look new on the
|
|
next sweep. This is the most likely implementation mistake in the whole integration.
|
|
6. **Cursor discipline.** `ChannelConnection.delta_cursor` holds the opaque `deltaLink`; it is never
|
|
parsed or reconstructed. The new `deltaLink` is persisted **in the same transaction as the final
|
|
page's rows**. On backfill, the cursor is established *before* the subscription is created.
|
|
7. **Nothing is parsed in the request path.** Attachments download as separate per-message jobs on
|
|
the `mail` queue, so each is independently retryable (ADR 0004), and CV parsing is a worker concern
|
|
(ADR 0001). A `410 resyncRequired` triggers a bounded re-sync, not a full re-ingest.
|
|
8. **Design to throttling behaviour, not to a number.** Honour `Retry-After` on 429 and 503
|
|
absolutely, cap per-mailbox concurrency at 2, hold a single-flight lock per `ChannelConnection`
|
|
so two workers never sync one mailbox, and treat sustained throttling as a monitored condition.
|
|
Graph service limits change and must be re-verified against current Microsoft documentation at
|
|
implementation time.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant G as Microsoft Graph
|
|
participant W as web webhook endpoint
|
|
participant Q as procrastinate queue
|
|
participant K as worker mail task
|
|
participant D as PostgreSQL
|
|
G->>W: notification, doorbell only
|
|
W->>D: insert webhook_receipt
|
|
W->>Q: enqueue debounced delta run
|
|
W-->>G: 202 under one second
|
|
Q->>K: delta run
|
|
K->>G: GET stored deltaLink
|
|
G-->>K: message pages
|
|
K->>D: insert raw_intake rows and enqueue attachment jobs
|
|
Note over K,D: new deltaLink persists in the same transaction as the last page
|
|
K->>Q: enqueue parse jobs
|
|
```
|
|
|
|
---
|
|
|
|
## Justification
|
|
|
|
The deciding factor is **least privilege, not latency or code volume**. Latency is solved on every
|
|
option — Graph notifications, IMAP `IDLE`, SaaS webhooks and SMTP forwarding all deliver in seconds,
|
|
and at A4 volumes a plain 5-minute poll would also be acceptable to the business. What differs
|
|
irreducibly is the shape of the credential. Option B's credential is the mailbox itself; Option D
|
|
gives up the ability to re-read the mailbox at all; Option C moves candidate PII outside controlled
|
|
infrastructure. Option A is the only configuration where the blast radius of a leaked credential is
|
|
one mailbox that contains only what candidates sent us voluntarily.
|
|
|
|
The second factor is that **Graph maintains state we would otherwise own**. An opaque `deltaLink` is
|
|
a cursor we cannot corrupt, mis-parse or reconstruct wrongly. Option B requires us to hold
|
|
`UIDVALIDITY`, `UIDNEXT` and `HIGHESTMODSEQ` per folder and to re-derive correctness after every
|
|
reconnect — a correctness burden landing on a two-person team.
|
|
|
|
**Why three mechanisms instead of one.** Push-only loses mail whenever a subscription expires,
|
|
a notification is dropped or the endpoint is briefly down; poll-only is either slow or wasteful. The
|
|
combination has no single point of loss, and the marginal cost is small because the reconciliation
|
|
sweep is a left-anti-join we want regardless as a data-quality probe. The nightly report's
|
|
missed-message count is an **alert, not a statistic**: if reconciliation routinely finds messages,
|
|
the fast path is broken and the sweep is masking it.
|
|
|
|
**Why the port abstraction is cheap here.** Option B's two hard edges — id stability and cursor
|
|
invalidation — are already handled by mechanisms the Graph path uses anyway: `UNIQUE (channel_id,
|
|
payload_sha256)` and a bounded re-sync driven by the reconciliation sweep. Building those as
|
|
first-class layers rather than Graph-specific workarounds is what makes the fallback a 5-8 day
|
|
adapter instead of a re-architecture. That is the payoff for taking dedupe layer 2 seriously.
|
|
|
|
**Tradeoff accepted explicitly.** Option A puts an external organisation on the Phase 1 critical
|
|
path. We buy that down by (a) filing the app registration request in **Phase 0**, before any code
|
|
depends on it, (b) building against a developer tenant test mailbox in the interim, and (c) keeping
|
|
Option B designed rather than merely mentioned. If A3 is refused, we lose 5-8 days, not a phase.
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
**Positive**
|
|
|
|
- A leaked mail credential exposes exactly one mailbox, and the exposure is provable via
|
|
`Test-ApplicationAccessPolicy` output recorded at go-live.
|
|
- Message loss is detectable rather than theoretical: the hourly and nightly sweeps produce a
|
|
countable anomaly figure per class, so "did we lose mail" is a query.
|
|
- Redelivery, delta re-read, sweep overlap and backfill re-run are all absorbed by
|
|
`ON CONFLICT (channel_id, external_message_id) DO NOTHING` — no application-side "have I seen
|
|
this" logic, therefore no place for that logic to be wrong.
|
|
- Because enqueue is transactional with the intake insert, a crash mid-backfill cannot leave an
|
|
intake row with no job or a job with no row.
|
|
- The mailbox stays a real mailbox. A recruiter can open Outlook and see what the system saw, which
|
|
is the cheapest possible debugging and business-continuity story.
|
|
- Terminal `rejected_unusable` and `quarantined` states mean an unparseable message becomes visible
|
|
work in a queue rather than a silent drop — the exact state the prototype cannot represent.
|
|
|
|
**Negative — costs we are accepting**
|
|
|
|
- **An external dependency we do not control.** If IT declines or delays the app registration, this
|
|
channel does not exist. Mitigation is scheduling, not architecture.
|
|
- **Standing operational work.** Subscription renewal every 15 minutes against a short expiry,
|
|
lifecycle notification handling (`reauthorizationRequired`, `subscriptionRemoved`, `missed`), and
|
|
`clientState` rotation on every recreate. This is code that exists purely to keep a subscription
|
|
alive and delivers no user-visible feature.
|
|
- **A public unauthenticated endpoint.** Graph cannot present our credentials, so
|
|
`POST /api/v1/webhooks/graph/mail` is unauthenticated by design and hardened instead
|
|
(constant-time `clientState` comparison, active-subscription check, body cap, per-source rate
|
|
limit, request id in every log line). This is the only such endpoint in the system and it is a
|
|
permanent review obligation.
|
|
- **Mailbox mutation.** `Mail.ReadWrite` moves processed messages and sets categories, so the
|
|
system changes what recruiters see. If the business objects, we drop the scope and rely on a
|
|
database-side cursor only, and lose processing visibility in Outlook.
|
|
- **Three mechanisms is more code than one.** Backfill, delta, notifications, renewal, lifecycle,
|
|
shallow sweep, deep sweep. Estimate 12-18 developer-days for the Graph path including tests, on
|
|
top of the intake schema. A naive poller would be 3-4.
|
|
- **Provider lock-in at the adapter layer**, deliberately confined there. The intake schema,
|
|
dedupe layers and reconciliation shape are provider-neutral.
|
|
- **Preserved-original strings reach the renderer.** `raw_intake.payload` and
|
|
`intake_parse_attempt.parsed` are stored unsanitised by design so the original is recoverable.
|
|
Combined with the 34 unescaped `innerHTML` sites (`_repo-findings.md` §E), the first real mailbox
|
|
connection turns every screen into a stored-XSS sink. **The Phase 0 escaping patch is a hard
|
|
prerequisite for connecting a real mailbox**, not a parallel task.
|
|
|
|
---
|
|
|
|
## Risks
|
|
|
|
| # | Risk | Severity | Mitigation |
|
|
|---|---|---|---|
|
|
| R1 | A3 refused or delayed; no application permissions granted | High | Request filed in Phase 0; developer-tenant mailbox for build; Option B designed, 5-8 days |
|
|
| R2 | A2 false — the careers address is a distribution list with no mailbox | High | Confirm with IT before any code; a list forwarding into a shared mailbox is acceptable, a list alone is a blocker |
|
|
| R3 | Mutable message id used by mistake, so every processed message re-ingests forever | High | `Prefer: IdType="ImmutableId"` in default client headers; a test asserting the header is present on every mail request; dedupe layer 2 (`payload_sha256`) catches the consequence even if the header is missed |
|
|
| R4 | Mailbox scoping misconfigured, giving tenant-wide mail read | Critical | Two-sided `Test-ApplicationAccessPolicy` verification as a recorded go-live check, repeated after any tenant change |
|
|
| R5 | Graph throttling or service-limit changes invalidate hardcoded assumptions | Medium | No hardcoded limits; `Retry-After` honoured absolutely; concurrency capped at 2; sustained throttling monitored |
|
|
| R6 | Silent subscription death — renewal fails and nobody notices | Medium | Renew at half-life on a 15-minute check; delete-and-recreate after 3 failures; `missed` lifecycle notification forces a delta run; the hourly sweep bounds worst-case latency to about one hour |
|
|
| R7 | Webhook endpoint abused as an unauthenticated enqueue amplifier | Medium | Debounce coalesces to at most one queued delta run per connection, so request volume does not translate into job volume; rate limit; body cap |
|
|
| R8 | Attacker-supplied content — zip bombs, deep MIME nesting, malicious PDFs | High | Attachments to object storage with `virus_scan_status` as a separate stage; parsing in the Phase 2 restricted worker queue with no outbound network and hard CPU/wall/memory caps; part-count and nesting-depth limits (relevant mainly to Option B, where no server-side guard exists) |
|
|
| R9 | Backfill floods the review queue and creates retention liability on day one | Medium | Operator-supplied `backfill_from`, default 90 days, surfaced as a setup choice not a constant; backfill runs at concurrency 1 and yields to steady-state sync |
|
|
| R10 | Real mailbox connected before the Phase 0 escaping patch ships | High | Sequencing is enforced, not advised: the mailbox credential is not provisioned until the CSP header and the CI escaping gate are both merged |
|
|
|
|
---
|
|
|
|
## Revisit conditions
|
|
|
|
Any one of these reopens this ADR. All are measurable from data the system already records.
|
|
|
|
| Trigger | Threshold | Reopens |
|
|
|---|---|---|
|
|
| Assumption A1 falsified | Careers mail is not Exchange Online | Provider choice; Gmail adapter or Option B |
|
|
| Assumption A3 refused | No admin consent for application permissions within 30 days of the Phase 0 request | Promote Option B |
|
|
| Assumption A2 falsified | No mailbox object exists behind the careers address | Blocker; escalate to the business before any build |
|
|
| Reconciliation is doing the real work | Nightly missed-message count non-zero on 2 consecutive runs, or shallow-sweep ingests exceeding 1% of daily message volume for 7 days | The fast path (webhook plus delta) is broken; fix before adding features |
|
|
| Subscription instability | More than 3 subscription recreations in any 7-day window | Renewal design, or drop to poll-plus-sweep only and accept minutes of latency |
|
|
| Cursor instability | More than one `410 resyncRequired` per week | Delta usage pattern and re-sync bounds |
|
|
| Throttling | 429 responses exceeding 2% of Graph calls over 24 hours after concurrency is already at 1 | Batching strategy, and whether per-mailbox serialisation is sufficient |
|
|
| Volume beyond A4 | Sustained above 2,000 documents/day, or above 5,000 inbound messages/day | Queue sizing, worker count, and whether the 5-minute delta tick is still appropriate |
|
|
| Latency regression | p95 mail-arrival to `raw_intake` row exceeding 5 minutes for a week, or p95 arrival to `parsed` exceeding 30 minutes | Worker sizing before provider choice |
|
|
| Channel count grows | A third or fourth mailbox is added, for example per-region careers addresses | Per-connection single-flight locking and concurrency caps; the model is per-connection already, so this is sizing, not redesign |
|
|
| A second provider becomes permanent | Both Graph and IMAP adapters run in production for more than one quarter | Whether the `MailProvider` port is the right abstraction or whether the two paths should diverge |
|
|
|
|
---
|
|
|
|
## Related
|
|
|
|
- `_decisions.md` — raw intake to candidate resolution model; invariants that stop a malformed email
|
|
creating a candidate; queue technology; process split.
|
|
- `04-integrations-and-processing.md` §2 — the full implementation design this ADR ratifies
|
|
(scopes, backfill, delta, subscription lifecycle, threading, attachments, reply sending, §2.12 fallback).
|
|
- ADR 0001 — the two-process split that keeps parsing off the request path.
|
|
- ADR 0003 — object storage for attachments, keyed by `sha256`.
|
|
- ADR 0004 — the Postgres-backed queue whose transactional enqueue makes intake loss-free.
|
|
- ADR 0008 — duplicate resolution. Message-level dedupe (layers 1-3) and identity-level dedupe
|
|
(layer 4) are deliberately different mechanisms; this ADR owns only the former.
|