HR-ATS-Portal/docs/architecture/adr/0003-object-storage-strategy.md

ADR 0003 — Object Storage for Document Bytes, With the Database as the Only Index

Status	Accepted — 2026-07-29
Scope	Where CV and attachment bytes live; container layout and key naming; the upload → quarantine → scan → promote path; how access is authorised; encryption; lifecycle; how erasure and the immutable audit archive coexist
Owner	Talha Ahmed (senior). The files module facade and the malware-scan gate are Talha's; the intake triage UI that surfaces failed/quarantined states is Ahmed's, with a Talha review checkpoint
Consistent with	_decisions.md Part 1 → Deployment topology ("object storage for CV blobs"), Logical module list — platform tier (module 3, files); Part 2 → Raw intake… (raw_intake_attachment.object_store_key, virus_scan_status), Candidate first-class columns… (candidate_document.sha256, object_store_key), Soft delete, PII classification and retention, Audit table strategy
Related ADRs	0002 (the database deliberately does not hold bytes), 0004 (scanning, parsing and retention sweeps are queue jobs)

---

Context

There is no file handling in the repository today, of any kind. The prototype's "CV Import" screen (js/import.js, 176 lines) is a UI simulation over locally generated data; there are zero network calls anywhere in js/ or index.html (_repo-findings.md §C), no storage client, no .env, and no infrastructure definitions (§B). This is a greenfield decision.

It is also the decision with the highest security stakes in the package, because of two facts that hold simultaneously:

1. The two primary Phase 1 intake sources are CV files and inbound Outlook mail — both attacker-supplied by design (_repo-findings.md §E). We are, as a product requirement, running parsers over untrusted binaries uploaded by strangers.
2. The existing rendering layer has no HTML escaping anywhere — 34 innerHTML assignments across 14 files, candidate-controlled values interpolated raw (js/candidates.js:68), inline handlers with interpolated ids (js/candidates.js:121). §E is rated P0 for exactly this reason.

So the storage design has to assume every byte is hostile until proven otherwise, and it must not create a second XSS surface by serving candidate-supplied content from an origin that shares a security context with the application.

Forces

Force	Evidence / figure	Implication
Volume is small	20k–60k applications/year, 200–600 documents/day at peak (assumptions); average CV ~400 KB → roughly 10–25 GB/year of originals, well under 100 GB in year one including derivatives and raw mail payloads	Cost and scale are not the deciding factors; correctness, security and erasure are
Bytes must land before they can be scanned	raw_intake_attachment.virus_scan_status must be clean before an intake_parse_attempt may be created — enforced in the service and by a trigger	A pre-scan holding area is structurally required, not optional
Erasure must actually delete bytes	Retention purge is pseudonymisation of rows but deletion of CV blobs; retention_action records blob_keys_deleted	Any storage feature that silently retains deleted bytes (versioning, long soft-delete windows) is in direct conflict
History must survive erasure	ats_result, *_history, *_version, raw_intake, candidate_merge are append-only and never deleted	Metadata rows outlive the bytes they point at — a dangling-key state that must be explicit, not an error
The audit archive needs the opposite guarantee	Closed monthly audit partitions are exported daily to write-once storage (Object Lock / immutable blob) with the partition's final row_hash	Two containers with contradictory retention policies must coexist
Access must be derived, never public	files module: "access derived from the owning domain object, never public URLs"	No permanent URLs, no anonymous containers, no guessable keys
Two developers, no ops staff	_repo-findings.md §I	A managed service with lifecycle rules beats anything self-operated

---

Options considered

Option A — Bytes in PostgreSQL (bytea or large objects)

Pros — real, and often underrated

• One store, one transaction, one backup. A CV insert and its metadata commit atomically. There is no dual-write, no orphaned blob, no orphaned row — which is genuinely attractive given how much of this design is about the raw-intake-before-candidate invariant.
• Erasure is a DELETE/UPDATE in the same transaction as pseudonymisation. No second system to reconcile.
• PITR covers documents too, so recovery is one story.
• Access control is the database's, which is already the single authorization chokepoint.
• At our volume (tens of GB/year) it is not even absurd on size grounds.

Cons

• Every byte goes through WAL, is captured in every base backup, and inflates backup/restore time and cost for data that never changes after write. A 25 GB/year blob load makes the database an order of magnitude larger than its relational content, and the database is the one component we cannot scale horizontally (ADR 0002).
• Streaming large objects through the web process ties up a request thread and a database connection for the duration — and connections are the scarce resource in this design (ADR 0002 R2, ADR 0004).
• bytea reads are all-or-nothing without large-object APIs; range requests and resumable downloads are awkward.
• No lifecycle tiering, no server-side immutability/Object Lock, so the audit archive requirement cannot be met by the same mechanism.
• Blob churn creates bloat and vacuum pressure on the same instance serving recruiter queries.

Verdict: the transactional-consistency argument is the strongest case against object storage, and it is a genuine loss. It is not enough: it would put a permanent, growing, write-once load on the single most scaling-constrained component we have, and it cannot satisfy the write-once audit archive.

Option B — Filesystem volume on the application host (or an NFS/SMB share)

Pros

• Simplest possible programming model — open() and write(), nothing to learn, trivial local development.
• No per-request signing, no SDK, no additional credentials.
• Very cheap.

Cons

• Breaks the deployment model directly: two revisions from one image on a managed container platform (ADR 0001) means ephemeral, replaceable instances. Persistent state on the app host makes revisions non-fungible and rolling deploys unsafe.
• An NFS/SMB share re-introduces a stateful component that two developers with no ops staff would have to back up, monitor, capacity-plan and restore — the exact cost we are avoiding.
• No server-side immutability, no lifecycle rules, no per-object access tokens, no independent encryption boundary.
• Untrusted files sitting on the same filesystem as application code is a materially worse security posture than a separate account with no execute semantics.

Option C — Managed object storage (Azure Blob Storage), database holds metadata and keys only

Pros

• Purpose-built for write-once, read-occasionally binaries; effectively unbounded, and cheap at our volume.
• Server-side features map one-to-one onto requirements we actually have: private containers, short-lived scoped signed URLs, lifecycle tiering, and immutability/Object Lock for the audit archive — which no other option provides.
• A separate credential and blast-radius boundary from the database. A SQL injection does not hand over CV bytes; a leaked storage key does not hand over the relational model.
• Downloads bypass the web process entirely, so a recruiter pulling a 20 MB PDF does not consume a request thread or a database connection.
• Serving candidate-supplied bytes from a different origin than the application is a real XSS mitigation, directly relevant to §E.
• Aligns with the Azure/M365 assumption already made in ADR 0001, so it lands in the same tenant as Entra ID and Graph.

Cons (stated)

• Loses transactional consistency between bytes and rows. Two failure modes are now possible and must be designed for explicitly: an orphaned blob (bytes written, transaction rolled back) and a dangling key (row committed, bytes missing).
• A second system to configure, secure, monitor and include in disaster-recovery drills.
• Signed-URL expiry, clock skew and CORS become real, debuggable-at-3pm problems.
• Local development needs an emulator or a real dev container, adding setup friction the current repo (with no build step at all, §B) does not have.
• Blob-level versioning and soft-delete — the features you would normally enable for safety — actively undermine the erasure guarantee. This has to be resolved deliberately rather than accepting defaults.

Option D — SharePoint / OneDrive via Microsoft Graph

Pros — a serious option given the M365 assumption, not a strawman

• No new infrastructure at all: the tenant, the licences and the Graph app registration already exist for mail intake (BRD §8.1).
• Enterprise retention labels, eDiscovery, DLP and audit come for free and are already understood by whoever handles compliance internally.
• Recruiters could see documents in a familiar tool without any UI work.
• Microsoft's own antivirus scanning applies on upload.

Cons

• Graph throttling and per-site item limits are designed for human collaboration, not for a service writing hundreds of items/day with retries; throttling responses become an intake reliability problem in the one flow that must never lose a document (BRD §6.3).
• Programmatic per-item ACLs are awkward, and the natural failure mode is a document library that is broadly readable inside the tenant — unacceptable for sensitive_personal CV content, and a silent bypass of iam.can().
• "Recruiters can browse the library directly" is a liability, not a feature: it routes around the audit access events the design requires (audit_event on profile view/export).
• Retention becomes governed by tenant policy rather than by retention_policy rows, splitting the retention mechanism across two systems — and the design deliberately keeps deletion per-record.
• No usable immutability primitive for the audit hash-chain export.

Verdict: rejected for the primary document store, but note that its compliance tooling is genuinely better than ours will be. If legal later requires tenant-level eDiscovery over CVs, this decision should be reopened rather than defended.

---

Decision

Managed object storage (Azure Blob Storage — labelled an assumption, consistent with the Azure/M365 alignment in ADR 0001) is the only store for document bytes. PostgreSQL holds metadata and keys only, and the database is the sole authority for what exists. No bytes in Postgres beyond small extracted text and jsonb payloads; no bytes on the application filesystem beyond a per-request temporary file that is deleted in a finally.

1. Containers, with deliberately different policies

Container	Contents	Public access	Versioning	Soft delete	Lifecycle	Immutability
intake-quarantine	Every inbound byte on arrival, pre-scan	Off	Off	Off	Delete 30 days after promotion or rejection	No
candidate-documents	Promoted originals (CV revisions, attachments)	Off	Off — deliberately	7 days, disclosed	Hot → Cool at 180 days. Never Archive tier	No
document-derivatives	Sanitised preview renditions produced by the worker	Off	Off	Off	Delete 90 days after last access; regenerable	No
audit-archive	Daily export of closed monthly audit.audit_event partitions	Off	Off	Off	Retain ≥13 months, then policy-driven	Yes — Object Lock / immutable blob, time-based
exports	Subject-access exports, report downloads	Off	Off	Off	Delete after 7 days, no exceptions	No

Two policy decisions in that table are non-obvious and are the point of this ADR:

• candidate-documents has blob versioning OFF and only a 7-day soft-delete window. The reflex is to enable both. We do not, because retention purge must genuinely delete CV bytes, and versioning would silently retain every prior version of a document we have told a data subject we erased. The 7-day soft-delete window is retained as accident insurance and must be disclosed in the retention policy as a recovery window — not quietly relied on.
• Archive tier is never used, despite being the cheapest option for old CVs. Rehydration takes hours, and both the retention purge and a subject-access export have to complete inside statutory timeframes. Cool tier is the floor.

2. Key naming

intake-quarantine/{yyyy}/{mm}/{raw_intake_public_id}/{seq}-{sha256_prefix12}.{ext}
candidate-documents/{candidate_public_id}/{candidate_document_public_id}-{sha256_prefix12}.{ext}
document-derivatives/{candidate_document_public_id}/preview-{n}.pdf
audit-archive/{yyyy}/audit_event_{yyyy}_{mm}.jsonl.zst
exports/{requesting_user_public_id}/{export_public_id}.zip

Rules:

• Keys are built from public_id (UUIDv7) only — never from bigint PKs, never from reference_code (which is enumerable by construction and internal-only, per ADR 0002), and never from a candidate's name or email.
• The sha256 prefix in the key is for integrity and human debuggability, not addressing. Bytes are not globally deduplicated across subjects even when two candidates submit a byte-identical CV. This is a conscious trade: global content-addressed dedupe would save trivial storage at our volume while making per-subject erasure a reference-counting problem, where deleting one candidate's document could either delete another's or silently retain the erased one. Duplicate detection still uses identical candidate_document.sha256 as a matching signal — that is a row comparison and needs no shared bytes.
• The database is the authority. Prefixes are an operational convenience for browsing, not the deletion mechanism: merges re-point documents between candidates without moving bytes, so the retention purge walks candidate_document/stored_file rows and deletes the keys it finds. A prefix-based delete would miss re-pointed documents.

3. The intake path — bytes are hostile until proven otherwise

graph TD
  IN["Inbound: Outlook via Graph · career portal upload · manual recruiter upload"] --> VAL{"Pre-persist validation<br/>magic bytes · 25 MB cap · MIME allowlist"}
  VAL -->|reject| REJ["raw_intake_attachment state = rejected<br/>reason recorded · surfaced in intake triage UI"]
  VAL -->|accept| Q[["intake-quarantine<br/>bytes land here first"]]
  Q --> SCAN["worker: malware scan<br/>queue = ingest"]
  SCAN -->|infected| QUAR["virus_scan_status = infected<br/>bytes retained 30d for forensics<br/>never promoted · never parsed"]
  SCAN -->|clean| PROMO["server-side copy to candidate-documents<br/>stored_file row committed"]
  PROMO --> PARSE["worker: parse under restricted OS user<br/>no outbound network · CPU/wall timeout<br/>queue = parse"]
  PARSE --> DERIV["sanitised preview derivative<br/>document-derivatives"]
  PROMO --> DELQ["delete quarantine copy"]

Enforced rules on that path:

#	Rule
1	Validation precedes persistence. Magic-byte allowlist (PDF, DOCX, DOC, ODT, RTF, TXT); extension and client Content-Type are never trusted; SVG and HTML rejected outright; .docm rejected; 25 MB cap; decompression-ratio cap; nested archives rejected
2	virus_scan_status = 'clean' is a gate, enforced in the service and by a database trigger — no intake_parse_attempt can exist for an unscanned or infected attachment
3	Rejection is a visible state, never a silent drop. Every terminal failure surfaces in the intake triage UI (BRD §6.3)
4	Uploads are server-mediated in Phase 1 — the client posts to /api/v1/..., the web process validates, then streams to quarantine. No direct-to-blob presigned upload, because validation must run before bytes are persisted anywhere durable, and at 200–600 documents/day the web-tier cost is negligible
5	Orphan reconciliation is a scheduled job, not a hope. A nightly maintenance task lists blobs with no stored_file row older than 24 h and deletes them; and lists stored_file rows whose key is missing and flags them as bytes_missing rather than throwing on read. This is the accepted price of losing transactional consistency (Option C's main cost), and it is paid explicitly

4. Read access

Rule	Detail
Authorisation first	Every download calls iam.can(actor, 'read', document) deriving permission from the owning domain object, then files.signed_url() mints a URL. There are no permanent URLs and no anonymous containers
TTL	120 seconds, read-only, single blob, IP-agnostic. Long enough for a click, short enough that a leaked URL in a chat log or referrer is worthless
Audit	Every issuance writes an audit_event access record (actor, document, purpose). This is why direct SharePoint browsing was rejected — it would bypass this
Content headers	Content-Disposition: attachment, an allowlisted Content-Type, X-Content-Type-Options: nosniff. Originals are download-only
Inline preview	Serves the sanitised derivative (embedded JavaScript and embedded files stripped during parse), never the original. Candidate-supplied bytes are never rendered inline from the application origin
Origin separation	Blobs are served from the storage domain, not the app origin, so a hostile document cannot execute in the application's security context — a structural complement to the §E escaping work

5. Encryption, credentials, environments

• Encryption at rest: platform-managed keys in Phase 1; customer-managed keys (Key Vault) deferred to Phase 3 and only if legal requires it. TLS in transit, enforced; HTTP disabled on the account.
• Credentials: managed identity from both web and worker, scoped per container to the minimum role (worker needs write on quarantine + derivatives and delete on quarantine; web needs read + the signing right). Shared account keys are disabled. No storage credential in application configuration.
• Environments: one storage account per environment, never shared. Staging is seeded with synthetic documents only — no production CV is ever copied to staging, and the non-production anonymisation script reads the pii_classification registry rather than a hand-maintained list.
• local uses a storage emulator or a dedicated dev account; the files facade is the only code that knows which.

---

Justification

The requirements pull in two directions that only object storage satisfies simultaneously. CV bytes must be deletable on demand (erasure), while audit exports must be undeletable (tamper evidence). One store with per-container policy handles both; Postgres handles neither well, and a filesystem handles neither at all.

We deliberately accept losing atomicity between bytes and rows, and pay for it in the open. This is the honest weak point of the decision, so it gets a named mechanism rather than a shrug: quarantine-first ordering means bytes always exist before the row that references them is promoted; a nightly reconciliation job cleans orphans; and bytes_missing is a first-class state because append-only history rows will outlive purged blobs by design. Note the asymmetry we chose: an orphaned blob is a cost problem, a dangling key is a UX problem, and neither is a correctness problem — whereas the reverse ordering (row first, bytes later) would let a clean-gated parse attempt reference bytes that never arrived.

Quarantine-before-promotion is not ceremony. The design runs parsers over attacker-supplied binaries as a core product function. Landing bytes in a container the parse path cannot read until a scan clears them makes "no unscanned file is ever parsed" a property of the storage layout plus a database trigger, rather than a code path someone can forget under deadline pressure.

Serving documents from a different origin, download-only, with sanitised previews, is the storage-layer half of the §E fix. Escaping HTML in the rendering layer stops candidate text from executing. It does nothing about a candidate-supplied PDF or HTML file opened inline from the app origin. Both halves are needed, and this ADR owns the second.

Turning off blob versioning is the decision most likely to be questioned, so the reasoning is recorded. Versioning plus a long soft-delete window is the standard safety configuration and we are declining it. The reason is that this platform makes an explicit promise — pseudonymise rows, delete CV bytes — and a storage feature that silently retains prior versions would make that promise false without anyone noticing. We accept a smaller accident-recovery margin (7 disclosed days, plus the fact that raw intake attachments are re-parseable from quarantine within 30 days of arrival) in exchange for an erasure guarantee we can defend.

---

Consequences

Positive

• Documents never touch the database's WAL, backups or connection pool, so the one component that cannot scale horizontally (ADR 0002) stays small and fast.
• Downloads bypass the web process entirely — no request thread, no DB connection, no memory spike on a 20 MB PDF.
• A separate credential and blast-radius boundary: compromising the database does not yield CV bytes, and vice versa.
• Immutable audit-archive gives the audit hash chain the one genuinely independent tamper check it has (layers 1–3 are all inside the database).
• Retention purge has a real deletion primitive, and retention_action.blob_keys_deleted makes each purge provable.
• Cost is negligible at projected volume, and Cool tiering handles the long tail without touching code.

Negative — costs we are accepting

Cost	Accepted because
No atomicity between bytes and rows. Orphaned blobs and dangling keys are both possible	Quarantine-first ordering, a nightly reconciliation job, and bytes_missing as an explicit state. Named as the price of Option C
A second stateful system to secure, monitor, drill and include in DR	Managed, no capacity planning, and it is the only option that provides immutability
Local development friction — an emulator or dev account is now required in a repo that today has no build step at all (§B)	The files facade is the only code aware of it; setup is documented once
Signed-URL operational surface: expiry, clock skew, CORS	120 s TTL is a deliberate trade of convenience for leak resistance; the failure mode is a re-click, not data loss
Reduced accident recovery (no versioning, 7-day soft delete)	The erasure guarantee outranks it; raw attachments remain re-parseable from quarantine for 30 days
No global content dedupe, so identical CVs are stored twice	Storage is cheap at this volume; reference counting would make erasure unsafe
Backup/restore is two systems, not one. A PITR restore of the database to time T does not restore blobs deleted after T	Blobs are write-once and the 7-day soft-delete window covers the realistic restore horizon. Documented in the DR runbook as a known non-atomic recovery
Server-mediated upload costs web-tier CPU and memory	Trivial at 200–600 documents/day; validation-before-persistence is worth more than the saving. Direct-to-blob has a numeric revisit trigger below

---

Risks

#	Risk	Likelihood	Impact	Mitigation
R1	A container is misconfigured to allow anonymous or broad read access, exposing CV PII	Low–Medium	Severe	Public access disabled at account level; infrastructure-as-code with a CI check asserting allowBlobPublicAccess = false and no container ACL is public; shared keys disabled so only managed identity works
R2	The malware-scan gate is bypassed by a code path that promotes or parses directly	Low	High	Gate enforced twice — service check and a database trigger on intake_parse_attempt; the parse queue reads only from candidate-documents, which quarantine bytes never enter until clean
R3	A signed URL leaks via referrer, screenshot or chat and is replayed	Medium	Medium	120 s TTL; read-only, single-blob scope; every issuance audited so replay is at least detectable after the fact
R4	Retention purge deletes rows but silently fails to delete blobs (or vice versa), leaving bytes we have claimed to erase	Medium	High — a compliance failure, not just a bug	Purge is a queue job with per-key result recording in retention_action; a weekly verifier re-checks that every purged stored_file key returns 404 and alerts on any that do not
R5	Orphaned blobs accumulate from failed uploads and rollbacks	High (expected)	Low	Nightly reconciliation job; cost is negligible either way
R6	A recruiter opens a hostile PDF inline and it executes	Low–Medium	High	Previews serve the sanitised derivative only; originals are Content-Disposition: attachment from a separate origin; SVG/HTML rejected at upload
R7	Legal determines CV storage must sit under tenant eDiscovery/retention labels	Medium	Medium	The files facade is the only integration point, so the backing store is swappable; Option D is explicitly recorded as the fallback rather than dismissed
R8	Storage region conflicts with a residency ruling across the six jurisdictions	Medium	Medium–High	Same posture as ADR 0002: one region, per-record retention, escalated to legal. Per-region storage accounts are as forbidden as per-region databases
R9	An immutability (Object Lock) policy is set too aggressively and blocks a legitimate audit redaction	Low	Medium	Audit payloads store hashes rather than values for sensitive_personal columns, so the archive should never need redaction; the narrow audit_event_redaction path applies to the live table only, and the archive is time-locked rather than legal-hold-locked

---

Revisit conditions

#	Trigger	Threshold	Expected response
T1	Total stored volume	>2 TB across all containers	Review tiering policy and per-document size cap; re-evaluate whether Cool-only is still right
T2	Egress	Sustained storage egress cost >20% of total platform infrastructure spend	Introduce a CDN in front of document-derivatives only (never originals)
T3	Upload cost on the web tier	p95 upload-request duration >10 s, or upload handling >20% of web-process CPU	Move to direct-to-blob upload with a server-issued short-scope write token, keeping validation as a post-upload gate in quarantine — the ordering guarantee is preserved because quarantine is already pre-scan
T4	Signed-URL issuance	>5,000 issuances/hour sustained	Introduce short-lived per-session caching of URLs; re-check that audit access-event volume is still tractable
T5	Scan latency	p95 time from arrival to virus_scan_status = clean >60 s, or >5 min at p99	Scale the ingest queue or move scanning to a dedicated worker revision (ADR 0001 T5)
T6	Document size	A legitimate business need for documents >25 MB (e.g. portfolio bundles)	Raise the cap only with a matching increase in parse timeout and memory cap, and re-test the decompression-ratio guard
T7	Compliance	Legal requires tenant-level eDiscovery/DLP over CVs, or customer-managed encryption keys	Reopen this ADR: Option D for eDiscovery, or Key Vault CMK, whichever is actually mandated
T8	Erasure verification	The weekly purge verifier reports any key still readable after a recorded deletion	Treat as a P1 incident; halt further purges until the cause is found, because the failure is silent by nature
T9	Provider	The Azure assumption is falsified (Utopia is not on Azure/M365)	The files facade and container model port unchanged to S3 + Object Lock; only the client and identity mechanism change. Nothing above depends on Azure-specific semantics beyond naming