HR-ATS-Portal/docs/architecture/adr/0002-primary-relational-database.md

# ADR 0002 — PostgreSQL 16+ as the Single Primary Relational Database

| | |
|---|---|
| **Status** | **Accepted** — 2026-07-29 |
| **Scope** | Database engine, instance topology, schema layout, extension set, and how schema change is authored and applied |
| **Owner** | Talha Ahmed (senior); every migration reviewed by Talha |
| **Consistent with** | `_decisions.md` Part 2 → *Database engine*, *Schema tooling and migration strategy*, *Phase 1 search strategy*, *Threshold for a separate search service*, *Audit table strategy* |
| **Related ADRs** | 0001 (one database is a premise of the modular monolith), 0003 (blobs are *not* in the database), 0004 (the job queue lives in this database) |

---

## Context

**Nothing in the repository constrains this choice.** Verified absent (`_repo-findings.md` §B): any database driver, any ORM or query builder, any migration directory or migration tool, any `.env` or `.env.example`, any `Dockerfile` or `docker-compose.yml`. The prototype's entire "database" is a seeded LCG generating 100 candidates plus jobs, interviews, assessments, offers and inbox rows in memory at page load (`js/data.js:8-10`, `js/data.js:110-131`); `localStorage` is used only for the theme preference (`js/app.js:64,193,198`). Nothing persists. This is therefore a free capability-fit decision, and it should be made by matching engine features to the brief's non-negotiable invariants rather than by familiarity.

The constraints are fixed before the options are opened: **one platform, one master data model, ONE relational database.** Not multi-tenant, no per-region databases, no second datastore (no Elasticsearch, no vector database) in Phase 1. Two developers, no ops staff.

### What the invariants actually demand of an engine

This is the decisive table. Every row is a brief-level requirement, not a preference.

| Requirement | Source | Feature required |
|---|---|---|
| Raw intake must exist before any candidate or application | brief §5.1; `js/data.js:284-300` shows the prototype has no unresolved intake state at all | `NOT NULL` FKs + `DEFERRABLE INITIALLY DEFERRED` constraint triggers for the "at least one contact channel at COMMIT" rule |
| Store full inbound email/webhook envelopes and parser output verbatim | intake channels are Outlook mail and CV files | `jsonb` + GIN |
| "One email, one identity" as a database fact | brief §5.2 | Partial + expression unique index on `lower()`-normalised address, tolerant of soft delete |
| Candidate identity separate from applications; at most one live application per (candidate, job) | brief §5.2; prototype carries `jobId` on the candidate (`js/data.js:117-127`) | `GENERATED ALWAYS AS … STORED` column + partial unique index whose predicate derives from status |
| Duplicate detection with typo tolerance, and manual review | brief §5.7 | `pg_trgm` GIN + `similarity()`, `unaccent` |
| Reversible merge with a per-operation undo log | brief §5.7 | Ordinary relational integrity + an append-only operation table; nothing exotic, but it *must* be transactional |
| Jobs, requirements and scoring configs versioned; historical scores must never drift | brief §5.3, §5.6; `aiScore: int(52,98)` today (`js/data.js:123`) | Immutable version rows with `UPDATE` revoked at the **column/table grant** level |
| ATS scores are per-application, append-only, version-pinned | brief §5.6 | Column-level `GRANT` to make `ats_result` insert-only; `superseded_by` chains |
| Current state **and** history both exist | brief §5.5; no history tables exist today (§F) | `tstzrange` + `btree_gist` + `EXCLUDE` constraints to prevent overlapping validity intervals |
| Interview scheduling without double-booking, timezone-correct | no timezone discipline today; hardcoded "today" `new Date('2026-07-09')` (`js/data.js:237`) | `timestamptz` + `tstzrange` + GiST `EXCLUDE` |
| Money with currency | `salary: int(90,190)*1000`, no currency field anywhere; offer validation only checks `> 0` (`js/data.js:126`, `js/offers.js:129`) | `numeric(14,2)` + ISO-4217 code, bound by `CHECK` |
| Append-only audit at tens of millions of rows, archivable | BRD §7.3 | Declarative `RANGE` partitioning; `pgcrypto` for the hash chain |
| Explainable candidate search in Phase 1, semantic search in Phase 2, **without a second datastore** | brief; forbidden infra | `tsvector`/`ts_rank_cd`/`setweight`, `pg_trgm`, then `pgvector` **in the same database** |
| The chatbot must never bypass access controls | brief | Row-level security and column-level privileges available for the Phase 2 AI role with **no new infrastructure** |
| Transactional job enqueue | BRD §6.3 — no document silently lost | `SKIP LOCKED` + `LISTEN`/`NOTIFY` (see ADR 0004) |

---

## Options considered

### Option A — PostgreSQL 16+ (single managed instance)

**Pros**
- Every single row of the table above maps to a first-class feature of one engine. Nothing has to move into application code.
- `pgvector` in the *same* database is what makes "no separate AI service in Phase 1" achievable rather than aspirational — hybrid retrieval (FTS/trigram candidate generation, vector rerank) needs no second store and no dual-write.
- Same index family (`pg_trgm` GIN) serves both candidate search and duplicate detection, so one mechanism gets tuned, understood and tested instead of two.
- Column-level `GRANT` and `REVOKE` let append-only-ness be enforced by the database rather than by convention — the difference between "we intend not to update scores" and "the application role cannot".
- Two developers get one engine, one backup story, one PITR story, and `psql` for ad-hoc work.
- Managed offerings (Azure Database for PostgreSQL Flexible Server) give PITR, automated backups and patching without an ops hire.

**Cons (stated)**
- Extension availability is a dependency on the managed provider. `pgvector` and `pg_partman` must be confirmed available and version-appropriate on the chosen tier before Phase 2 planning is finalised.
- Connection-per-backend model means every connection carries a memory cost, so the connection budget has to be planned rather than assumed. Phase 1 headroom is ample (~14 steady-state connections, budgeted in ADR 0004 §5), but the number grows multiplicatively with web replicas and worker processes.
- Vertical scaling only in Phase 1 (no read replica). The database is the one component whose saturation has no cheap horizontal answer.
- Major-version upgrades on a managed instance require a planned maintenance window with real downtime.
- FTS ranking quality is below a dedicated search engine's — accepted, with a named escalation ladder rather than pretended away.

### Option B — MySQL 8 / MariaDB

**Pros — genuine**
- Extremely widely known; the easiest hiring and Stack Overflow surface, which matters with a junior developer.
- Excellent replication tooling and mature managed offerings everywhere.
- InnoDB is fast and predictable for the OLTP shape of most ATS traffic.
- Native JSON type and generated columns exist, so two of the requirements above are covered.

**Cons — and they are disqualifying, one by one**
- No trigram similarity index → typo-tolerant duplicate detection and fuzzy name search move into application code or a second datastore. This is the platform's single most important matching feature.
- No vector type → a separate vector database becomes mandatory in Phase 2, violating a hard constraint.
- No partial (`WHERE`) indexes → "one live application per (candidate, job), tolerant of soft delete and of merge supersession" cannot be expressed as a constraint. It becomes an application check, which is exactly the class of invariant the prototype already fails.
- No exclusion constraints → interview double-booking prevention moves into application code with a race window.
- No deferrable constraint triggers → the "at least one contact channel at COMMIT" rule cannot be expressed.
- Weaker declarative partitioning ergonomics for the audit archive.

The pattern is the point: MySQL does not merely score lower, it relocates four or five load-bearing invariants from the database into application code written under time pressure by a two-person team. That is the documented failure mode of the existing prototype.

### Option C — Microsoft SQL Server

**Pros**
- Best-in-class tooling, query optimiser and execution-plan diagnostics.
- Native temporal (system-versioned) tables — a real advantage for the current-state-plus-history requirement, and the only option here that ships it.
- Filtered indexes (equivalent to partial indexes), `CHECK` constraints, and strong JSON support.
- Natural fit if Utopia is otherwise a Microsoft estate, which it plausibly is given M365/Outlook (**assumption**).

**Cons**
- Licensing cost with no capability gain that matters to us: system-versioned tables mirror *columns*, not *transitions*, so they do not actually answer "who moved this candidate from Screening to Interview and why" — we would still hand-write transition tables (see `_decisions.md` Part 2, *Current state plus history*).
- No trigram index and no first-party vector index at the maturity we need → same second-datastore problem as MySQL for both dedupe and Phase 2 semantics.
- Full-text search is a separate service component with a coarser index-maintenance model.
- The Python parsing/AI ecosystem this platform depends on (ADR 0001) is Postgres-centric; driver and extension ergonomics are worse.

### Option D — MongoDB or another document store

**Pros**
- Storing raw email envelopes and heterogeneous parser output as documents is genuinely natural, and schema-per-document suits the messy shape of CV extraction.
- Horizontal scaling and flexible-schema iteration are real strengths early in a greenfield project.
- Atlas Search bundles full-text and vector search, which would cover Phase 1 and Phase 2 retrieval in one product.

**Cons**
- The entire brief is a list of *relational invariants*: raw-intake-before-candidate, identity separated from applications, per-application version-pinned scores, at-most-one-live-application, reversible merge with an undo log, non-overlapping validity intervals. In a document store every one of those becomes an application convention.
- That is precisely the failure the repository already exhibits — `js/data.js:117-127` is a flat denormalised document per candidate, and it is why one candidate cannot hold two applications. Choosing a document store would institutionalise the prototype's data model as the production one.
- Cross-document transactions exist but are the exception rather than the default, and the Phase 1 write path spans four aggregates.
- The reversible merge undo log and audit hash chain both depend on strict ordering and on a role that physically cannot `UPDATE` — much weaker in this model.
- `jsonb` in Postgres already covers the legitimate document-shaped needs (raw payloads, parser output, `before`/`after` audit values) without giving up constraints.

### Option E — PostgreSQL plus specialised stores in Phase 1 (Elasticsearch and/or a vector DB)

**Pros**
- Best search relevance and richest faceting available; per-field BM25 tuning and learning-to-rank become possible.
- Removes search load from the transactional database entirely.

**Cons**
- Forbidden by constraint, and absurd against the evidence: the prototype holds 100 generated rows (`js/data.js:112`), and the realistic corpus is 10⁴–10⁵ candidates accumulated over years (**assumption**).
- Introduces a permanent dual-write plus reindex-drift cost that two developers feel every week, and a second backup/restore story.
- In practice most "we need Elasticsearch" moments are an unindexed query or an untuned ranking function.

### Option F — SQLite

**Pros:** zero operational cost, trivially reproducible in CI, single-file backups.
**Cons:** no partitioning, weak write concurrency, no `LISTEN`/`NOTIFY` for the queue, no trigram/vector story, no role-level or column-level privileges. Viable for a single-user tool; not for a 66-seat multi-writer system with a worker process.

---

## Decision

**PostgreSQL 16+ (target 17). ONE managed instance, ONE logical database.**

### 1. Instance and layout

| Item | Decision |
|---|---|
| Hosting | Managed (Azure Database for PostgreSQL Flexible Server — **assumption**, tracking the Azure/M365 alignment in ADR 0001), PITR enabled, 14-day automated backups. No self-managed Postgres on a VM. |
| Sizing (Phase 1) | 2 vCPU / 8 GB. No read replica. Vertical scaling is the Phase 1–4 answer. |
| Instances | Exactly one per environment (local via docker compose, staging, production). No per-region and no per-module databases. |
| Schemas | `app` (domain), `ref` (controlled vocabularies), `audit` (partitioned append-only log), `ai` (AiRun, prompt/model versions, embeddings), `staging` (raw intake landing) |
| Timezone | Server **and** application role set `timezone = 'UTC'`. All instants are `timestamptz`. |
| Roles | Separate `migration` role (owns DDL), `app` role (DML only, with `UPDATE`/`DELETE` revoked on append-only tables), and a Phase 2 `ai_query` role for the chatbot path with RLS policies and column privileges excluding `sensitive_personal` columns. |

### 2. Extensions

| Phase | Extensions | Purpose |
|---|---|---|
| 1 | `pg_trgm` | Fuzzy name/employer search **and** duplicate detection — one mechanism, two consumers |
| 1 | `unaccent` | Normalisation pipeline for names and free text |
| 1 | `btree_gist` | `EXCLUDE` constraints combining scalar equality with `tstzrange` overlap (history intervals, interview double-booking) |
| 1 | `pgcrypto` | `sha256` for the audit hash chain, document checksums and deterministic retention tokens |
| 2 | `pgvector` | Semantic retrieval in the same database, HNSW index, embeddings versioned per model |
| 2 | `pg_partman` *or* a small scheduled SQL function | Monthly audit partition management |

**Confirm extension availability on the target managed tier before Phase 2 planning closes.** This is an explicit action item, not an assumption to discover late.

### 3. Schema authoring and migration — the SQL-first rule

**The schema is authored as plain SQL. The ORM maps to the schema; it never generates it.**

Reconciling this with the Django choice in ADR 0001 (`_decisions.md` Part 1 requires a `makemigrations --check` gate; Part 2 requires SQL-authored DDL), the binding mechanism is:

| Element | Decision |
|---|---|
| Runner | Django's own migration executor — it is the runner that "matches the backend language", it is the one `procrastinate` ships migrations against, and it makes CI test-database creation apply the real schema |
| Migration content | Hand-written SQL inside `migrations.SeparateDatabaseAndState(database_operations=[RunSQL(...)], state_operations=[...])`. The SQL is the truth; `state_operations` keeps Django's model state honest |
| Drift detection | `makemigrations --check --dry-run` as a required CI gate. Because state is declared explicitly, a model change without a migration fails the build, and a migration whose declared state diverges from its SQL surfaces as drift |
| Down-migrations | Not written. Recovery is forward-fix plus PITR |
| Review | Every migration reviewed by Talha, no exceptions |
| Tests | pytest against a **real Postgres service container**, never SQLite — the design depends on `jsonb`, partial unique indexes, FTS, `pg_trgm`, `EXCLUDE` constraints and `LISTEN`/`NOTIFY` |

```mermaid
graph TD
  SQL["db/migrations/*.sql<br/>hand-written DDL — the source of truth"] --> RUN["Django migration executor<br/>SeparateDatabaseAndState"]
  RUN --> PGDB[("PostgreSQL<br/>app · ref · audit · ai · staging")]
  RUN --> STATE["Django model state"]
  STATE --> CHECK["CI: makemigrations --check<br/>drift gate"]
  MODELS["Django models<br/>mirror the schema"] --> CHECK
  PGDB --> CI["CI: pytest against real Postgres<br/>service container"]
```

---

## Justification

**One engine covers every invariant; every alternative relocates invariants into application code.** That is the whole argument, and the requirements table above is its evidence. The prototype is a live demonstration of what happens when invariants live in application conventions: a flat candidate array where a person cannot hold two applications (`js/data.js:117-127`), a random integer masquerading as a score (`js/data.js:123`), no history at all, no currency on money, and an RBAC matrix nothing reads (`js/rbac.js:78`). The correction is to push invariants *down* into the engine, which requires an engine that can hold them.

**Postgres is the only mainstream engine where duplicate detection and Phase 2 semantic search need no new infrastructure.** Trigram similarity and vector search inside the same database is what turns "no separate AI service, no Elasticsearch in Phase 1" from a restriction we are grudgingly obeying into the natural design. On MySQL or SQL Server, honouring the same constraint would mean writing fuzzy matching by hand.

**Column-level privileges are the mechanism behind three separate brief requirements.** "Scores are append-only", "versions are immutable", "audit cannot be altered" are all implemented as `REVOKE UPDATE, DELETE` from the application role, plus a `BEFORE UPDATE OR DELETE` trigger that raises. Two independent layers, both travelling with the schema. No ORM-level convention achieves this.

**SQL-first migrations, stated as the tradeoff it is.** We give up the convenience of `makemigrations` autogeneration, and the junior developer has to read DDL. We accept that because the invariants *are* partial unique indexes with `WHERE` clauses, expression indexes on `lower()`, regex `CHECK`s, `DEFERRABLE INITIALLY DEFERRED` constraint triggers, GiST `EXCLUDE` constraints, generated columns, `RANGE` partitions and column-level `GRANT`s — and an ORM expresses approximately none of them. If the schema were ORM-declared, all of this would live in raw-SQL escape hatches anyway while the ORM's model of truth silently diverged. Plain SQL also makes the schema reviewable as a diff, which matters when one of two developers is junior and the other is the only reviewer.

**One database is also a security decision.** RLS and column privileges for the Phase 2 chatbot role mean the access boundary can be enforced by the database rather than by prompt engineering — with zero new infrastructure. A polyglot topology would put candidate PII in a second store with its own, weaker, access model.

---

## Consequences

### Positive

- Every non-negotiable invariant in the brief is expressible as schema. Correctness is enforced at the lowest possible layer.
- One backup, one PITR, one restore drill, one `psql` — the entire persistence operational surface for two developers.
- Duplicate detection and search share tuned indexes, halving the surface that must be understood and tested.
- Phase 2 semantics (`pgvector`) and Phase 2 chatbot isolation (RLS + column grants) both arrive without new components.
- The job queue is transactional with domain writes because it is in the same database (ADR 0004).
- Analytics read models are SQL views owned by one module — no ETL, no second warehouse in Phase 1.

### Negative — costs we are accepting

| Cost | Accepted because |
|---|---|
| **The database is a single point of failure and the only stateful component.** No read replica in Phase 1 | It is managed, so recovery is PITR rather than a runbook we must write; business-hours internal tool |
| **Vertical scaling only.** Saturation has no cheap horizontal answer | 66 seats; the escalation ladder (tune → replica → materialise) is defined before it is needed |
| **Search relevance is below a dedicated engine's** | Explicit numeric triggers replace vibes; a BM25 extension (`pg_search`/ParadeDB) sits on the ladder before any second datastore |
| **Connections are scarce.** A 2 vCPU instance's connection budget must be shared between web threads, worker concurrency and each worker's `LISTEN` connection | Budgeted explicitly in ADR 0004; PgBouncer in transaction mode is the named next step |
| **Extension availability is a managed-provider dependency** | Named as a pre-Phase-2 verification action, not an assumption |
| **SQL-first migrations cost the team autogeneration convenience and raise the floor of SQL knowledge required** | The invariants are unexpressible otherwise; Talha reviews every migration; it is also a genuine learning workstream for the junior |
| **Major-version upgrades need a real maintenance window** | Internal, business-hours; the six-jurisdiction spread narrows but does not eliminate the window |
| **Audit and queue tables share the engine with OLTP**, so their bloat and autovacuum behaviour affect recruiter latency | Audit is `RANGE`-partitioned so archival is `DETACH`, not mass `DELETE`; queue table health is an explicit monitoring item (ADR 0004) |

---

## Risks

| # | Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|---|
| R1 | `pgvector` or `pg_partman` unavailable or version-lagging on the managed tier chosen | Medium | Medium | Verify before Phase 2 planning closes; fallback is a scheduled plpgsql partition function (already named) and, for vectors, deferring hybrid retrieval rather than adding a vector DB |
| R2 | Connection exhaustion as web replicas and worker processes multiply (not a Phase 1 concern — ~14 steady-state against several hundred available) | Low in Phase 1, Medium later | High (hard outage) | Explicit connection budget in ADR 0004 §5; PgBouncer transaction pooling as the pre-planned fix at 40% of `max_connections`, with a session-mode path for the queue's `LISTEN` connection; alert at 70% |
| R3 | Audit partition growth degrades vacuum/IO and hurts recruiter latency | Medium | Medium | Monthly `RANGE` partitions; detach + archive after 13 months; audit retention set independently of candidate retention |
| R4 | GIN index write amplification on `candidate_search_index` slows the intake path | Low–Medium | Medium | Search index is one row per candidate maintained by trigger, not a generated column on a wide table; reindex is a targeted `UPDATE` |
| R5 | SQL-first migrations drift from Django model state, so ORM queries assume a schema that does not exist | Medium | Medium | `SeparateDatabaseAndState` + `makemigrations --check` as a required gate; tests run against a real Postgres built from the same migrations |
| R6 | The team, under deadline pressure, "temporarily" adds a second datastore | Low–Medium | High (violates a hard constraint and creates permanent dual-write) | The escalation ladder and numeric triggers below; any second datastore requires a new ADR superseding this one |
| R7 | Single region conflicts with a legal ruling on data residency across the six jurisdictions | Medium | High | Escalated to legal as an open question; retention and deletion are implemented **per record**, not per region, so a ruling changes policy rows, not topology. Per-region databases remain forbidden |

---

## Revisit conditions

### Reopen the engine choice only if

| # | Trigger | Threshold |
|---|---|---|
| E1 | A required capability is genuinely absent from Postgres including extensions | Named, with a failed spike documented |
| E2 | Legal mandates data residency that a single instance cannot satisfy | A written legal determination — and even then the first response is region migration, not multiple databases |

### Reopen the *topology* (replica, partitioning strategy, pooling) when

| # | Trigger | Threshold |
|---|---|---|
| T1 | Database size | Total >500 GB, or `audit.audit_event` >200 million rows |
| T2 | Candidate corpus | Candidate rows exceed ~2,000,000, or indexed searchable text exceeds ~50 GB |
| T3 | Transactional latency | p95 for indexed single-row reads >100 ms, or p95 write >200 ms, after query and index tuning |
| T4 | CPU | Sustained DB CPU >70% during business hours for 2+ weeks at 4 vCPU / 16 GB |
| T5 | Connections | Peak connections >70% of `max_connections` (→ PgBouncer, before any sizing change) |
| T6 | Vector memory | HNSW index working set no longer fits comfortably in shared buffers at the current tier |

### Reopen "search stays in Postgres" only when ANY of these actually fires

| # | Trigger | Threshold |
|---|---|---|
| S1 | Corpus | As T2 |
| S2 | Latency | p95 search >500 ms on the tuned FTS + trigram path **after** index tuning **and after** moving search to a read replica |
| S3 | Throughput | Sustained >50 search queries/second **and** measurable degradation of transactional write latency |
| S4 | Capability | A requirement Postgres genuinely cannot serve: live per-field BM25 relevance experimentation, learning-to-rank, sub-second facets across >~10 dimensions, or cross-entity typo-tolerant autocomplete under 50 ms |

**Exhaust these first, in order:** index and query tuning → a read replica dedicated to search → a materialised search table → a BM25 extension (`pg_search`/ParadeDB) if ranking quality alone is the gap.

**Honest projection, labelled an assumption:** an internal Utopia Brands recruiting platform will hold on the order of 10⁴–10⁵ candidates accumulated over several years — three to four orders of magnitude below trigger S1/T2. The realistic conclusion is that Postgres FTS plus trigram will **very likely never be outgrown** for this system, and a read-replica-for-search is the ceiling of what will ever be needed.