HR-ATS-Portal/docs/architecture/adr/0002-primary-relational-database.md

ADR 0002 — PostgreSQL 16+ as the Single Primary Relational Database

Status	Accepted — 2026-07-29
Scope	Database engine, instance topology, schema layout, extension set, and how schema change is authored and applied
Owner	Talha Ahmed (senior); every migration reviewed by Talha
Consistent with	_decisions.md Part 2 → Database engine, Schema tooling and migration strategy, Phase 1 search strategy, Threshold for a separate search service, Audit table strategy
Related ADRs	0001 (one database is a premise of the modular monolith), 0003 (blobs are not in the database), 0004 (the job queue lives in this database)

---

Context

Nothing in the repository constrains this choice. Verified absent (_repo-findings.md §B): any database driver, any ORM or query builder, any migration directory or migration tool, any .env or .env.example, any Dockerfile or docker-compose.yml. The prototype's entire "database" is a seeded LCG generating 100 candidates plus jobs, interviews, assessments, offers and inbox rows in memory at page load (js/data.js:8-10, js/data.js:110-131); localStorage is used only for the theme preference (js/app.js:64,193,198). Nothing persists. This is therefore a free capability-fit decision, and it should be made by matching engine features to the brief's non-negotiable invariants rather than by familiarity.

The constraints are fixed before the options are opened: one platform, one master data model, ONE relational database. Not multi-tenant, no per-region databases, no second datastore (no Elasticsearch, no vector database) in Phase 1. Two developers, no ops staff.

What the invariants actually demand of an engine

This is the decisive table. Every row is a brief-level requirement, not a preference.

Requirement	Source	Feature required
Raw intake must exist before any candidate or application	brief §5.1; js/data.js:284-300 shows the prototype has no unresolved intake state at all	NOT NULL FKs + DEFERRABLE INITIALLY DEFERRED constraint triggers for the "at least one contact channel at COMMIT" rule
Store full inbound email/webhook envelopes and parser output verbatim	intake channels are Outlook mail and CV files	jsonb + GIN
"One email, one identity" as a database fact	brief §5.2	Partial + expression unique index on lower()-normalised address, tolerant of soft delete
Candidate identity separate from applications; at most one live application per (candidate, job)	brief §5.2; prototype carries jobId on the candidate (js/data.js:117-127)	GENERATED ALWAYS AS … STORED column + partial unique index whose predicate derives from status
Duplicate detection with typo tolerance, and manual review	brief §5.7	pg_trgm GIN + similarity(), unaccent
Reversible merge with a per-operation undo log	brief §5.7	Ordinary relational integrity + an append-only operation table; nothing exotic, but it must be transactional
Jobs, requirements and scoring configs versioned; historical scores must never drift	brief §5.3, §5.6; aiScore: int(52,98) today (js/data.js:123)	Immutable version rows with UPDATE revoked at the column/table grant level
ATS scores are per-application, append-only, version-pinned	brief §5.6	Column-level GRANT to make ats_result insert-only; superseded_by chains
Current state and history both exist	brief §5.5; no history tables exist today (§F)	tstzrange + btree_gist + EXCLUDE constraints to prevent overlapping validity intervals
Interview scheduling without double-booking, timezone-correct	no timezone discipline today; hardcoded "today" new Date('2026-07-09') (js/data.js:237)	timestamptz + tstzrange + GiST EXCLUDE
Money with currency	salary: int(90,190)*1000, no currency field anywhere; offer validation only checks > 0 (js/data.js:126, js/offers.js:129)	numeric(14,2) + ISO-4217 code, bound by CHECK
Append-only audit at tens of millions of rows, archivable	BRD §7.3	Declarative RANGE partitioning; pgcrypto for the hash chain
Explainable candidate search in Phase 1, semantic search in Phase 2, without a second datastore	brief; forbidden infra	tsvector/ts_rank_cd/setweight, pg_trgm, then pgvector in the same database
The chatbot must never bypass access controls	brief	Row-level security and column-level privileges available for the Phase 2 AI role with no new infrastructure
Transactional job enqueue	BRD §6.3 — no document silently lost	SKIP LOCKED + LISTEN/NOTIFY (see ADR 0004)

---

Options considered

Option A — PostgreSQL 16+ (single managed instance)

Pros

• Every single row of the table above maps to a first-class feature of one engine. Nothing has to move into application code.
• pgvector in the same database is what makes "no separate AI service in Phase 1" achievable rather than aspirational — hybrid retrieval (FTS/trigram candidate generation, vector rerank) needs no second store and no dual-write.
• Same index family (pg_trgm GIN) serves both candidate search and duplicate detection, so one mechanism gets tuned, understood and tested instead of two.
• Column-level GRANT and REVOKE let append-only-ness be enforced by the database rather than by convention — the difference between "we intend not to update scores" and "the application role cannot".
• Two developers get one engine, one backup story, one PITR story, and psql for ad-hoc work.
• Managed offerings (Azure Database for PostgreSQL Flexible Server) give PITR, automated backups and patching without an ops hire.

Cons (stated)

• Extension availability is a dependency on the managed provider. pgvector and pg_partman must be confirmed available and version-appropriate on the chosen tier before Phase 2 planning is finalised.
• Connection-per-backend model means every connection carries a memory cost, so the connection budget has to be planned rather than assumed. Phase 1 headroom is ample (~14 steady-state connections, budgeted in ADR 0004 §5), but the number grows multiplicatively with web replicas and worker processes.
• Vertical scaling only in Phase 1 (no read replica). The database is the one component whose saturation has no cheap horizontal answer.
• Major-version upgrades on a managed instance require a planned maintenance window with real downtime.
• FTS ranking quality is below a dedicated search engine's — accepted, with a named escalation ladder rather than pretended away.

Option B — MySQL 8 / MariaDB

Pros — genuine

• Extremely widely known; the easiest hiring and Stack Overflow surface, which matters with a junior developer.
• Excellent replication tooling and mature managed offerings everywhere.
• InnoDB is fast and predictable for the OLTP shape of most ATS traffic.
• Native JSON type and generated columns exist, so two of the requirements above are covered.

Cons — and they are disqualifying, one by one

• No trigram similarity index → typo-tolerant duplicate detection and fuzzy name search move into application code or a second datastore. This is the platform's single most important matching feature.
• No vector type → a separate vector database becomes mandatory in Phase 2, violating a hard constraint.
• No partial (WHERE) indexes → "one live application per (candidate, job), tolerant of soft delete and of merge supersession" cannot be expressed as a constraint. It becomes an application check, which is exactly the class of invariant the prototype already fails.
• No exclusion constraints → interview double-booking prevention moves into application code with a race window.
• No deferrable constraint triggers → the "at least one contact channel at COMMIT" rule cannot be expressed.
• Weaker declarative partitioning ergonomics for the audit archive.

The pattern is the point: MySQL does not merely score lower, it relocates four or five load-bearing invariants from the database into application code written under time pressure by a two-person team. That is the documented failure mode of the existing prototype.

Option C — Microsoft SQL Server

Pros

• Best-in-class tooling, query optimiser and execution-plan diagnostics.
• Native temporal (system-versioned) tables — a real advantage for the current-state-plus-history requirement, and the only option here that ships it.
• Filtered indexes (equivalent to partial indexes), CHECK constraints, and strong JSON support.
• Natural fit if Utopia is otherwise a Microsoft estate, which it plausibly is given M365/Outlook (assumption).

Cons

• Licensing cost with no capability gain that matters to us: system-versioned tables mirror columns, not transitions, so they do not actually answer "who moved this candidate from Screening to Interview and why" — we would still hand-write transition tables (see _decisions.md Part 2, Current state plus history).
• No trigram index and no first-party vector index at the maturity we need → same second-datastore problem as MySQL for both dedupe and Phase 2 semantics.
• Full-text search is a separate service component with a coarser index-maintenance model.
• The Python parsing/AI ecosystem this platform depends on (ADR 0001) is Postgres-centric; driver and extension ergonomics are worse.

Option D — MongoDB or another document store

Pros

• Storing raw email envelopes and heterogeneous parser output as documents is genuinely natural, and schema-per-document suits the messy shape of CV extraction.
• Horizontal scaling and flexible-schema iteration are real strengths early in a greenfield project.
• Atlas Search bundles full-text and vector search, which would cover Phase 1 and Phase 2 retrieval in one product.

Cons

• The entire brief is a list of relational invariants: raw-intake-before-candidate, identity separated from applications, per-application version-pinned scores, at-most-one-live-application, reversible merge with an undo log, non-overlapping validity intervals. In a document store every one of those becomes an application convention.
• That is precisely the failure the repository already exhibits — js/data.js:117-127 is a flat denormalised document per candidate, and it is why one candidate cannot hold two applications. Choosing a document store would institutionalise the prototype's data model as the production one.
• Cross-document transactions exist but are the exception rather than the default, and the Phase 1 write path spans four aggregates.
• The reversible merge undo log and audit hash chain both depend on strict ordering and on a role that physically cannot UPDATE — much weaker in this model.
• jsonb in Postgres already covers the legitimate document-shaped needs (raw payloads, parser output, before/after audit values) without giving up constraints.

Option E — PostgreSQL plus specialised stores in Phase 1 (Elasticsearch and/or a vector DB)

Pros

• Best search relevance and richest faceting available; per-field BM25 tuning and learning-to-rank become possible.
• Removes search load from the transactional database entirely.

Cons

• Forbidden by constraint, and absurd against the evidence: the prototype holds 100 generated rows (js/data.js:112), and the realistic corpus is 10⁴–10⁵ candidates accumulated over years (assumption).
• Introduces a permanent dual-write plus reindex-drift cost that two developers feel every week, and a second backup/restore story.
• In practice most "we need Elasticsearch" moments are an unindexed query or an untuned ranking function.

Option F — SQLite

Pros: zero operational cost, trivially reproducible in CI, single-file backups. Cons: no partitioning, weak write concurrency, no LISTEN/NOTIFY for the queue, no trigram/vector story, no role-level or column-level privileges. Viable for a single-user tool; not for a 66-seat multi-writer system with a worker process.

---

Decision

PostgreSQL 16+ (target 17). ONE managed instance, ONE logical database.

1. Instance and layout

Item	Decision
Hosting	Managed (Azure Database for PostgreSQL Flexible Server — assumption, tracking the Azure/M365 alignment in ADR 0001), PITR enabled, 14-day automated backups. No self-managed Postgres on a VM.
Sizing (Phase 1)	2 vCPU / 8 GB. No read replica. Vertical scaling is the Phase 1–4 answer.
Instances	Exactly one per environment (local via docker compose, staging, production). No per-region and no per-module databases.
Schemas	app (domain), ref (controlled vocabularies), audit (partitioned append-only log), ai (AiRun, prompt/model versions, embeddings), staging (raw intake landing)
Timezone	Server and application role set timezone = 'UTC'. All instants are timestamptz.
Roles	Separate migration role (owns DDL), app role (DML only, with UPDATE/DELETE revoked on append-only tables), and a Phase 2 ai_query role for the chatbot path with RLS policies and column privileges excluding sensitive_personal columns.

2. Extensions

Phase	Extensions	Purpose
1	pg_trgm	Fuzzy name/employer search and duplicate detection — one mechanism, two consumers
1	unaccent	Normalisation pipeline for names and free text
1	btree_gist	EXCLUDE constraints combining scalar equality with tstzrange overlap (history intervals, interview double-booking)
1	pgcrypto	sha256 for the audit hash chain, document checksums and deterministic retention tokens
2	pgvector	Semantic retrieval in the same database, HNSW index, embeddings versioned per model
2	pg_partman or a small scheduled SQL function	Monthly audit partition management

Confirm extension availability on the target managed tier before Phase 2 planning closes. This is an explicit action item, not an assumption to discover late.

3. Schema authoring and migration — the SQL-first rule

The schema is authored as plain SQL. The ORM maps to the schema; it never generates it.

Reconciling this with the Django choice in ADR 0001 (_decisions.md Part 1 requires a makemigrations --check gate; Part 2 requires SQL-authored DDL), the binding mechanism is:

Element	Decision
Runner	Django's own migration executor — it is the runner that "matches the backend language", it is the one procrastinate ships migrations against, and it makes CI test-database creation apply the real schema
Migration content	Hand-written SQL inside migrations.SeparateDatabaseAndState(database_operations=[RunSQL(...)], state_operations=[...]). The SQL is the truth; state_operations keeps Django's model state honest
Drift detection	makemigrations --check --dry-run as a required CI gate. Because state is declared explicitly, a model change without a migration fails the build, and a migration whose declared state diverges from its SQL surfaces as drift
Down-migrations	Not written. Recovery is forward-fix plus PITR
Review	Every migration reviewed by Talha, no exceptions
Tests	pytest against a real Postgres service container, never SQLite — the design depends on jsonb, partial unique indexes, FTS, pg_trgm, EXCLUDE constraints and LISTEN/NOTIFY

graph TD
  SQL["db/migrations/*.sql<br/>hand-written DDL — the source of truth"] --> RUN["Django migration executor<br/>SeparateDatabaseAndState"]
  RUN --> PGDB[("PostgreSQL<br/>app · ref · audit · ai · staging")]
  RUN --> STATE["Django model state"]
  STATE --> CHECK["CI: makemigrations --check<br/>drift gate"]
  MODELS["Django models<br/>mirror the schema"] --> CHECK
  PGDB --> CI["CI: pytest against real Postgres<br/>service container"]

---

Justification

One engine covers every invariant; every alternative relocates invariants into application code. That is the whole argument, and the requirements table above is its evidence. The prototype is a live demonstration of what happens when invariants live in application conventions: a flat candidate array where a person cannot hold two applications (js/data.js:117-127), a random integer masquerading as a score (js/data.js:123), no history at all, no currency on money, and an RBAC matrix nothing reads (js/rbac.js:78). The correction is to push invariants down into the engine, which requires an engine that can hold them.

Postgres is the only mainstream engine where duplicate detection and Phase 2 semantic search need no new infrastructure. Trigram similarity and vector search inside the same database is what turns "no separate AI service, no Elasticsearch in Phase 1" from a restriction we are grudgingly obeying into the natural design. On MySQL or SQL Server, honouring the same constraint would mean writing fuzzy matching by hand.

Column-level privileges are the mechanism behind three separate brief requirements. "Scores are append-only", "versions are immutable", "audit cannot be altered" are all implemented as REVOKE UPDATE, DELETE from the application role, plus a BEFORE UPDATE OR DELETE trigger that raises. Two independent layers, both travelling with the schema. No ORM-level convention achieves this.

SQL-first migrations, stated as the tradeoff it is. We give up the convenience of makemigrations autogeneration, and the junior developer has to read DDL. We accept that because the invariants are partial unique indexes with WHERE clauses, expression indexes on lower(), regex CHECKs, DEFERRABLE INITIALLY DEFERRED constraint triggers, GiST EXCLUDE constraints, generated columns, RANGE partitions and column-level GRANTs — and an ORM expresses approximately none of them. If the schema were ORM-declared, all of this would live in raw-SQL escape hatches anyway while the ORM's model of truth silently diverged. Plain SQL also makes the schema reviewable as a diff, which matters when one of two developers is junior and the other is the only reviewer.

One database is also a security decision. RLS and column privileges for the Phase 2 chatbot role mean the access boundary can be enforced by the database rather than by prompt engineering — with zero new infrastructure. A polyglot topology would put candidate PII in a second store with its own, weaker, access model.

---

Consequences

Positive

• Every non-negotiable invariant in the brief is expressible as schema. Correctness is enforced at the lowest possible layer.
• One backup, one PITR, one restore drill, one psql — the entire persistence operational surface for two developers.
• Duplicate detection and search share tuned indexes, halving the surface that must be understood and tested.
• Phase 2 semantics (pgvector) and Phase 2 chatbot isolation (RLS + column grants) both arrive without new components.
• The job queue is transactional with domain writes because it is in the same database (ADR 0004).
• Analytics read models are SQL views owned by one module — no ETL, no second warehouse in Phase 1.

Negative — costs we are accepting

Cost	Accepted because
The database is a single point of failure and the only stateful component. No read replica in Phase 1	It is managed, so recovery is PITR rather than a runbook we must write; business-hours internal tool
Vertical scaling only. Saturation has no cheap horizontal answer	66 seats; the escalation ladder (tune → replica → materialise) is defined before it is needed
Search relevance is below a dedicated engine's	Explicit numeric triggers replace vibes; a BM25 extension (pg_search/ParadeDB) sits on the ladder before any second datastore
Connections are scarce. A 2 vCPU instance's connection budget must be shared between web threads, worker concurrency and each worker's LISTEN connection	Budgeted explicitly in ADR 0004; PgBouncer in transaction mode is the named next step
Extension availability is a managed-provider dependency	Named as a pre-Phase-2 verification action, not an assumption
SQL-first migrations cost the team autogeneration convenience and raise the floor of SQL knowledge required	The invariants are unexpressible otherwise; Talha reviews every migration; it is also a genuine learning workstream for the junior
Major-version upgrades need a real maintenance window	Internal, business-hours; the six-jurisdiction spread narrows but does not eliminate the window
Audit and queue tables share the engine with OLTP, so their bloat and autovacuum behaviour affect recruiter latency	Audit is RANGE-partitioned so archival is DETACH, not mass DELETE; queue table health is an explicit monitoring item (ADR 0004)

---

Risks

#	Risk	Likelihood	Impact	Mitigation
R1	pgvector or pg_partman unavailable or version-lagging on the managed tier chosen	Medium	Medium	Verify before Phase 2 planning closes; fallback is a scheduled plpgsql partition function (already named) and, for vectors, deferring hybrid retrieval rather than adding a vector DB
R2	Connection exhaustion as web replicas and worker processes multiply (not a Phase 1 concern — ~14 steady-state against several hundred available)	Low in Phase 1, Medium later	High (hard outage)	Explicit connection budget in ADR 0004 §5; PgBouncer transaction pooling as the pre-planned fix at 40% of max_connections, with a session-mode path for the queue's LISTEN connection; alert at 70%
R3	Audit partition growth degrades vacuum/IO and hurts recruiter latency	Medium	Medium	Monthly RANGE partitions; detach + archive after 13 months; audit retention set independently of candidate retention
R4	GIN index write amplification on candidate_search_index slows the intake path	Low–Medium	Medium	Search index is one row per candidate maintained by trigger, not a generated column on a wide table; reindex is a targeted UPDATE
R5	SQL-first migrations drift from Django model state, so ORM queries assume a schema that does not exist	Medium	Medium	SeparateDatabaseAndState + makemigrations --check as a required gate; tests run against a real Postgres built from the same migrations
R6	The team, under deadline pressure, "temporarily" adds a second datastore	Low–Medium	High (violates a hard constraint and creates permanent dual-write)	The escalation ladder and numeric triggers below; any second datastore requires a new ADR superseding this one
R7	Single region conflicts with a legal ruling on data residency across the six jurisdictions	Medium	High	Escalated to legal as an open question; retention and deletion are implemented per record, not per region, so a ruling changes policy rows, not topology. Per-region databases remain forbidden

---

Revisit conditions

Reopen the engine choice only if

#	Trigger	Threshold
E1	A required capability is genuinely absent from Postgres including extensions	Named, with a failed spike documented
E2	Legal mandates data residency that a single instance cannot satisfy	A written legal determination — and even then the first response is region migration, not multiple databases

Reopen the topology (replica, partitioning strategy, pooling) when

#	Trigger	Threshold
T1	Database size	Total >500 GB, or audit.audit_event >200 million rows
T2	Candidate corpus	Candidate rows exceed ~2,000,000, or indexed searchable text exceeds ~50 GB
T3	Transactional latency	p95 for indexed single-row reads >100 ms, or p95 write >200 ms, after query and index tuning
T4	CPU	Sustained DB CPU >70% during business hours for 2+ weeks at 4 vCPU / 16 GB
T5	Connections	Peak connections >70% of max_connections (→ PgBouncer, before any sizing change)
T6	Vector memory	HNSW index working set no longer fits comfortably in shared buffers at the current tier

Reopen "search stays in Postgres" only when ANY of these actually fires

#	Trigger	Threshold
S1	Corpus	As T2
S2	Latency	p95 search >500 ms on the tuned FTS + trigram path after index tuning and after moving search to a read replica
S3	Throughput	Sustained >50 search queries/second and measurable degradation of transactional write latency
S4	Capability	A requirement Postgres genuinely cannot serve: live per-field BM25 relevance experimentation, learning-to-rank, sub-second facets across >~10 dimensions, or cross-entity typo-tolerant autocomplete under 50 ms

Exhaust these first, in order: index and query tuning → a read replica dedicated to search → a materialised search table → a BM25 extension (pg_search/ParadeDB) if ranking quality alone is the gap.

Honest projection, labelled an assumption: an internal Utopia Brands recruiting platform will hold on the order of 10⁴–10⁵ candidates accumulated over several years — three to four orders of magnitude below trigger S1/T2. The realistic conclusion is that Postgres FTS plus trigram will very likely never be outgrown for this system, and a read-replica-for-search is the ceiling of what will ever be needed.