108 KiB
System Architecture — Utopia Brands Recruitment & ATS Platform
Status / Scope of this document
Status: Design baseline for implementation. Binding within the limits set by
docs/architecture/_decisions.md Part 1 — where this document adds detail it elaborates
those decisions; where it disagrees, the disagreement is recorded in §15 as a risk to
reconcile, never applied silently.
Scope: the runtime shape of the platform — processes, module boundaries, deployment topology, data flow through the four intake and interaction paths, background work, failure behaviour, scaling triggers, observability and the deploy/rollback procedure.
Out of scope: the relational schema (see _decisions.md Part 2 and the data-model
document), REST endpoint-level contracts, and the UI screen specifications.
Evidence base: docs/architecture/_repo-findings.md (verified repository inspection,
2026-07-29) and the assignment brief. There is no meeting transcript — none exists in
the repository or the surrounding filesystem (_repo-findings.md §B). The assignment
prompt is the authoritative requirements source, and the distinction between CONFIRMED
(§3) and UNCONFIRMED (§4) requirements is preserved throughout.
What exists today: a static, browser-only frontend prototype — index.html,
css/styles.css (1269 lines), 22 unbundled scripts under js/, and a 36-line
devserver.py for local preview. There is no backend, no database, no authentication, no
authorization, no tests, no build step, no Docker and no environment files
(_repo-findings.md §B, §D). Everything described below as backend is greenfield. The
assets worth retaining are the design system (css/styles.css), the UI primitives
(js/ui.js:251), the canvas chart engine (js/charts.js) and the 23-route information
architecture (js/app.js:7-16).
Assumptions are labelled ASSUMPTION inline and collected in §14.
1. Recommended architecture
A modular monolith: one Django 5 / DRF codebase, one container image, two runtime processes, one PostgreSQL 16 database, one object store, one Redis used only as a cache.
| Element | Choice | Why, in one line |
|---|---|---|
| Backend | Python 3.12 + Django 5 + DRF, psycopg3, drf-spectacular | Phase 1 intake is PDF/DOCX/OCR and Outlook mail; that ecosystem is decisively Python, and Django supplies auth, permissions, admin and migrations that would otherwise be a month of platform work for two people |
| Processes | web (uvicorn ASGI) and worker (procrastinate consumer), same image, different entrypoint |
Parsing a scanned CV is CPU-bound and multi-second; a recruiter request is I/O-bound and sub-second. They must not share a request thread |
| Database | One managed PostgreSQL 16+, schemas app / ref / audit / ai / staging |
One master data model, one backup story, one psql. Forbidden to split |
| Queue | procrastinate — durable queue inside the same PostgreSQL |
Enqueue commits in the same transaction as the intake row, so the "document silently lost" bug class cannot occur without building an outbox |
| Cache | Managed Redis — cache, rate limiting, sessions. Never a broker, never a store of record | One durable component, not two |
| Blobs | Object storage, accessed only through the files module with signed URLs |
Retention and deletion obligations attach to blobs and must be enforceable in one place |
| Frontend | Progressive strangler migration to Vite + TypeScript + React 18; css/styles.css content-frozen and js/charts.js retained verbatim |
The CSS is the verified asset (dual themes, WCAG 2.1 AA across 23 routes × 2 themes); the string-template innerHTML rendering layer is the liability (34 unescaped sites, _repo-findings.md §E) |
| Module boundaries | Python packages, one service.py facade each, enforced by import-linter contracts in CI |
Boundaries that are not machine-checked decay; the prototype already demonstrates the failure mode with everything on window (index.html:264-285) |
How the architecture discharges each non-negotiable constraint
| Constraint | Structural mechanism | Where it lives |
|---|---|---|
| Raw intake exists before candidate creation | candidate.created_from_raw_intake_id and job_application.raw_intake_id are NOT NULL against non-deferrable FKs. Manual recruiter entry is not an exception — the UI writes a manual_ui intake row first |
intake module + database FK |
| Candidate identity separate from applications | candidate and job_application are distinct aggregates in distinct modules; nothing on candidate references a job |
candidate, application |
| ATS scores are per-application | ApplicationScore keys on application_id, never candidate_id; append-only with superseded_by |
scoring |
| Jobs, requirements, scoring configs versioned | Immutable version rows with current_version_id pointers and UPDATE revoked; every downstream row pins the version it consumed |
requisition, scoring |
| Duplicate detection with manual review and reversible merge | Detection is a background scan producing suspected links; merge is additive re-pointing with a per-operation undo log |
duplicate_review |
| Recruiter assignment flexible and historical | Two concrete interval tables, JobAssignment and JobApplicationAssignment — never one polymorphic (subject_type, subject_id) table, because a polymorphic FK cannot be enforced by the database. [valid_from, valid_to) intervals, valid_to IS NULL = current, a GiST EXCLUDE constraint against overlapping identical (subject, user, role) intervals, and a partial unique index giving exactly one current primary recruiter per job. One assignment service facade presents both |
assignment |
| Current state and history | Denormalised current column on the aggregate, maintained by trigger, alongside typed history tables; plus the append-only audit.audit_event |
every domain module + audit |
| AI explainable, versioned, reviewable | Every invocation writes an AiRun (capability, model id + version, prompt template version, input ref, raw output, tokens, cost, latency) before its result is usable; AiReview records verdicts |
ai_orchestration |
| AI never auto-rejects | intelligence tier is forbidden by an import-linter contract from importing domain modules for write; application.transition() refuses terminal-negative transitions unless actor_kind = 'user' — the exact literal, per _decisions.md RULING-01. actor_kind is ('user','system','integration','ai_agent'); there is no human value, and a guard spelled actor_type != 'human' would compare against a value absent from the CHECK and so fail closed on every legitimate human rejection or throw |
dependency graph + service guard + DB check |
| Chatbot never bypasses access control | ai_orchestration.invoke() takes the human actor and calls iam.can() with it. There is no service account and no system principal for the assistant. Phase 1 gives the assistant no SQL access at all — only whitelisted, typed, parameterised query intents over the same service layer the UI uses |
assistant, identity |
| Not multi-tenant, one database | No tenant column, no region column, no tenant module. Retention and residency are per-record, never per-topology |
schema-wide |
Deliberately not introduced in Phase 1
Kubernetes, Kafka or any log broker, microservices, Elasticsearch/OpenSearch, a second database, a separate vector database, a separate AI or parsing service, a BFF, GraphQL, self-hosted model inference, read replicas. Each is either forbidden by constraint or unjustified at 66 named seats and a projected 20k–60k applications/year (ASSUMPTION). §10 gives the numeric trigger that would change each answer.
2. Architecture comparison
Three candidates, scored 1 (worst) to 5 (best) against weighted criteria. Weights reflect this project's actual constraints: two developers with one reviewer, 66 users, one mandated database, and a Phase 1 critical path that must be transactional.
| # | Criterion | Weight | Modular monolith (2 processes) | Microservices (per-module) | Hybrid (monolith + extracted AI/parsing service) |
|---|---|---|---|---|---|
| 1 | Fit to team size — 2 devs, 1 reviewer, no ops staff | 5 | 5 — one pipeline, one dashboard set, one runbook | 1 — 20+ pipelines and contract versioning for two people is malpractice | 3 — two pipelines, two images, two on-call surfaces |
| 2 | Fit to existing stack and repo assets | 3 | 5 — no backend exists to preserve (§B); one API for one SPA consumer | 3 — nothing to reuse, and the SPA gains an aggregation problem | 4 — same as monolith plus one seam |
| 3 | Module boundary strength | 4 | 4 — package isolation + service.py facade + CI contracts; strong but bypassable by a determined developer |
5 — network boundary is unbypassable | 4 |
| 4 | Deployment complexity | 4 | 5 — one image, one revision pair, one manual promote | 1 — service mesh or hand-rolled discovery, N deploy orders, N schema migrations | 3 — two images with a version-compatibility matrix |
| 5 | Scalability to projected load | 3 | 5 — ~25 peak concurrent users, 200–600 docs/day peak (ASSUMPTION) is comfortably inside one 2-vCPU web + one 2-vCPU worker | 5 — also sufficient, at far higher cost | 5 |
| 6 | Failure isolation | 4 | 3 — the genuine loss; bought back by the worker already being a separate process and an AI circuit breaker | 5 | 4 |
| 7 | Security posture — single authorization decision point | 5 | 5 — one iam.can(); directly serves the chatbot constraint |
2 — every service re-derives authorization, or a shared library drifts between them | 4 — the AI service must be given a scoped, auditable channel back into can() |
| 8 | Development speed, Phase 1 | 5 | 5 — the intake→score slice is one repository and one transaction | 1 — a saga plus compensations across four services | 3 |
| 9 | Operational burden | 4 | 5 — one durable component to back up (Postgres), because the queue lives in it | 1 — N services, N queues, distributed tracing mandatory not optional | 3 |
| 10 | Testability | 4 | 4 — each module facade is a test seam; real Postgres in CI; no inter-service mocks | 2 — contract tests, test doubles, environment orchestration | 3 |
| 11 | Transactional integrity across intake→parse→candidate→application→score | 5 | 5 — one Postgres transaction | 1 — eventual consistency in the one flow that must never lose a document | 3 — parse result arrives asynchronously, which is already the design |
| Weighted total (max 230) | 214 | 105 | 160 |
Verdict: modular monolith. Microservices lose on every criterion that this project is actually constrained by, and would drag in forbidden infrastructure (a cluster, or per-service databases against the one-database rule). The hybrid is the only serious alternative, and it is rejected as a Phase 1 starting point but adopted as the Phase 2+ evolution path — the worker/queue split already gives its main benefit, and §10 names the exact triggers that would promote the parsing/AI worker to a separate deployable. Starting hybrid would pay the cost of a second deployable before any evidence justified it.
Honest statement of what is given up: with one app host there is no high availability. A platform-level restart is a few minutes of downtime. That is acceptable for an internal recruiting tool used in business hours, but the six-jurisdiction user spread narrows the maintenance window, and this should be told to the business rather than discovered during the first deploy that overlaps Singapore hours.
3. System context
Diagram 1 — System context
graph TB
subgraph PEOPLE["People"]
REC["Recruiter<br/>15 seats"]
HM["Hiring manager<br/>12 seats"]
INTV["Interviewer<br/>24 seats"]
LEAD["Talent lead / HR admin<br/>6 seats"]
EXEC["Dept head / CEO<br/>3 seats"]
CAND["Candidate<br/>external, token-gated"]
end
subgraph PLATFORM["TalentFlow ATS — one deployable image, two processes"]
WEBP["web process<br/>REST API v1 + React bundle + streaming assistant"]
WORKP["worker process<br/>parse, score, poll, sweep"]
PG[("PostgreSQL 16<br/>domain + queue + audit")]
OBJ[("Object storage<br/>CV and letter blobs")]
RDS[("Redis<br/>cache, rate limit, sessions")]
end
subgraph EXTERNAL["External systems"]
ENTRA["Entra ID<br/>SSO, ASSUMPTION"]
GRAPH["Microsoft Graph<br/>recruiting mailbox"]
CAREERS["Careers site<br/>public application form"]
BOARDS["Job boards<br/>Phase 3 in, Phase 4 out"]
AIP["AI provider API<br/>contracted, under DPA"]
SMTP["Outbound email<br/>transactional"]
end
REC --> WEBP
HM --> WEBP
INTV --> WEBP
LEAD --> WEBP
EXEC --> WEBP
CAND --> CAREERS
CAND -->|"token-gated status page"| WEBP
WEBP --> PG
WEBP --> OBJ
WEBP --> RDS
WORKP --> PG
WORKP --> OBJ
WEBP -.->|"enqueue in the same transaction"| PG
PG -.->|"LISTEN / NOTIFY"| WORKP
WEBP --> ENTRA
CAREERS -->|"HTTPS form post"| WEBP
WORKP -->|"delta poll, Mail.Read"| GRAPH
WORKP -->|"scoring, extraction, summarisation"| AIP
WEBP -->|"assistant streaming only"| AIP
WORKP --> SMTP
WORKP -->|"Phase 3 inbound / Phase 4 outbound"| BOARDS
External dependencies and what happens when each is unavailable
| System | Direction | Owner of the risk | Failure behaviour |
|---|---|---|---|
| Entra ID | Inbound auth (ASSUMPTION — pending confirmation Utopia runs M365) | Corporate IT | Nobody can log in. Break-glass: a small set of local Django superuser accounts with MFA, used only for incident response, every login audited |
| Microsoft Graph | Outbound poll of the recruiting mailbox | Corporate IT — needs an app registration, admin consent for Mail.Read, and a dedicated mailbox |
Polling backs off; the delta cursor is not advanced, so nothing is lost. Channel health goes amber in the intake UI after 2 missed cycles |
| Careers site | Inbound HTTPS form post | Marketing / web | Applications stop arriving from that channel. The endpoint is idempotent on submission hash, so a retrying client cannot double-create |
| AI provider | Outbound HTTPS | Vendor, under a DPA | Circuit breaker opens; scoring jobs park in deferred, not failed; the platform stays fully usable without scores (§9) |
| Object storage | Read/write of CV blobs | Cloud platform | Intake refuses new attachments rather than recording an intake row with a missing blob; existing text-extracted content still renders |
| Job boards | Phase 3 in, Phase 4 out | Vendors | Out of the Phase 1 critical path entirely |
Trust boundaries
- Public internet → web process. Anything arriving here is hostile until proven otherwise: careers-form posts, candidate document uploads, token-gated status pages.
- Corporate identity → session. Authenticated but not authorised; every request still
passes
iam.can(). - Attacker-supplied file → parser. The sharpest boundary in the system. Parser
libraries running over arbitrary PDF and DOCX files are a real RCE and
resource-exhaustion surface. Phase 1 mitigations: hard per-document CPU and wall
timeouts, a memory cap, no outbound network from the parse step. Phase 2 promotes this
to a
worker-untrustedqueue running under a restricted OS user. This is the split trigger most likely to fire first (§10). - Stored raw string → rendered pixel. The database deliberately preserves originals
unsanitised (
address_original,full_name_original,raw_intake.payload,intake_parse_attempt.parsed) so the arrival record is recoverable. That is correct for storage and guarantees attacker-controlled strings reach the rendering layer, so escaping must be an output-side property: JSX escaping in the React app, and the Phase 0UI.esc()pass plus a CSP withoutunsafe-inlineon the prototype (_repo-findings.md§E). - Human actor → AI provider. No data crosses this boundary that the asking human
cannot already see, because
invoke()checks the human actor's permissions before any context is assembled.
4. Logical module catalogue
25 logical modules in five tiers, plus one cross-cutting API layer that is not a module.
Every module lives in the monolith. The Runtime column states which process executes
the module's code: web (request path), worker (queue path), or both.
Dependency rule, enforced by import-linter layered contracts in CI:
surfaces → core domain → platform
intelligence → core domain (READ only) + platform
identity, audit and config are ambient dependencies of every module. Core domain
modules must never import intelligence or surfaces.
Diagram 2 — Application components and module map
graph TD
API["API layer<br/>DRF /api/v1, OpenAPI, auth middleware,<br/>one error envelope, request-id, rate limits"]
subgraph SURFACES["Surfaces tier"]
S_AN["analytics"]
S_AS["assistant"]
S_IN["integrations_inbound"]
S_OU["integrations_outbound"]
S_WL["worklist"]
end
subgraph INTELLIGENCE["Intelligence tier — read-only into domain"]
I_AI["ai_orchestration"]
I_SC["scoring"]
I_FE["fairness_evaluation"]
I_DP["document_parsing"]
end
subgraph DOMAIN["Core domain tier"]
D_IK["intake"]
D_CA["candidate"]
D_DR["duplicate_review"]
D_RQ["requisition"]
D_AP["application"]
D_PL["pipeline"]
D_AS["assignment"]
D_IV["interview"]
D_AM["assessment"]
D_OF["offer"]
D_TP["talent_pool"]
end
subgraph PLATFORM["Platform tier — ambient"]
P_ID["identity"]
P_AU["audit"]
P_FI["files"]
P_CF["config"]
P_NO["notifications"]
end
API --> SURFACES
API --> DOMAIN
SURFACES --> DOMAIN
DOMAIN --> PLATFORM
INTELLIGENCE --> PLATFORM
INTELLIGENCE -.->|"read only — no domain writes"| DOMAIN
D_AP -.->|"accept_suggestion, human actor"| I_SC
classDef workeronly fill:#6b4a9e,stroke:#3b2a5e,color:#fff
classDef webonly fill:#1f6f5c,stroke:#0f3f34,color:#fff
class I_DP,I_FE workeronly
class S_AS,S_AN webonly
Purple = worker-only execution. Green = web-only execution. Everything unmarked runs in
both processes. Note the dotted edges: intelligence reads domain state but never writes it,
and the only path from a suggestion into domain state is a domain module calling
accept_suggestion() with a human actor. That is what makes "AI never auto-rejects" a
property of the dependency graph rather than a policy in a document.
4.1 Platform tier
| # | Module | Responsibility | Main entities | APIs / interfaces | Depends on | Permission boundary | Background tasks | Phase | Runtime |
|---|---|---|---|---|---|---|---|---|---|
| 1 | identity |
Users, roles, scoped role assignments, sessions, SSO, and the single authorization decision point | User, Role, Permission, RoleAssignment (scope: brand / department / requisition), Session |
authenticate(), can(actor, action, resource), scopes_for(user), DRF permission classes; POST /api/v1/auth/*, GET /api/v1/me |
— | Admin-only writes; every user reads own profile | Session and token cleanup | 0–1 | both |
| 2 | audit |
Append-only record of every state change, access event and AI-influenced decision | AuditEvent(actor, actor_kind ∈ user/system/integration/ai_agent, on_behalf_of, action, entity ref, before, after, request_id, ai_run_id, outcome, occurred_at) |
record(), query(); GET /api/v1/audit (admin) |
identity | Read: admin + compliance. No update or delete path exists — grants revoked, trigger raises, hash chain verifies | Nightly hash-chain verification, monthly partition create, 13-month partition detach and archive | 1 | both |
| 3 | files |
Blob storage, checksums, malware scan hook, signed URLs, retention and deletion | StoredFile(sha256, mime, size, storage_key, scan_status, retention_class) |
store(), signed_url(ttl), delete_for_subject() |
identity | Access derived from the owning domain object. No public blob URLs ever | Virus scan, text-extract handoff, retention sweep, orphan-blob reconciliation | 1 | both |
| 4 | config |
Controlled vocabularies, workspace settings, email and letter templates, branding | RefValue, Setting, TemplateVersion |
values_for(vocab), setting(key), render_template(); GET /api/v1/ref/{vocab} |
identity | configure permission only. Django admin is the Phase 1 UI, which is what lets the Settings screen wait until Phase 4 |
Cache warm and invalidate | 1 | both |
| 5 | notifications |
In-app notifications and transactional outbound email to recruiters and candidates | Notification, NotificationPreference, OutboundMessage(status, provider_ref) |
notify(), send_email(); GET /api/v1/notifications |
identity, config, files | Recipient sees own only | Digest send, delivery retry with backoff, bounce and complaint handling | 1 minimal send / 2 full pipeline — Phase 1 gets the OutboundMessage row, send_email() through the MailProvider port, the idempotency guard and NDR classification, because Phase 1 owns inbound mail and an NDR with no row to attach to cannot be classified (04 §9.1 row 5, 07 §17 divergence 6, 08 §7 finding 10). Everything in the Background-tasks column is Phase 2 |
both |
4.2 Core domain tier
| # | Module | Responsibility | Main entities | APIs / interfaces | Depends on | Permission boundary | Background tasks | Phase | Runtime |
|---|---|---|---|---|---|---|---|---|---|
| 6 | intake |
The raw layer. Every inbound submission is recorded and triaged before any candidate exists | IntakeChannel, RawIntake(channel, external_message_id, received_at, payload jsonb, payload_sha256, state), RawIntakeAttachment → StoredFile, IntakeParseAttempt (append-only), IntakeResolution(kind, decision_mode, decided_by, reason) |
ingest(channel, payload) — idempotent on (channel, external_message_id) and on (channel, payload_sha256); queue(), resolve(intake, decision), retry(intake); GET/POST /api/v1/intake |
files, document_parsing, candidate, config | Recruiter triages. Nobody can hard-delete a submission. Terminal states rejected_unusable and quarantined carry no candidate |
Parse dispatch, exponential-backoff retry, stale-queue alert, dead-letter sweep | 1 | both |
| 7 | candidate |
Person identity, independent of any application | Candidate, CandidateEmail, CandidatePhone, CandidateFieldProvenance, CandidateSkill, CandidateDocument, CandidateSearchIndex, consent and retention fields |
create_from_intake(), apply_parsed_profile(), search(query, filters) — Postgres FTS + trigram, erase(subject); GET/PATCH /api/v1/candidates |
files, config | Recruiter and HR read-write. Interviewer sees only candidates on their own assigned interviews | Search-index refresh, retention sweep (pseudonymisation, not deletion), embedding generation from Phase 2 | 1 | both |
| 8 | duplicate_review |
Detect duplicates, mark them, support manual review and a reversible merge | DuplicateCandidatePair(a, b, CHECK a < b, score, signals jsonb, detector_version, matching_config_version_id, state ∈ open/confirmed_duplicate/confirmed_distinct/merged) — confirmed_distinct is the permanent "not a duplicate" memory the detector must consult and skip, not a dismissed queue item; CandidateMerge(performed_by_user_id NOT NULL, reason NOT NULL), CandidateMergeOperation(op_kind, table, previous_value, seq) |
find_candidates(candidate), confirm_distinct(), confirm_merge(), undo_merge() — reverse seq order under strict stack discipline; GET/POST /api/v1/duplicate-pairs |
candidate, audit | Merge and unmerge require HR-admin. Recruiters may only flag | Nightly rescan of new and updated candidates | 1 detect / 2 merge UI | both |
| 9 | requisition |
The versioned job of record, with weighted requirements and an approval chain | Requisition(current_version_id, status), RequisitionVersion (immutable: title, dept, BU, location, employment type, grade, salary range as amount + ISO-4217 currency, description, effective_from, created_by), RequisitionRequirement (version-scoped, weighted), ApprovalStep |
create_draft(), publish_version(), current_version(), version_at(ts); GET/POST /api/v1/requisitions |
config, identity | Create and edit: recruiter + HR. Publish and approve: hiring manager or department head | Deadline sweep, stale-draft sweep | 1 | both |
| 10 | application |
The candidate ↔ requisition-version join. Owns stage and status, and is the unit ATS scores attach to | Application(candidate_id, requisition_version_id, job_id, raw_intake_id NOT NULL, source_channel, current_stage, status, attempt_no, applied_at), ApplicationStageHistory(from, to, actor, actor_kind, reason, ts), SourceAttribution |
apply(), transition(app, to_stage, actor) — refuses terminal-negative transitions unless actor_kind = 'user' (the exact literal; _decisions.md RULING-01 — there is no human value in the enum), accept_suggestion(suggestion, actor), history(); GET/POST/PATCH /api/v1/applications |
candidate, requisition, pipeline, audit | Recruiter or assigned owner writes; hiring manager approves; interviewer read-only | SLA breach detection, cooling-off evaluation | 1 | both |
| 11 | pipeline |
Stage definitions and per-requisition stage configuration. Owns the rules, not the state | StageDefinition, PipelineConfig(requisition_version_id nullable = default), TransitionRule |
stages_for(requisition_version), is_allowed(from, to) |
config | configure permission only — deliberately not the recruiter edit permission that governs stage changes |
— | 1 default / 3 custom | web |
| 12 | assignment |
Flexible, historical recruiter and manager ownership | Two concrete tables, never one polymorphic table. JobAssignment(job_id, user_id, role_id → ref.assignment_role, valid_from, valid_to NULL = current, assigned_by, reason, allocation_pct) and JobApplicationAssignment(job_application_id, …identical shape…). Both carry EXCLUDE USING gist (<subject> WITH =, user_id WITH =, role_id WITH =, tstzrange(valid_from, valid_to) WITH &&) plus a partial unique index (job_id) WHERE valid_to IS NULL AND role_id = primary_recruiter — exactly one current primary recruiter per job. Roles are a reference table (primary_recruiter, supporting_recruiter, sourcer, coordinator, hiring_manager, interviewer), not an inline enum. Denormalised current_primary_recruiter_id on both subjects, trigger-maintained |
One assignment service facade over both tables: assign(), unassign(), transfer(), owners_at(subject, ts), workload(user, window); two REST families GET/POST /api/v1/jobs/{id}/assignments and GET/POST /api/v1/applications/{id}/assignments, plus GET /api/v1/assignments/me. No polymorphic POST /api/v1/assignments taking subject_type |
identity, application, requisition | HR-admin reassigns; recruiters view own workload | Workload rollup | 1 | both |
| 13 | interview |
Scheduling and structured scorecards | Interview(application_id, type, mode, starts_at utc, scheduling_timezone, local_start_wall, status), InterviewParticipant, Scorecard(interviewer, criteria scores, recommendation, submitted_at, locked), ScorecardTemplateVersion |
schedule(), reschedule(), submit_scorecard(); GET/POST /api/v1/interviews |
application, identity, notifications, config | Interviewer sees and scores only own interviews. Scorecards lock on submit | Reminders, calendar sync, no-show sweep, nightly wall-clock reconciliation | 2 | both |
| 14 | assessment |
Assign and record structured tests | AssessmentTemplateVersion, AssessmentAssignment, AssessmentResult |
assign(), record_result() |
application, notifications | Recruiter assigns; results visible to recruiter + hiring manager | Invite send, expiry sweep | 3 | both |
| 15 | offer |
Offer lifecycle with mandatory human approval and issue | Offer(application_id, version chain, status, base amount + currency, components, dates), OfferApprovalStep, OfferStatusHistory, OfferLetterDocument |
draft(), submit_for_approval(), approve(), issue() — human confirmation required, never automated |
application, config, files, notifications, identity | Draft: recruiter. Approve: department head or HR-admin. Issue: never automated | Expiry sweep, approval reminder | 3 | both |
| 16 | talent_pool |
Re-surface previously sourced candidates | Pool, TalentPoolMembership(candidate, pool, reason, added_by), SavedSegment |
add(), rematch(pool, requisition_version) |
candidate, application, scoring | Recruiter-managed | Periodic rematch | 3 | both |
4.3 Intelligence tier
| # | Module | Responsibility | Main entities | APIs / interfaces | Depends on | Permission boundary | Background tasks | Phase | Runtime |
|---|---|---|---|---|---|---|---|---|---|
| 17 | document_parsing |
Turn an attachment into structured, confidence-scored fields | ParsedDocument(file, parser_name, parser_version, text, extracted jsonb, per-field confidence), ParseIssue |
parse(file) -> ParsedDocument |
files, config | Internal. Output is visible wherever the source document is visible | All of it. Runs in worker; Phase 2 moves to the worker-untrusted queue under a restricted OS user with no outbound network |
1 | worker |
| 18 | ai_orchestration |
The only holder of a model-provider client. Capability registry, prompt and model versioning, run ledger, review surface | AiCapability, PromptTemplateVersion, ModelConfigVersion, AiRun(capability, model id + version, prompt version, input_ref, output jsonb, tokens, cost, latency, status), AiSuggestion, AiReview(run, reviewer, verdict, note), AiFeedback |
invoke(capability, context, actor) — actor is always the human, run(id), status(job_id), review(); GET /api/v1/ai/runs, GET /api/v1/ai/jobs/{id} |
identity, audit, config | Invocation is checked against the human actor's permissions. There is no service account. Run ledger readable by admin + compliance | Batch invocation, evaluation runs, cost rollup, circuit-breaker state probe | 1 framework + AI-1/2/3; 2–4 remainder | both — worker for batch, web for the one streaming endpoint |
| 19 | scoring |
Per-application ATS match score, reproducible and explainable | ScoringConfigVersion(weights, feature set, excluded-attribute list, active_from, evaluation_run_id), ApplicationScore(application_id, config_version, model_version, algorithm_code_version, score 0-100, band, computed_at, superseded_by), ScoreComponent(feature, weight_applied, contribution, evidence_ref), SkillMatch(matched[], missing[]) |
score(application), explain(score_id), rescore_batch(requisition_version) |
application (read), requisition (read), document_parsing, ai_orchestration | Score visibility is a configurable role setting (BRD OQ-5, unresolved). Config changes are admin-only and gated on a passing evaluation run | Batch rescore on requisition-version publish or scoring-config activation | 1 | worker |
| 20 | fairness_evaluation |
Disparate-impact evaluation per scoring and model version. Gates activation | EvaluationRun, EvaluationMetric, EvaluationDataset |
evaluate(config_version), results(config_version) |
scoring, audit | Results readable by business and legal, not only engineering | Evaluation batch | 3 — must complete before ranking is released to recruiters | worker |
4.4 Surfaces tier
| # | Module | Responsibility | Main entities | APIs / interfaces | Depends on | Permission boundary | Background tasks | Phase | Runtime |
|---|---|---|---|---|---|---|---|---|---|
| 21 | assistant |
Conversational surface over permitted data. Tool-calls only into existing module service facades | Conversation, Message, ToolInvocation(tool, args, result_ref, actor) |
ask(actor, question, screen_context) — streaming; POST /api/v1/assistant/ask (SSE) |
ai_orchestration, identity, domain service facades | Every tool call carries the asking user's identity. No SQL access in Phase 1 — only whitelisted, typed, parameterised query intents. Text-to-SQL against any application role is prohibited | None in Phase 1 | 2 read-only / 4 full tool use | web |
| 22 | integrations_inbound |
Channel adapters for the 11 inbound channels, all funnelling into intake.ingest() |
ChannelConnection(type, credential_ref, delta_cursor, health), IngestionRun(channel, started, items, failures), DeadLetter |
one fetch() -> [RawSubmission] per adapter; POST /api/v1/inbound/{channel} for push channels |
intake, files, config | Admin configures connections. Credentials live only in the secret store, never in the database | Graph delta polling, webhook receipt, OAuth token refresh, dead-letter retry | 1: Outlook + careers portal + manual upload. 3: 4 job boards. 4: referral / agency / campus / walk-in forms | both |
| 23 | integrations_outbound |
Publish requisition versions to the 8 external platforms | JobPosting(requisition_version, platform, external_id, state, cost_band), PublishAttempt |
publish(), unpublish(), sync_state() |
requisition, config | Publish requires HR-admin — it has cost implications | Publish, state reconciliation | 4 | worker |
| 24 | analytics |
Read-only reporting and dashboard KPIs over materialised read models | SavedReport, ReportRun. Owns read-only SQL views, no domain tables |
kpis(), funnel(), time_to_hire(), source_performance(), recruiter_performance(); GET /api/v1/analytics/* |
read-model views over application, requisition, assignment, offer | Scoped by the viewer's role: recruiter sees own, CEO and department heads see aggregate | Nightly materialised-view refresh | 2 KPIs / 3 report library | web for reads, worker for refresh |
| 25 | worklist |
Recruiter tasks and SLA items, including AI next-best-action suggestions | Task(subject_ref, assignee, due_at, status, origin ∈ manual/rule/ai_suggestion) |
create(), complete(), for_user(); GET /api/v1/worklist/tasks (module-prefixed, per 06 §4.5 — there is no /api/v1/tasks) |
identity, application, notifications | Assignee and their manager | Rule evaluation, overdue sweep | 2 | both |
4.5 The API layer — cross-cutting, not a module
DRF under a versioned /api/v1/ namespace. One auth middleware, one error envelope, one
pagination and filtering convention, request-id propagation into logs and audit rows,
per-user rate limits in Redis, and an OpenAPI schema generated by drf-spectacular from
which the TypeScript client is generated — so the contract is written once. Owned by Talha.
4.6 Modules deliberately not created
| Tempting module | Actual disposition | Why |
|---|---|---|
search |
Postgres FTS + pg_trgm over a trigger-maintained candidate_search_index table, inside candidate |
Sufficient to ~2M candidate rows; §10 has the numeric trigger |
embeddings |
A pgvector column on the candidate profile version, Phase 2 |
Keeping vectors in the same database is what makes "no separate AI service in Phase 1" achievable rather than aspirational |
workflow_engine |
State machines live in the owning module's service | Speculative until three modules demonstrably need the same engine |
calendar (prototype route, FR-16) |
A read view over interview + worklist |
Owns no entities. A Calendar table would duplicate interview state |
managers (FR-15) |
Managers are User rows with a role plus assignment rows |
The prototype keeps a parallel Manager entity with its own name, title and department. Two sources of truth for a person is how permissions drift |
recruiterhub (FR-9) |
assignment (workload, history) + analytics (efficiency, SLA) |
A dashboard over other modules' data |
aistudio (FR-19) |
ai_orchestration admin surface over AiCapability status |
It is an activation surface, not a domain |
settings (FR-22) |
Split: security / users / roles → identity; vocabularies / templates / branding → config |
The current screen is inert chrome (js/settings.js:148-154); splitting puts each setting behind the permission that actually governs it |
help (FR-23) |
Static docs plus client-side search, Phase 4 | No entities, no backend |
reporting_warehouse |
Read-model SQL views inside analytics |
No second datastore at this volume |
tenant / region |
Does not exist | Excluded by constraint |
5. Component boundaries
5.1 The five rules
- One public entry point per module. Every module is a Python package whose only
public surface is
service.py, withdto.pyfor the typed data it exchanges. Cross-module imports may touch<module>.serviceand<module>.dtoand nothing else — nevermodels,views,selectorsortasks. - No module reads or writes another module's tables. The sole exception is
analytics, which owns read-only SQL views declared in migrations. The exception is auditable because it appears in a schema diff. - Core domain never imports intelligence or surfaces. AI output is attached by a domain module accepting a suggestion. This is what makes "AI never auto-rejects" structurally true.
identity,auditandconfigare ambient and may be imported by anything.- Violations fail the build.
import-linterlayered contracts plus a forbidden-import contract. A boundary breach is a red pipeline, not a review argument.
5.2 Code layout
ats/
api/ # DRF routers, versioned URLconf, error envelope, schema
platform_/
identity/ audit/ files/ config/ notifications/
domain/
intake/ candidate/ duplicate_review/ requisition/ application/
pipeline/ assignment/ interview/ assessment/ offer/ talent_pool/
intelligence/
document_parsing/ ai_orchestration/ scoring/ fairness_evaluation/
surfaces/
assistant/ integrations_inbound/ integrations_outbound/ analytics/ worklist/
<each module>/
service.py # THE public facade — the only importable symbol set
dto.py # typed inputs and outputs
models.py # ORM models, mapped to the plain-SQL schema (§12.4)
selectors.py # read queries, module-private
tasks.py # procrastinate task bodies, module-private
views.py # DRF views, module-private
tests/
db/migrations/ # ordered plain-SQL migration files — the schema authority
web/ # Vite + TS + React app
src/styles/styles.css # git mv of css/styles.css, content-frozen
src/lib/charts.js # js/charts.js, retained verbatim
5.3 What the facade guarantees
The facade is not decoration — it is the test seam, the transaction boundary and the authorization checkpoint. Every public facade function:
- takes an explicit
actorand callsiam.can(actor, action, resource)before touching data, so there is exactly one authorization implementation to review; - owns its transaction, using
transaction.atomic()— callers never open transactions across two facades, because a cross-module operation that needs atomicity belongs behind one facade; - returns DTOs, never ORM instances, so a schema change cannot leak across a boundary;
- is type-annotated and checked by mypy. The facade signature plus mypy is the inter-module contract; in-process contract tests would be ceremony.
5.4 Where boundaries deliberately bend
| Bend | Justification | Containment |
|---|---|---|
analytics reads across boundaries |
Reporting queries that respect module boundaries would be N round trips or a duplicated warehouse | Read-only SQL views declared in migrations, reviewable in a diff, never writes |
assistant may call many domain facades |
It is a surface over the whole product | Every call carries the asking human's identity and goes through the same can() the REST API uses |
| History and audit rows are written by database triggers, not by module code | A code path that forgets to write history is invisible; a trigger cannot forget | Actor and reason reach the trigger via transaction-local SET LOCAL app.actor_user_id. Named risk: any path that omits the SET LOCAL still writes history but attributes it to system with actor_unknown = true. That count is an alarmed data-quality metric (§11) |
6. Deployment boundaries
Diagram 3 — Deployment topology, per environment
graph TB
subgraph EDGE["Platform edge"]
CDN["CDN + TLS + WAF<br/>serves the built React bundle"]
end
subgraph APPS["Managed container platform — Azure Container Apps, ASSUMPTION"]
subgraph IMG["ONE container image, two revisions"]
WEB["revision: web<br/>entrypoint uvicorn<br/>2 vCPU / 4 GB<br/>2 workers x 4 threads<br/>autoscale 1-4 replicas"]
WRK["revision: worker<br/>entrypoint procrastinate<br/>2 vCPU / 4 GB<br/>concurrency 4<br/>autoscale 1-3 replicas"]
end
end
subgraph DATA["Managed stateful services"]
PG[("PostgreSQL 16 Flexible Server<br/>2 vCPU / 8 GB<br/>PITR, 14-day backups<br/>pg_trgm, unaccent, btree_gist, pgcrypto<br/>schemas: app ref audit ai staging")]
BLOB[("Blob storage<br/>CV and letter blobs<br/>per-container policy — see ADR 0003")]
REDIS[("Redis<br/>cache, rate limit, sessions<br/>NEVER a broker")]
KV[("Key Vault<br/>Graph creds, AI key, DB password")]
IMM[("Immutable blob container<br/>closed audit partitions,<br/>write-once")]
end
subgraph OBS["Observability"]
LOGS["Log workspace<br/>structured JSON, request-id correlated"]
ERR["Error tracking<br/>Sentry, self-hosted or EU region"]
UPT["External uptime probe<br/>hits /healthz from outside the platform"]
end
CDN --> WEB
WEB --> PG
WEB --> REDIS
WEB --> BLOB
WEB --> KV
WRK --> PG
WRK --> BLOB
WRK --> KV
PG -.->|"LISTEN / NOTIFY wakes the consumer"| WRK
PG -->|"nightly closed-partition export"| IMM
WEB --> LOGS
WRK --> LOGS
WEB --> ERR
WRK --> ERR
UPT --> CDN
6.1 Process boundaries and why there are exactly two
| Boundary | What it separates | Why it is a real boundary, not taxonomy |
|---|---|---|
web / worker |
Sub-second I/O-bound request handling from multi-second CPU-bound parsing, model calls, batch rescoring and sweeps | Different resource profile, different failure mode, different timeout budget. A scanned CV must never occupy a request thread |
worker-default / worker-untrusted (Phase 2) |
Trusted background work from parsing attacker-supplied files | Security, not scale. Same image, same codebase, different queue and different OS user: restricted user, no outbound network, hard CPU and wall timeout, memory cap |
| Web request path / streaming assistant path | Standard request/response from a long-lived SSE connection | The streaming endpoint is the one async Django view under uvicorn; everything else is sync |
No third deployable exists in Phases 0–4. It would duplicate the data model, the auth layer and the deploy surface for two developers, and every requirement it would serve (documented versioned API, graceful degradation, retrievable async job status) is already met by an internal module boundary plus a circuit breaker and a job-status endpoint.
6.2 Environments
| Environment | Composition | Data | Integrations | Promote |
|---|---|---|---|---|
| local | docker compose: web, worker, postgres:16, redis, MinIO (S3-compatible) |
Anonymised fixture set generated by a script that reads pii_classification; never a production copy |
Graph and AI provider mocked by default, real credentials opt-in via .env.local (no .env exists in the repo today, _repo-findings.md §B) |
— |
| staging | Same image, one replica each, smaller database tier | Anonymised fixtures plus real test-mailbox traffic | Real integrations pointed at a dedicated test mailbox and sandbox job-board accounts | Auto-deploy on merge to main |
| production | Web 1–4 replicas, worker 1–3, PITR enabled | Real | Real | Manual promote, Talha only |
No per-developer cloud environment. Two developers do not need six environments; they need one that behaves like production.
6.3 Region and residency
One region for the single database. Which region is a legal decision, not an architectural one — postings span six jurisdictions (BRD OQ-4, unresolved). Retention and deletion are implemented per record, including derived embeddings, not per region. If legal requires in-jurisdiction storage, that conflicts directly with the no-per-region-databases constraint and requires an explicit business exception; it is not solvable with topology.
7. Data flow
7.1 The common write path
Every state-changing request follows the same shape, which is why it is worth stating once:
- Edge terminates TLS; the web process assigns a
request_id. - Auth middleware resolves the session to a
User;SET LOCAL app.actor_user_idis issued on the connection so history triggers can attribute the change. - The DRF view validates input against a serializer and delegates to exactly one module facade.
- The facade calls
iam.can(actor, action, resource). Denial is a 403 and anaudit_eventwithoutcome = 'denied'and a denial reason — denials are as interesting as grants. - Inside
transaction.atomic(): domain writes, history rows written by trigger, and any background job enqueued into the same PostgreSQL. Commit is atomic across the domain row and its job. audit.record()writes the append-only event withrequest_idand, if applicable,ai_run_id.- Response returns the public UUIDv7
public_id, never a bigint primary key.
Diagram 4 — Email application flow (Outlook, inbound channel #1)
sequenceDiagram
autonumber
participant GR as Microsoft Graph
participant AD as inbound adapter on worker
participant IK as intake
participant FI as files
participant DP as document_parsing on worker
participant DR as duplicate_review
participant CA as candidate
participant AP as application
participant SC as scoring on worker
participant WL as worklist
participant AU as audit
Note over AD: periodic task on the mail queue, every 2 minutes
AD->>GR: delta query on the recruiting mailbox
GR-->>AD: messages plus attachments since the stored cursor
loop per message
AD->>IK: ingest channel=outlook, external_message_id, payload
Note over IK: UNIQUE on channel plus external_message_id makes redelivery a no-op
IK->>FI: store each attachment, sha256, virus scan pending
FI-->>IK: StoredFile refs
IK->>IK: RawIntake state=received, enqueue parse in the SAME transaction
IK->>AU: record intake received
end
AD->>AD: advance delta cursor ONLY after all messages commit
DP->>FI: fetch blob by storage key
DP->>DP: extract text and fields, per-field confidence, hard timeout
DP-->>IK: IntakeParseAttempt appended, status succeeded partial or failed
alt parse failed or confidence below threshold
IK->>IK: state=needs_review
IK->>WL: task for the intake triage queue
Note over IK: terminal without a candidate is a representable state
else parse usable
IK->>DR: find candidates by normalised email, trigram name, phone
alt confident single match
DR-->>IK: existing candidate
IK->>CA: apply_parsed_profile writes candidate_field_provenance rows per field, source parse attempt pinned
else no match
IK->>CA: create_from_intake, created_from_raw_intake_id NOT NULL
else ambiguous
DR->>DR: DuplicateCandidatePair state=open
IK->>IK: state=needs_review, human decides
end
IK->>AP: apply candidate, current requisition_version, raw_intake_id
Note over AP: uq_application_live blocks a second live application for the same job
AP->>SC: enqueue score for this application
SC->>SC: pin config version, model version, parse attempt, job version
SC-->>AP: ApplicationScore plus ScoreComponent rows, append-only
AP->>AU: record application created and scored with ai_run_id
AP->>WL: recruiter task new scored application
end
Load-bearing properties. The delta cursor advances only after every message in the
batch has committed, so a crash mid-batch replays rather than skips. Idempotency lives in a
database unique constraint, not in adapter code. A parse failure produces a reviewable
terminal state, not a silent drop and not a ghost candidate — which is exactly the state
the prototype's pre-resolved inbox cannot represent (js/data.js:284-300). Nothing on this
path can auto-reject: the only transitions taken are forward and neutral.
Diagram 5 — Website application flow (careers portal)
sequenceDiagram
autonumber
participant CD as Candidate browser
participant CDN as Edge
participant WEB as web process
participant IK as intake
participant FI as files
participant Q as queue in Postgres
participant DP as document_parsing on worker
participant NO as notifications
CD->>CDN: GET careers form for a published requisition_version
CDN->>WEB: unauthenticated read, rate limited per IP
WEB-->>CD: form rendered from the PUBLISHED version only
CD->>WEB: POST application, fields plus CV file
WEB->>WEB: rate limit, size and MIME allowlist, no rich text accepted
WEB->>FI: store blob, sha256 computed, scan_status=pending
WEB->>IK: ingest channel=careers_portal, external_ref=submission_uuid
Note over IK: UNIQUE on channel plus payload_sha256 makes a double submit idempotent
IK->>Q: enqueue parse, SAME transaction as the RawIntake insert
WEB-->>CD: 202 Accepted plus a reference code, NO score and NO status shown
Note over CD,WEB: the candidate is never told a score exists
Q-->>DP: LISTEN NOTIFY wakes the consumer
DP->>DP: parse, then the same resolution path as diagram 4
IK->>NO: acknowledgement email, template from config
Note over NO: a candidate-facing status page is gated by candidate_access_token, hashed and expiring, never by public_id
Why 202 and not 201. The response cannot wait on parsing, and the candidate must not
learn anything about internal state. The reference code is internal-shaped
(APP-30001-style) but the candidate-facing link uses the UUIDv7 public_id behind an
expiring hashed token — public_id is an identifier, never a capability.
Diagram 6 — Manual upload flow (recruiter, and bulk import)
sequenceDiagram
autonumber
participant RC as Recruiter
participant WEB as web process
participant ID as identity
participant FI as files
participant IK as intake
participant DP as document_parsing on worker
participant CA as candidate
participant AP as application
participant AU as audit
RC->>WEB: upload one or many CVs, optional target requisition
WEB->>ID: can actor=recruiter action=intake.create
ID-->>WEB: allow, scoped to the recruiter's brands and departments
loop per file
WEB->>FI: store blob, sha256, scan_status=pending
WEB->>IK: ingest channel=manual_ui, external_ref=upload_batch plus index
Note over IK: manual entry is NOT an exception, a raw_intake row is created first
IK->>DP: enqueue parse, same transaction
end
WEB-->>RC: batch accepted, N queued, live progress by polling the job status endpoint
DP-->>IK: IntakeParseAttempt per file
IK-->>WEB: triage queue populated
RC->>WEB: review a parsed record, correct low-confidence fields
Note over RC,WEB: low-confidence fields are left EMPTY, never guessed
RC->>IK: resolve kind=create_candidate, decision_mode=human
IK->>CA: create_from_intake
CA-->>IK: candidate public_id
opt a target requisition was chosen
IK->>AP: apply, raw_intake_id carried through
end
IK->>AU: record resolution with actor, decision_mode and reason
Why manual entry goes through intake. Exempting it would create a second, unaudited
path to candidate creation and would make "raw intake before candidate" a convention rather
than a database fact. The cost is one extra row per manual entry. The benefit is that
created_from_raw_intake_id can be NOT NULL with no exceptions, which is what makes the
guarantee real.
Diagram 7 — AI chatbot flow
sequenceDiagram
autonumber
participant U as Recruiter
participant WEB as web process, async view
participant AS as assistant
participant ID as identity
participant AI as ai_orchestration
participant SVC as domain service facades
participant PR as AI provider
participant AU as audit
U->>WEB: POST assistant ask, question plus screen_context, SSE opened
WEB->>AS: ask actor=U, question, context
AS->>ID: can actor=U action=assistant.use
ID-->>AS: allow
AS->>AI: invoke capability=assistant_answer, actor=U
Note over AI: the HUMAN actor is passed, there is no service account and no system principal
AI->>AI: resolve PromptTemplateVersion and ModelConfigVersion
AI->>AI: open AiRun row BEFORE the call, status=running
AI->>PR: model call with the whitelisted tool schema, streaming
loop tool calls requested by the model
PR-->>AI: tool call, one of a whitelisted set of typed query intents
AI->>ID: can actor=U action=read resource=the requested subject
alt denied
ID-->>AI: deny
AI-->>PR: tool result denied, no data returned
AI->>AU: audit_event outcome=denied, actor_kind=ai_agent, on_behalf_of=U
else allowed
AI->>SVC: the SAME facade the REST API calls, parameterised, never SQL
SVC-->>AI: DTO, already scoped to U
AI->>AU: access event actor_kind=ai_agent, on_behalf_of=U, entity ref
end
end
PR-->>AI: final tokens streamed
AI->>AI: close AiRun, tokens, cost, latency, status=succeeded
AI-->>AS: answer stream plus ai_run_id
AS-->>WEB: SSE chunks
WEB-->>U: answer rendered with a provenance badge linking to the AiRun
Note over U,AI: the assistant can suggest an action but cannot perform one. Any state change is a separate authorised UI action by U
Load-bearing properties. Four things are true by construction, not by prompt:
- The assistant has no capability the UI does not have — same facades, same
can(), so there is one authorization implementation to review rather than two. - There is no SQL access in Phase 1. Text-to-SQL against any application role is
prohibited; ad-hoc querying, if ever required, arrives in Phase 2 as a dedicated
PostgreSQL role with row-level security keyed to
current_setting('app.actor_user_id')and column privileges excludingsensitive_personalcolumns — enforcement by the database, not by prompt engineering. - Every answer is attributable: an
AiRunrow exists before the result is usable, and the UI links to it. - The assistant cannot write.
surfaces → intelligencemay not reach domain writes, and any suggestion becomes state only when the human takes the action.
8. Background workers
8.1 Queue conventions
Postgres-backed procrastinate, six named queues. Redis is never a broker.
| Queue | Work | Concurrency | Priority | Notes |
|---|---|---|---|---|
ingest |
Normalise and land a raw submission | 4 | high | Must never starve — this is the "nothing is silently lost" path |
parse |
document_parsing.parse() |
2 | normal | CPU-bound. Phase 2: splits into parse-untrusted on the restricted worker |
score |
scoring.score(), rescore_batch() |
2 | normal | Queueing lock per requisition version so a rescore batch cannot interleave with itself |
ai |
Non-interactive model invocations | 2 | normal | Subject to the circuit breaker |
mail |
Graph delta poll, outbound send, bounce handling | 1 | normal | Serialised per channel connection by a queueing lock, so two pollers cannot both advance the cursor |
maintenance |
Sweeps, refreshes, verifications, retention | 1 | low | Windowed to off-peak where the six-jurisdiction spread allows |
Every task body: idempotent, keyed on the domain row, safe to run twice. Every task
declares max_attempts and a backoff. There is no silent drop — the terminal state is
failed, and failed surfaces in the intake UI as a reviewable item with the error.
8.2 Task catalogue
| Task | Queue | Trigger | Idempotency key | Retry | On final failure |
|---|---|---|---|---|---|
poll_mail_channel |
periodic, 2 min | channel connection lock | 5, exponential to 30 min | Channel health amber, alert; cursor not advanced | |
land_submission |
ingest | enqueued by adapter or web POST | (channel, external_message_id) unique index |
5 | Dead letter row, admin-visible |
virus_scan |
ingest | on StoredFile insert |
sha256 |
3 | Blob quarantined, intake quarantined (terminal, no candidate) |
parse_document |
parse | on intake landing, or manual retry | (file_id, parser_version) |
3, then needs_review |
New IntakeParseAttempt with status=failed; intake to needs_review |
detect_duplicates |
score | on candidate create or profile update, plus nightly rescan | candidate_id + version |
3 | Link left absent, alert; never auto-merges |
score_application |
score | on application create, requisition-version publish, scoring-config activation | (application_id, config_version) |
3, deferred while breaker open | Application visible without a score; badge reads "not yet scored" |
rescore_requisition_batch |
score | on publish or config activation | requisition-version queueing lock | 2 | Partial batch is safe — scores are append-only with superseded_by |
evaluate_fairness |
ai | on scoring-config draft | config_version |
2 | Config cannot activate — the gate holds closed |
refresh_search_index |
maintenance | trigger-driven plus nightly full pass | candidate_id |
3 | Stale search results, alerted |
refresh_read_models |
maintenance | nightly | view name | 2 | Dashboard shows a staleness timestamp rather than wrong-looking numbers |
verify_audit_hash_chain |
maintenance | nightly | partition | 1 | Page immediately — a break means tampering or corruption |
create_audit_partition |
maintenance | monthly, and 2 months ahead | partition name | 3 | Alert before it can cause insert failures |
export_closed_audit_partition |
maintenance | monthly | partition name | 3 | Alert; immutable off-box copy is the only independent tamper check |
retention_purge |
maintenance | nightly | (policy, subject) |
1 | Alert. Skips retention_hold rows and candidates in an unreversed merge |
sla_sweep |
maintenance | hourly | application id | 3 | Worklist item not created; alerted |
send_notification |
on notify() |
OutboundMessage.id |
5 | OutboundMessage status=failed, visible to the sender |
|
refresh_oauth_token |
10 min before expiry | connection id | 5 | Channel health red, alert | |
retry_dead_letters |
ingest | hourly | dead letter id | 1 per cycle, 24 cycles | Stays in dead letter for manual action |
8.3 Ordering and concurrency hazards addressed explicitly
- Per-candidate dedupe and per-requisition rescore use procrastinate queueing locks
declaratively. Without them, two concurrent scans create mirrored
DuplicateCandidatePairrows and two rescore batches interleave into inconsistentsuperseded_bychains. - Mail cursor advance is serialised per connection and happens only after commit.
- Scores are never updated in place. A rescore inserts and marks the predecessor superseded, so a partially completed batch is a consistent state rather than a corrupt one.
- Deploy-time drain. The worker gets
SIGTERM, stops accepting, finishes in-flight tasks up to a 120 s grace period, then exits. Because jobs live in Postgres, anything not finished is simply still queued.
9. Failure handling
9.1 Failure modes
| Failure | Detection | Behaviour | Recovery | User-visible effect |
|---|---|---|---|---|
| AI provider down, throttled or slow | Error rate and p95 per capability; circuit breaker in ai_orchestration |
Breaker opens after 5 failures in 60 s; score and ai jobs move to deferred, not failed; interactive assistant returns an explicit "AI unavailable" state |
Half-open probe every 60 s; deferred jobs resume automatically | Everything works except AI features. No scores appear; no score is wrong |
| Worker crash or OOM | Container restart count, task heartbeat | In-flight tasks are unacknowledged and re-run — which is why idempotency is mandatory | Automatic restart; three crash-loops in an hour pages | Delayed parsing and scoring. No data loss |
| Poison document — a PDF that hangs or exhausts the parser | Per-document wall and CPU timeout, memory cap | Attempt recorded as failed with the error; after max_attempts the intake goes to needs_review |
Recruiter reviews and can key manually; parser fix replays historic intake via a new parser_version |
One document needs review. This is the trigger most likely to promote parsing to its own deployable (§10) |
| PostgreSQL failover on the managed service | Connection errors; readiness probe fails | Web returns 503 with a retry hint; connection pool re-establishes; queue is unaffected because it is in the same database | Managed failover, typically 60–120 s | A short outage. No lost writes — anything uncommitted was never acknowledged |
| Object storage unavailable | files.store() errors |
Intake refuses the attachment rather than recording an intake row pointing at a blob that does not exist | Retry with backoff; alert | Uploads fail loudly and are retried by the user or the adapter |
| Graph token expired or consent revoked | Token refresh failure | Channel health red; polling stops; cursor frozen | Corporate IT re-consents; polling resumes from the frozen cursor | Email intake pauses; nothing is lost |
| Queue backlog | queue_depth, time_to_start p95 per queue |
Autoscale worker replicas up to 3 | If p95 time-to-start for interactive AI exceeds 10 s while web p95 stays under 300 ms, that is a soft split trigger (§10) | Slower parsing and scoring |
| Careers-form abuse or scripted spam | Per-IP rate limit in Redis; submission-hash duplicates | Rate limit returns 429; duplicate hashes are idempotent no-ops | Tighten limits; add a challenge only if it becomes real | Legitimate applicants unaffected |
| Malformed email that cannot become a candidate | candidate_email CHECK regex fails; the deferrable contactability trigger raises at COMMIT |
The whole transaction aborts. Intake stays needs_review. No ghost candidate is created |
Recruiter corrects and resolves manually | One item in the triage queue |
| Audit hash-chain break | Nightly verifier | Immediate page. The break is evidence, not prevention — layers 1 and 2 (revoked grants, raising trigger) are the prevention, and only the off-box immutable export is an independent check | Forensic comparison against the immutable export | None directly; a compliance incident |
History rows attributed to system because SET LOCAL was missing |
actor_unknown count metric |
Treated as a data-quality alarm, not noise | Fix the code path; back-attribution is generally impossible | Attribution gaps in history — invisible without this alarm, which is why it exists |
| Migration fails mid-deploy | Deploy step exit code | Deploy aborts before the new revision takes traffic; the old revision keeps serving the old schema | Forward-fix migration, plus PITR if data was touched. No down-migrations exist — §12.4 | None if the expand/contract discipline was followed |
9.2 Degradation matrix
The design goal is that the platform is useful with AI entirely absent (BRD NFR-7).
| Component down | Still works | Stops |
|---|---|---|
| AI provider | Login, intake landing, manual triage, candidate and requisition CRUD, applications, stages, assignment, interviews, analytics | Scoring, AI extraction assistance, assistant, next-best-action suggestions |
| Worker process | All read and write UI operations; intake landing (the row and its job both commit) | Parsing, scoring, mail polling, notifications, sweeps — all resume from the queue when the worker returns |
| Redis | Everything, more slowly; sessions fall back to the database-backed backend | Rate limiting degrades to a database counter; cache misses become queries |
| Object storage | Browsing all existing records and text already extracted | New uploads, CV downloads, letter generation |
| Graph | Careers portal, manual upload, all internal work | Email channel intake |
| PostgreSQL | Nothing | Everything. It is the single point of failure, accepted deliberately, mitigated by managed HA-capable hosting and PITR |
10. Scalability
10.1 Current sizing and headroom
| Dimension | Phase 1 projection | Basis | Headroom on the recommended tier |
|---|---|---|---|
| Named seats | 66 | BRD §4 (2+4+15+12+8+24+1) | — |
| Peak concurrent users | 20–25 | ASSUMPTION — 24 of the 66 are interviewers who touch only their own interviews | One 2-vCPU web replica is generously provisioned; autoscale to 4 |
| Applications per year | 20k–60k | ASSUMPTION | Trivial for PostgreSQL |
| Documents per day at peak | 200–600 | ASSUMPTION | One 2-vCPU worker with concurrency 4 handles this with room; at 30 s per scanned CV, 600 documents is roughly 75 worker-minutes spread over a day |
| Blob volume, year one | well under 1 TB | ASSUMPTION | — |
| Candidate rows, several years | 10^4–10^5 | ASSUMPTION | Three to four orders of magnitude below the search-extraction trigger |
| Queue throughput | hundreds of jobs/hour | derived from the above | Postgres LISTEN/NOTIFY queues handle thousands/hour comfortably |
10.2 The scaling ladder — exhaust in order
- Query and index tuning. Most "we need X" moments are an unindexed query.
- Vertical scale. Web and worker to 4 vCPU / 8 GB; PostgreSQL to 4 vCPU / 16 GB.
- Horizontal scale of the web revision. Stateless behind the platform's load balancer; sessions are in Redis or the database.
- Horizontal scale of the worker revision, with queueing locks preventing the duplicate-work hazards §8.3 names.
- Split the worker by queue. Already planned for Phase 2 as
worker-untrusted, on a security argument rather than a scale one. - A read replica for analytics and search.
- Only then extract a service.
10.3 Concrete triggers that would justify extracting a service
Reviewed quarterly. Any single hard trigger justifies a separate deployable; soft
triggers need two sustained for 2+ weeks. These are the same triggers as _decisions.md
Part 1, restated with the metric that measures each.
| # | Trigger | Threshold | Metric that fires it | Hard? | Extract what |
|---|---|---|---|---|---|
| T1 | Dependency conflict | An ML or OCR dependency cannot coexist in the web image, or pushes it past ~2 GB | Image size in CI | Hard | Parsing/AI worker |
| T2 | Hardware profile divergence | Parsing or inference needs a GPU, or sustained >4 vCPU / >8 GB | Worker CPU and RSS p95 | Hard | Parsing/AI worker |
| T3 | Runtime isolation | Untrusted-file handling needs a sandbox the worker process cannot provide, beyond the Phase 2 restricted queue | A parser CVE, or a confirmed sandbox escape in review | Hard | Parsing service with a hardened runtime |
| T4 | Non-Python runtime | A required model runtime is not Python | — | Hard | Inference service |
| T5 | Queue starvation | p95 time-to-start for interactive AI tasks >10 s while web p95 <300 ms, after vertical scaling of the worker host is exhausted | queue_time_to_start_p95{queue="ai"} vs http_request_duration_p95 |
Soft | AI worker |
| T6 | Release cadence conflict | Prompt or model changes need >2 deploys/week while domain code is release-gated | Deploy log | Soft | AI orchestration service |
| T7 | Blast radius | Worker OOM or crash-loops have taken the shared host down twice in a quarter | Container restart count and incident log | Soft | Worker |
| T8 | Search outgrown | Candidate rows >~2,000,000 or indexed text >~50 GB or p95 search >500 ms after tuning and after moving search to a read replica or sustained >50 queries/s degrading write latency | search_latency_p95, table sizes |
Hard on (a); soft on the rest | Search service — but exhaust tuning, replica, materialised table and a BM25 extension first |
| T9 | Queue technology outgrown | Sustained >20 jobs/second, a need for chord/fan-in semantics, or worker count >4 | jobs_completed_rate, replica count |
Soft | Migrate to Celery + Redis with a transactional outbox, to preserve the enqueue guarantee |
Explicitly never a trigger: "AI feels like a different concern", org-chart preference, résumé-driven development, or multi-tenant / regional residency (excluded by constraint).
Honest projection. On the labelled volume assumptions, T8(a) sits three to four orders of magnitude away, so PostgreSQL FTS plus trigram will very likely never be outgrown by this system, and a read replica is the realistic ceiling of what will ever be needed. T3 is the trigger most likely to fire early, and it is a security argument, not a scale one.
11. Monitoring
11.1 Service level objectives
| SLO | Target | Measured |
|---|---|---|
| API availability, business hours | 99.5% monthly | External uptime probe against /healthz |
| Read endpoint p95 | < 300 ms | Server-side histogram, excluding the streaming endpoint |
| Write endpoint p95 | < 600 ms | Same |
| Intake landing to parse start, p95 | < 60 s | received_at to first IntakeParseAttempt.started_at |
| Parse completion, p95 | < 5 min for a text PDF, < 15 min for a scanned document | Attempt duration by document class |
| Application created to score available, p95 | < 10 min | Application applied_at to ApplicationScore.computed_at |
| Documents lost | zero, always | Reconciliation: Graph message count vs raw_intake count per channel per day |
11.2 Instrumentation
Metrics — OpenTelemetry to the platform's metric store:
http_requests_total{route,method,status},http_request_duration_seconds{route}queue_depth{queue},queue_time_to_start_seconds{queue},task_duration_seconds{task},task_failures_total{task,attempt},dead_letter_total{channel}intake_landed_total{channel},intake_state_total{state},parse_confidence_bucket{field},parse_failures_total{parser_version,reason}ai_runs_total{capability,status},ai_tokens_total{capability},ai_cost_total{capability},ai_latency_seconds{capability},ai_breaker_state{provider}authz_denials_total{action,role}— a spike is either a permission bug or an attackhistory_actor_unknown_total— theSET LOCALgap alarm from §5.4db_connections_in_use,db_slow_queries_total,pg_table_bytes{table}
Logs — structured JSON, one line per request and per task, always carrying
request_id, actor_id, module, outcome. Never the value of a
sensitive_personal column. Log volume is not a compliance record — audit.audit_event is.
Traces — sampled at 10%, always-on for the intake and scoring paths, because those are the multi-hop flows where "where did the 40 seconds go" is a real question.
Business dashboards — built by Ahmed on the retained js/charts.js engine: intake
funnel by channel and state, triage queue age, parse confidence distribution, score band
distribution, time-to-hire, recruiter workload. These double as operational signals: a
sudden shift in the score-band distribution usually means a model or config change, not a
change in candidate quality.
11.3 Alerts
| Alert | Condition | Severity | Why it is worth waking someone |
|---|---|---|---|
| Audit hash-chain break | Nightly verifier fails | Page | Tampering or corruption of the compliance record |
| Intake reconciliation mismatch | Graph message count ≠ landed count for a channel-day | Page | Violates the one guarantee that must never break |
| Web 5xx rate | >2% over 5 min | Page | — |
| Database connections | >80% of the pool for 10 min | Page | Precedes total unavailability |
| Queue starvation | time_to_start_p95{ai} >10 s for 15 min |
Ticket | Soft split trigger T5 |
| Parse failure rate | >20% of attempts over 1 h | Ticket | Parser regression or a new document format |
| AI breaker open | Open >15 min | Ticket | Degradation is working; the vendor needs chasing |
| AI cost | Daily spend >2× the 7-day mean | Ticket | Prompt bug or a runaway batch |
| Channel health | Any connection red >30 min | Ticket | Intake pause |
actor_unknown history rows |
Any increase day over day | Ticket | An unattributed write path exists |
| Retention purge failure | Nightly job failed | Ticket | Legal obligation |
| Disk or table growth | audit_event partition >2× projection |
Ticket | Sizing assumption is wrong (§14) |
Error tracking — Sentry, self-hosted or in an EU region, with a scrubbing config
derived from the pii_classification table so no candidate PII enters a third-party error
store. Release-tagged, so a regression is attributable to a deploy in one click.
12. Deployment recommendation
Recommended cloud: Azure — Container Apps, Database for PostgreSQL Flexible Server, Blob Storage, Key Vault, Entra ID. ASSUMPTION: Utopia Brands runs Microsoft 365, since Outlook is inbound channel #1. If true, SSO and the Graph app registration land in the same tenant and the identity problem largely disappears — the single largest free reduction in integration risk available. If false, the architecture is unchanged and the equivalent AWS or GCP services substitute directly.
12.1 Local development
docker compose up # web, worker, postgres:16, redis, minio
make migrate # applies db/migrations/*.sql in order
make seed # anonymised fixtures generated from pii_classification
cd web && npm run dev # Vite dev server, proxying /api to the web container
Rules: never a production data copy — the fixture generator reads pii_classification and
applies each column's masking_strategy. Graph and the AI provider are mocked by default;
real credentials are opt-in through .env.local, which is gitignored (the repository has no
.env today, _repo-findings.md §B). The retained devserver.py stays only for viewing
the frozen prototype.
12.2 CI pipeline — one required workflow, GitHub Actions
.github/ does not exist today (_repo-findings.md §B), so this is entirely additive.
| Stage | Gate | Fails the build when |
|---|---|---|
| Lint | ruff, mypy --strict on facades |
Style or type error |
| Boundaries | import-linter layered + forbidden contracts |
A module imports another module's models, or core domain imports intelligence |
| Schema — gate 1 (ORM) | makemigrations --check --dry-run: models vs declared migration state (§12.4) |
A model changed without a migration declaring the same state |
| Schema — gate 2 (SQL) | Plain-SQL migrations applied to a scratch database, then pg_dump --schema-only --no-owner plus the trigger/column-privilege catalogue query, diffed against the committed expected dump (§12.4) |
Any database object exists that no reviewed migration accounts for — the gate that covers everything the ORM cannot see |
| Backend tests | pytest against a real PostgreSQL service container |
Any failure. Never SQLite — the design depends on jsonb, partial and expression unique indexes, FTS, pg_trgm, deferrable constraint triggers, GiST EXCLUDE and LISTEN/NOTIFY |
| Constraint tests | Every trigger and constraint has a test that attempts the forbidden write and asserts the exception | A guard silently stops guarding |
| PII registry | Every column on a candidate-touching table has a pii_classification row |
A new PII column arrives unclassified |
| Frontend | tsc --noEmit, ESLint with react/no-danger as error, stylelint enforcing var(--…) token usage, Vitest |
A dangerouslySetInnerHTML, or a hardcoded colour |
| Prototype guard | Grep gate failing on a new unescaped ${ inside an HTML template literal |
Someone reintroduces the §E exposure |
| Docs — evidence citations | python3 tools/check_evidence_citations.py (stdlib only, no dependencies). Checks every path:line citation in docs/architecture/**/*.md at three levels: the file exists; every cited line is in range; the ~30 load-bearing anchors still match an expected pattern; and no sentence cites an anchor belonging to a different claim |
A citation goes stale because js/*.js moved, or a claim is attached to the wrong line. This package's argument is "verified by direct inspection", so a citation a reviewer can spot-check and find wrong costs more credibility than the fact is worth. Runs in the same job as the other documentation gates because it needs no services and finishes in under a second |
| E2E | 5 Playwright journeys: login; ingest a CV and see it parsed; promote a submission to candidate; create and publish a requisition version; move an application through two stages | Any failure |
| Image | Build, scan, size check against the T1 threshold | Vulnerabilities, or image >2 GB |
Branch protection: no direct pushes to main; every PR requires Talha's review. That is the
mandated review checkpoint for the junior's work, enforced by the platform rather than by
habit.
12.3 Promotion
graph LR
PR["Pull request"] --> CI["CI: all gates"]
CI --> MERGE["Merge to main"]
MERGE --> IMG["Build one image, tagged with the commit sha"]
IMG --> STG["Auto-deploy: staging<br/>migrate then roll web then roll worker"]
STG --> SMOKE["Playwright smoke against staging"]
SMOKE --> GATE{"Manual promote<br/>Talha"}
GATE -->|"approve"| PRD["Production<br/>same image, same order"]
GATE -->|"reject"| STOP["Stop, fix forward"]
Deploy order is always: migrate → roll web → roll worker. The worker rolls last
because it must never process a job written by code that assumes a schema the worker does
not yet have.
12.4 Migrations
The schema authority is db/migrations/*.sql — ordered, up-only, plain SQL. The ORM
maps to that schema; it never generates it. The invariants in this design live in objects
Django's migration autogeneration would never write, in two distinct groups that matter for
what follows. Group A — partial unique indexes with WHERE clauses, expression indexes on
lower(), CHECK constraints with regexes, GiST EXCLUDE constraints — Django cannot
autogenerate but can declare, in Meta.constraints and Meta.indexes. Group B — plpgsql
triggers, DEFERRABLE INITIALLY DEFERRED constraint triggers, generated columns, RANGE
partitions, and the column-level GRANTs that make ats_result and audit_event append-only
— Django cannot express at all, at any layer. The distinction is the whole basis of the ruling:
Group A stays visible to Django's own drift check, Group B needs a different check entirely.
Reconciling this with the Django choice — flagged as a cross-document dependency in
_decisions.md Part 2 (risk on line 463) and resolved here decisively.
The mechanism below is the canonical text. It is the ruling ADR 0017 adopts, and
03-database-design.md §32.1, 04-integrations-and-processing.md §9.1 #2,
05-security-rbac-ai-governance.md §9.1 I-1 and 07-implementation-plan.md §17 #1 quote it
verbatim rather than paraphrasing it — four near-identical paraphrases is how the
managed flag ended up pointing in two opposite directions across five documents. The
decision is a mechanism, not a flag setting:
Migration authority ruling — canonical text (ADR 0017). Quote it; do not paraphrase it.
db/migrations/NNN_*.sqlis the schema authority. Every Django migration isSeparateDatabaseAndState(database_operations=[RunSQL(<that file>)], state_operations=[…]), so Django owns ordering and the applied-state ledger and authors no DDL. Every model staysmanaged = True;managed = Falseis used on no table, because it would remove exactly the tables that carry invariants from the one gate watching them.makemigrations --checkcompares models against declared migration state — never against the live database — so it is kept as the model-vs-state gate, and it is kept quiet not by a flag but by declaring every object Django can model inMeta.constraints/Meta.indexes(CheckConstraint,UniqueConstraint(condition=…),Index(Lower(…)),ExclusionConstraint) and mirroring those same declarations instate_operations. Objects Django cannot model at all — triggers, column-levelGRANT/REVOKE,RANGEpartitions and their attach/detach, generated columns,DEFERRABLE INITIALLY DEFERREDconstraint triggers, andprocrastinate's vendor-managed migrations — are named in an explicit, revieweddb/schema-ignore.toml, and are covered instead by a second, SQL-level gate: CI builds a database by running every migration, capturespg_dump --schema-only --no-ownerplus a catalogue query for triggers and column privileges, and diffs that against the committed expected dump; any difference fails the build, and updating the expected dump is a reviewed part of the migration PR. Two gates, two failure modes, neither one silently lying: the ORM gate catches a model that has drifted from state, the SQL gate catches a database object that no migration created — or that a migration created and nobody reviewed. Signed off as ADR 0017 before migration001is written; it restates the mechanism already binding inadr/0002-primary-relational-database.md§3.
Which gate owns which object, so neither developer has to guess:
| Object | Declared in Meta |
In state_operations |
Watched by |
|---|---|---|---|
| Column, type, nullability, FK, plain index | yes (implicitly, by the field) | yes | makemigrations --check |
CHECK including regex (CheckConstraint) |
yes | yes | makemigrations --check |
Partial unique index (UniqueConstraint(condition=…)) |
yes | yes | makemigrations --check |
Expression index on lower() (Index(Lower(…))) |
yes | yes | makemigrations --check |
GiST EXCLUDE (ExclusionConstraint) |
yes | yes | makemigrations --check |
| Trigger (history, immutability, audit hash chain, search index) | no | no | pg_dump diff + catalogue query |
Column-level GRANT/REVOKE (append-only ats_result, audit_event) |
no | no | catalogue query on information_schema.column_privileges |
RANGE partition, ATTACH/DETACH |
no | no | pg_dump diff |
| Generated column | no | no | pg_dump diff |
DEFERRABLE INITIALLY DEFERRED constraint trigger |
no | no | pg_dump diff |
procrastinate vendor migrations |
n/a — vendor-managed | n/a | ignore-listed explicitly, never by omission |
Consequence stated plainly: the team gives up makemigrations autogeneration and pays for a
second CI gate and a committed schema dump. In exchange the schema is reviewable as a SQL diff
and every object has exactly one gate watching it. With one junior developer and constraints
that are the design, that trade is correct.
Expand/contract, mandatory, because there are no down-migrations:
| Step | Deploy | Rule |
|---|---|---|
| Expand | N | Add nullable columns, new tables, new indexes CONCURRENTLY. Never drop, never rename, never tighten |
| Migrate data | N or N+1 | Backfill in batched background jobs, never in a migration — a long-running migration blocks the deploy and can lock a table |
| Contract | N+2, at least one deploy later | Drop the old column only after no running code references it |
Rules that hold without exception: every migration is reviewed by Talha; migrations run as a
dedicated migration role, never the application role (which lacks DDL and lacks UPDATE and
DELETE on append-only tables); CREATE INDEX CONCURRENTLY on any table with real volume;
recovery from a bad migration is forward-fix plus PITR, never a down-migration.
12.5 Backup and recovery
| Asset | Mechanism | Retention | RPO | RTO |
|---|---|---|---|---|
| PostgreSQL | Managed automated backup + WAL archiving + PITR | 14 days PITR, monthly full retained 12 months | ≤ 5 min | 1–2 h for a full restore; minutes for a managed failover |
Blobs — candidate-documents |
Blob versioning OFF, deliberately (ADR 0003 §1). The only recovery path is the soft-delete window | 7-day soft delete, disclosed in the retention policy | Near zero — blobs are write-once | Minutes inside the window; unrecoverable after it |
Blobs — intake-quarantine, document-derivatives, exports |
No versioning, no soft delete. Recovery is re-derivation, not restore: quarantine bytes stay re-parseable for 30 days, derivatives are regenerable from the original, an export is re-runnable from its report_run row |
Lifecycle deletion only — quarantine 30 days after promotion or rejection; derivatives 90 days after last access; exports 7 days, no exceptions |
n/a — none of these is a system of record | Regenerate, do not restore |
| Blob storage account | Geo-redundant replication (GZRS, ASSUMPTION). Protects against region loss only, not against deletion — a deleted blob is deleted in both regions | — | — | Hours, via region failover — see the region-loss row below |
Closed audit partitions (audit-archive) |
Monthly export to a write-once immutable container (Object Lock, time-based) with the partition's final row_hash recorded |
7 years (ASSUMPTION — legal to confirm); container policy ≥13 months then policy-driven | 1 month | Hours |
| Secrets | Key Vault soft delete + purge protection | 90 days | — | Minutes |
| Code and migrations | Git, plus every image tagged by commit sha in the registry | Indefinite | — | Minutes |
Why the blob rows are deliberately weaker than they could be. Versioning plus a long
soft-delete window is the standard safety configuration for object storage, and this design
declines it for candidate-documents on purpose — the reasoning is recorded in
adr/0003-object-storage-strategy.md §1 and its Consequences, which is the authoritative source
for per-container policy. Retention purge (03-database-design.md §31.2) promises to delete CV
bytes, and blob versioning or a 90-day soft-delete window would silently keep a recoverable copy
of a document we have told a data subject was erased, making REQ-DAT-03 / REQ-DAT-06 false without
anyone noticing. The accident-recovery margin is therefore 7 disclosed days, and no more.
Restore across the two stores is non-atomic, and that is accepted. A PostgreSQL PITR to time
T does not resurrect blobs deleted after T; the database will hold candidate_document rows
whose stored_file.deleted_from_store_at is stamped and whose bytes are gone. The DR runbook
states this explicitly rather than letting the operator discover it mid-incident: after any PITR
past a purge run, reconcile stored_file against the container and mark unrecoverable documents,
do not retry the fetch.
Restore drills. A staging PITR restore to an arbitrary timestamp, quarterly, timed and written up. An untested backup is a belief, not a backup — and with two developers, the person who would perform the restore under pressure is the same person who must have practised it.
Documented recovery scenarios:
| Scenario | Procedure |
|---|---|
| Bad deploy | Revision rollback (§12.7) |
| Bad migration, no data loss | Forward-fix migration |
| Bad migration with data corruption | PITR to just before it, into a new instance; verify; repoint; replay intake from the frozen delta cursors, which is safe because landing is idempotent |
| Accidental bulk delete | Soft delete means the rows are still there — restore is an UPDATE deleted_at = NULL, audited |
| Region loss | Restore from geo-redundant backup into a second region. Accepted RTO: hours. There is no warm standby, deliberately |
| Suspected audit tampering | Compare live partitions against the immutable export and recompute the hash chain |
12.6 Health checks
| Endpoint | Checks | Used by | Rule |
|---|---|---|---|
GET /healthz |
Process is alive. No dependency checks | Liveness probe, external uptime monitor | Must never touch the database — a database blip must not trigger a restart storm |
GET /readyz |
Database SELECT 1, Redis ping, blob HEAD, applied-migration version matches the image's expected version |
Readiness probe, deploy gate | Fails → the replica is pulled from rotation but not killed |
GET /internal/queue-health |
Per-queue depth, oldest job age, worker heartbeat age, breaker state | Ops dashboard, alerting | Admin-authenticated |
GET /internal/version |
Commit sha, image tag, migration version | Incident triage | Admin-authenticated |
| Worker heartbeat row | Each worker writes a heartbeat every 30 s | queue-health, alerting |
A worker that stops heartbeating is presumed dead; its unacknowledged jobs re-run |
12.7 Rollback
| Situation | Action | Preconditions |
|---|---|---|
| Bad application code, schema unchanged | Shift 100% of traffic back to the previous Container Apps revision. Seconds | The previous image is still in the registry — keep the last 10 |
| Bad code after an expand migration | Same revision rollback. Safe because expand-only migrations are additive and the previous code ignores the new columns | The expand/contract discipline in §12.4 was followed. This is the entire reason it is mandatory |
| Bad code after a contract migration | No clean rollback. Forward-fix, or PITR | Which is why contract steps are separated by at least one deploy and reviewed as the riskiest migration class |
| Bad frontend only | Redeploy the previous static bundle from the CDN. The API is untouched | Bundles are content-hashed and retained |
| A single bad AI prompt or model config | Deactivate the PromptTemplateVersion or ModelConfigVersion row — no deploy at all. This is why they are versioned data rather than code |
The versioning pattern in _decisions.md |
| A bad scoring config | Deactivate the ScoringConfigVersion; previously displayed scores do not change, because every ApplicationScore pins its config, model, algorithm and parse versions |
Append-only scores |
| A bad merge | undo_merge(), in reverse seq order under stack discipline |
The CandidateMergeOperation log recorded every re-parent |
Rollback rule of thumb, written down because it is the one people forget: the previous revision must always be able to run against the current schema. That single rule is what makes expand/contract non-negotiable and what makes rollback a 30-second operation instead of an incident.
13. Architecture Decision Record summary
Full ADRs live in docs/architecture/adr/, one file per decision, numbered NNNN-slug.md.
Those files are the authoritative register. This table is the index and the one-line
rationale; it deliberately does not duplicate the alternatives analysis, and every row below
corresponds to a file that exists.
| ADR | File | Decision | Status | Consequence if reversed |
|---|---|---|---|---|
| 0001 | 0001-modular-monolith-versus-microservices.md |
Modular monolith, one Django 5 / DRF codebase, one image, exactly two processes (web, worker) with named hard and soft split triggers (§10.3), one PostgreSQL |
Accepted | Reversing to services requires either per-service databases (forbidden) or a shared database (the anti-pattern). A third deployable duplicates the data model, auth layer and deploy surface for two developers |
| 0002 | 0002-primary-relational-database.md |
PostgreSQL 16+ as the single primary relational database; schemas app/ref/audit/ai/staging; Phase 1 extensions pg_trgm, unaccent, btree_gist, pgcrypto |
Accepted | Every load-bearing invariant — partial unique indexes, DEFERRABLE constraint triggers, GiST EXCLUDE, column-level GRANTs — moves into application code, which is the prototype's demonstrated failure mode |
| 0003 | 0003-object-storage-strategy.md |
Document bytes live in object storage; the database holds only the index (object_store_key, sha256, virus_scan_status); upload → quarantine → scan → promote |
Accepted | Bytes in the database inflate every backup and PITR window, and erasure loses the one place where "delete the blob, keep the audit row" is expressible |
| 0004 | 0004-background-job-queue.md |
procrastinate, a PostgreSQL-backed queue; Redis is cache and rate-limit only |
Accepted | Losing transactional enqueue reintroduces the dual-write bug class in the one flow that must never lose data |
| 0005 | 0005-email-integration-method.md |
Microsoft Graph delta polling of the careers mailbox as inbound channel #1, with IMAP/SMTP as the named fallback | Proposed — assumption-dependent on IT confirmation of A1–A3; the design is settled, the premise is not | Option B (IMAP/SMTP) is promoted with no change to the surrounding intake design; only the identity and co-location benefit is lost |
| 0006 | 0006-candidate-search-strategy.md |
PostgreSQL FTS + pg_trgm over a trigger-maintained candidate_search_index in Phase 1; pgvector in the same database for Phase 2 semantic search |
Accepted | A search cluster adds a permanent dual-write and reindex-drift cost, ~4 orders of magnitude before it is needed |
| 0007 | 0007-job-and-scoring-versioning.md |
Jobs, requirements and scoring configs are immutable version rows with UPDATE revoked; ats_result pins six versions per score |
Accepted | A displayed historical score drifts when a config is edited, and "same candidate, same role, same score" (BRD §6.2) becomes unverifiable |
| 0008 | 0008-candidate-duplicate-resolution-strategy.md |
Duplicate detection with six signals, mandatory human review, additive-re-pointing merge and a reversible per-operation undo log | Accepted | Merges become destructive; a wrong merge is unrecoverable and the losing candidate's public_id, already in sent email, stops resolving |
| 0009 | 0009-permission-enforcement-strategy.md |
Application service layer (iam.can() / iam.scope()) as the primary authorization boundary; RLS as Phase 2 defence in depth |
Accepted | Authorization re-derives per view, and the constraint that the chatbot never bypasses access control loses its single chokepoint |
| 0010 | 0010-chatbot-controlled-query-architecture.md |
Allowlisted typed tool intents with a required_permission_key; no generated SQL, no chatbot service account |
Accepted | Text-to-SQL or a service account makes the access-control bypass structural rather than merely possible |
| 0011 | 0011-ai-provider-abstraction-and-versioning.md |
One narrow AiProvider port; all model access through ai_orchestration; the invocation row is written before its result is usable; human actor propagated; suggestions never write domain state |
Accepted, assumption-dependent on A5 (BRD OQ-1 unresolved) | "AI never auto-rejects" and "the chatbot never bypasses access control" revert to policy statements |
| 0012 | 0012-deployment-topology.md |
One image, two revisions, one managed container platform, one managed PostgreSQL. Azure Container Apps + Flexible Server + Blob + Key Vault + Entra ID | Accepted, assumption-dependent on A1 | Cloud-portable; the topology is unchanged and only the M365 co-location benefit is lost |
| 0013 | 0013-frontend-strangler-migration.md |
Frontend strangler migration to Vite + TS + React 18; css/styles.css content-frozen as the design contract; js/charts.js retained behind one wrapper |
Accepted | Retaining the prototype rendering layer keeps the _repo-findings.md §E XSS exposure as a per-line discipline forever, and the complex forms still to be built stall a two-person team |
| 0014 | 0014-phase-0-xss-csp-hardening.md |
Phase 0 XSS/CSP hardening of the prototype, independent of the migration, 2–3 developer-days | Accepted | Decouples the security deadline from the migration schedule; reversing bets that the migration never slips and that nobody ever points the prototype at real data |
| 0015 | 0015-module-boundary-enforcement.md |
import-linter layered contracts plus a forbidden-import contract as the boundary enforcement mechanism, failing the build |
Accepted | Boundaries become convention; the prototype's window namespace is the demonstrated failure mode, and "AI never auto-rejects" loses its structural guarantee |
| 0016 | 0016-real-postgres-in-ci.md |
Real PostgreSQL service container in CI, never SQLite; constraint tests assert that forbidden writes raise | Accepted | The team starts avoiding the Postgres features the design depends on, and the suite stops telling the truth |
| 0017 | 0017-plain-sql-migrations-as-schema-authority.md |
Plain-SQL migrations under db/migrations/ are the schema authority; Django runs them via SeparateDatabaseAndState(RunSQL + state_operations); managed = True everywhere; two drift gates — makemigrations --check for models vs state, a pg_dump/catalogue diff for everything the ORM cannot express, with an explicit db/schema-ignore.toml. Adopts the canonical text in §12.4 verbatim |
Proposed — resolves the Part 1 / Part 2 collision in §15 (I1). Must be signed off before migration 001 |
Either the invariants move into ORM escape hatches and drift, or Django's migration idioms are fought every sprint. Reverting to managed = False on invariant-bearing tables would blind both gates on exactly the tables that matter |
| 0018 | 0018-backend-language-and-framework.md |
Python 3.12 / Django 5 / DRF, psycopg3, drf-spectacular; async only for the model-streaming endpoint. Language decided by the CV-parsing and fairness-evaluation ecosystems; framework decided by team shape |
Accepted | Loses first-class PDF/DOCX/OCR, rapidfuzz, pgvector and the free admin back-office; the 7 controlled vocabularies would need a Phase 1 Settings UI, and auth, permissions and the migration workflow become a month of hand-built platform work |
One residual gap, stated rather than papered over. It has no file of its own and should get one before Phase 1 closes:
| Decision | Where it is currently argued | Why it still needs its own ADR |
|---|---|---|
| The four cross-cutting persistence patterns — versioning, history, append-only scores, money-plus-currency — as one binding ruling | ADR 0007 covers the versioning third; the rest live in _decisions.md Part 2 and 03-database-design.md |
Retrofitting history means backfilling data that was never captured — the one thing that cannot be fixed later — so the pattern, not just its first application, deserves a decision record |
Numbering rule (binding). The adr/NNNN-slug.md filenames are the register. There is no parallel
ADR-001…ADR-013 series — an earlier revision of this section carried one, and four of its
thirteen entries pointed at documents that did not exist while five real ADRs were absent from the
index. A new decision takes the next free file number; a decision cited as an ADR anywhere in the
package must have a file (07 T-08).
14. Assumptions register
Every item is an ASSUMPTION, not a finding. Each names what would change if it is wrong.
| # | Assumption | If wrong |
|---|---|---|
| A1 | Utopia Brands runs Microsoft 365, making Entra ID SSO and Graph co-located | ADR 0012 (deployment topology) changes cloud and ADR 0005 (email) falls back to IMAP/SMTP; SSO becomes a separate integration. Architecture unchanged |
| A2 | Peak concurrency 20–25 of 66 seats | Web replica count and database tier change. No structural change below roughly 10× |
| A3 | 20k–60k applications/year; 200–600 documents/day at peak; <1 TB blobs in year one | Worker sizing and the T2 hardware trigger re-derive. Bulk job-board feeds could be 1–2 orders higher |
| A4 | 10^4–10^5 candidate rows over several years | The T8 search threshold re-derives; a replica may become necessary earlier |
| A5 | Model hosting is a contracted API provider under a DPA (BRD OQ-1, unresolved) | If legal requires self-hosting, Phase 1 gains GPU infrastructure, model serving and MLOps that two developers cannot absorb. The single decision most likely to break this design |
| A6 | Audit retention of 7 years satisfies legal | Partition archive policy changes; sizing re-derives |
| A7 | Business-hours-only availability is acceptable, so a single app host with no HA is acceptable | HA means a second replica plus session affinity review, and a database HA tier. Cost roughly doubles |
| A8 | A recruiter review step for low-confidence parsed fields is acceptable (BRD §11 asks for no re-keying; BRD OQ-6 unresolved) | If unqualified accuracy is demanded, the expectation cannot be met by engineering and must be renegotiated |
| A9 | Historic hiring outcome data exists for fairness evaluation (BRD OQ-2, unresolved) | The Phase 3 activation gate could block the ranking release with no engineering fix available |
| A10 | A single region satisfies six jurisdictions (BRD OQ-4, unresolved) | Direct conflict with the one-database constraint; needs an explicit business exception, not an architectural workaround |
| A11 | Roughly 25 named seats need write access; interviewers are read-plus-scorecard only | Permission model unchanged, sizing changes |
15. Inconsistencies with _decisions.md to reconcile
Recorded rather than silently resolved, per the instruction at the top of _decisions.md.
The binding resolution for all seven is in
_open-items.md. This table is the evidence and the position this document took; the ruling is there, and where the two differ the ruling wins. I1 → RULING-02 (whose ADR was written asadr/0017-plain-sql-migrations-as-schema-authority.md; RULING-02's own text proposed0013, which was already taken by the frontend strangler migration, so0017stands and §13 is the authoritative index), I2 → RULING-07, I3 → C-02, I4 → RULING-03 (_glossary.mdis the published glossary this row asks for), I5 → C-03, I6 → C-09 (a risk, carried as03§32.2 R8), I7 → RULING-04.
| # | Inconsistency | Where | Severity | Position taken in this document |
|---|---|---|---|---|
| I1 | Migration tooling. Part 1 selects Django partly because "migrations are built in" and puts makemigrations --check in CI. Part 2 mandates ordered plain-SQL migrations, states the ORM "never generates" the schema, and explicitly rejects Django autogenerated migrations. Part 2's own risk list (line 463) flags this as a cross-document dependency |
Part 1 §"Backend language", §"Testing, CI"; Part 2 §"Schema tooling" | High — it changes daily developer workflow and who owns the schema | Settled by the canonical migration authority ruling in §12.4, which every other document in this package quotes verbatim: plain SQL is the authority, Django is the runner via SeparateDatabaseAndState, managed = True on every model (managed = False on nothing), and drift is caught by two gates — makemigrations --check for models-vs-state, and a pg_dump/catalogue diff for the triggers, grants, partitions and generated columns the ORM cannot express. The flag was never the decision; the second gate is. Needs ADR 0017 (adr/0017-plain-sql-migrations-as-schema-authority.md, currently Proposed) signed off before migration 001 |
| I2 | pgvector phase. Part 1's deployment topology enables pgvector on the Phase 1 database. Part 2 lists pgvector as "Phase 2 only" |
Part 1 §"Deployment topology"; Part 2 §"Database engine" | Low | Read as compatible: the extension may be installed in Phase 1 provisioning, but no Phase 1 feature uses it. Worth one sentence in the provisioning runbook so nobody assumes semantic search is available |
| I3 | Module count. The Part 1 summary paragraph says "26 modules with a junior"; the module tables and the consolidation section both land on 25 plus one API layer | Part 1 §"Backend language" vs §"Modules consolidated" | Low, cosmetic | 25 modules plus one cross-cutting API layer, used consistently in §4 |
| I4 | Entity naming. Part 1 uses InboundSubmission, ProcessingAttempt, Requisition/RequisitionVersion, MergeOperation. Part 2 uses raw_intake, intake_parse_attempt, job/job_version, candidate_merge_operation |
Part 1 module tables vs Part 2 throughout | Medium — two vocabularies in one design will produce two sets of names in code | §4 uses Part 2's physical table names for entities and keeps Part 1's module names. A single glossary should be published before Phase 1 code is written; requisition vs job is the one most likely to cause real confusion, since Part 2's uniqueness rules key on job_id |
| I5 | Search index ownership. Part 1 gives candidate.search() "Postgres FTS + trigram"; Part 2 requires a separate trigger-maintained candidate_search_index table because a generated tsvector column cannot see child tables |
Part 1 module 7; Part 2 §"Phase 1 search strategy" | Low | §4.2 lists CandidateSearchIndex as a candidate-owned entity. No conflict once stated |
| I6 | RLS timing. Part 1's AI boundary implies the human-actor check is the whole enforcement. Part 2 defers RLS to Phase 2 and flags that Phase 1 PII protection therefore rests entirely on a brand-new authorization layer with no database backstop | Part 1 §"AI boundary"; Part 2 §"Chatbot isolation", risk line 464 | Medium, and it is a risk, not a contradiction | Carried forward as a risk. The mitigation is concrete: one centralised authorization module, no direct repository access from views, and a test asserting every candidate-reading endpoint passes through iam.can() |
| I8 | CandidateProfileVersion does not exist — resolved, not open. Part 1's module 7 lists CandidateProfileVersion ("from a parsed doc") among the candidate module's entities, and diagram 4 in §7 of this document had the step "apply_parsed_profile as a new CandidateProfileVersion". 03-database-design.md §30.3 lists it under "Named but deliberately not created": a whole-profile version row would duplicate intake_parse_attempt.parsed, which already is the immutable parsed profile, and the question the version was actually for — "where did this candidate's job title come from, and how confident were we" — is per-field, not per-profile |
Part 1 module 7; this document §4.2 module 7 and §7 diagram 4; 03 §13.2, §30.3 |
Resolved — was a stale entity in two places | Part 2 wins. §4.2 module 7 now names CandidateFieldProvenance (app.candidate_field_provenance, 03 §13.2) and diagram 4's step reads "apply_parsed_profile — writes candidate_field_provenance rows per field, source parse attempt pinned". 03 §32.1 row 3 anticipated exactly this failure — "confirm the Part 1 module entity list is updated, or the API document will describe an entity that does not exist" — and this document was the one that had not been updated; 06-api-boundaries.md already correctly avoids the entity. Recorded here so a reader can see it was decided rather than overlooked |
| I7 | Assignment table shape — resolved, not open. Part 1's module table defines a single polymorphic Assignment(subject_type ∈ requisition/application, subject_id, …). Part 2 explicitly rejects it and mandates two concrete tables, job_assignment and job_application_assignment, because a polymorphic subject FK cannot be enforced by the database at all — and unenforceable references to jobs and candidates are precisely what this design exists to eliminate |
Part 1 module 12; Part 2 §"Which entities get current-plus-history". Also 00-scope-classification.md §10 row 2 ("Part 2 wins"), 03-database-design.md §16 and §30.3, 06-api-boundaries.md §9.1 #3 |
Resolved — was a direct contradiction, now closed | Part 2 wins. §1 and §4.2 module 12 in this document have been corrected to name JobAssignment and JobApplicationAssignment as two concrete tables with the GiST EXCLUDE overlap constraint and the partial unique index on the current primary recruiter. The module boundary is unaffected: one assignment service facade and one permission surface present both tables, so the module inventory still stands at 25. Earlier revisions of this document carried Part 1's polymorphic shape verbatim, which is how the stale definition survived three other documents rejecting it — recorded here so a reader can see it was decided rather than overlooked |
Nothing in this document diverges from _decisions.md Part 2 on schema shape. Against
Part 1, the divergences are exactly I1–I8 above; I7 and I8 are closed in Part 2's favour and the
rest are open. Outside those, nothing here diverges from Part 1 on architecture style,
process count, queue technology, module inventory, phasing, or the deployment topology.