HR-ATS-Portal/docs/architecture/02-system-architecture.md

108 KiB
Raw Permalink Blame History

System Architecture — Utopia Brands Recruitment & ATS Platform

Status / Scope of this document

Status: Design baseline for implementation. Binding within the limits set by docs/architecture/_decisions.md Part 1 — where this document adds detail it elaborates those decisions; where it disagrees, the disagreement is recorded in §15 as a risk to reconcile, never applied silently.

Scope: the runtime shape of the platform — processes, module boundaries, deployment topology, data flow through the four intake and interaction paths, background work, failure behaviour, scaling triggers, observability and the deploy/rollback procedure.

Out of scope: the relational schema (see _decisions.md Part 2 and the data-model document), REST endpoint-level contracts, and the UI screen specifications.

Evidence base: docs/architecture/_repo-findings.md (verified repository inspection, 2026-07-29) and the assignment brief. There is no meeting transcript — none exists in the repository or the surrounding filesystem (_repo-findings.md §B). The assignment prompt is the authoritative requirements source, and the distinction between CONFIRMED (§3) and UNCONFIRMED (§4) requirements is preserved throughout.

What exists today: a static, browser-only frontend prototype — index.html, css/styles.css (1269 lines), 22 unbundled scripts under js/, and a 36-line devserver.py for local preview. There is no backend, no database, no authentication, no authorization, no tests, no build step, no Docker and no environment files (_repo-findings.md §B, §D). Everything described below as backend is greenfield. The assets worth retaining are the design system (css/styles.css), the UI primitives (js/ui.js:251), the canvas chart engine (js/charts.js) and the 23-route information architecture (js/app.js:7-16).

Assumptions are labelled ASSUMPTION inline and collected in §14.


A modular monolith: one Django 5 / DRF codebase, one container image, two runtime processes, one PostgreSQL 16 database, one object store, one Redis used only as a cache.

Element Choice Why, in one line
Backend Python 3.12 + Django 5 + DRF, psycopg3, drf-spectacular Phase 1 intake is PDF/DOCX/OCR and Outlook mail; that ecosystem is decisively Python, and Django supplies auth, permissions, admin and migrations that would otherwise be a month of platform work for two people
Processes web (uvicorn ASGI) and worker (procrastinate consumer), same image, different entrypoint Parsing a scanned CV is CPU-bound and multi-second; a recruiter request is I/O-bound and sub-second. They must not share a request thread
Database One managed PostgreSQL 16+, schemas app / ref / audit / ai / staging One master data model, one backup story, one psql. Forbidden to split
Queue procrastinate — durable queue inside the same PostgreSQL Enqueue commits in the same transaction as the intake row, so the "document silently lost" bug class cannot occur without building an outbox
Cache Managed Redis — cache, rate limiting, sessions. Never a broker, never a store of record One durable component, not two
Blobs Object storage, accessed only through the files module with signed URLs Retention and deletion obligations attach to blobs and must be enforceable in one place
Frontend Progressive strangler migration to Vite + TypeScript + React 18; css/styles.css content-frozen and js/charts.js retained verbatim The CSS is the verified asset (dual themes, WCAG 2.1 AA across 23 routes × 2 themes); the string-template innerHTML rendering layer is the liability (34 unescaped sites, _repo-findings.md §E)
Module boundaries Python packages, one service.py facade each, enforced by import-linter contracts in CI Boundaries that are not machine-checked decay; the prototype already demonstrates the failure mode with everything on window (index.html:264-285)

How the architecture discharges each non-negotiable constraint

Constraint Structural mechanism Where it lives
Raw intake exists before candidate creation candidate.created_from_raw_intake_id and job_application.raw_intake_id are NOT NULL against non-deferrable FKs. Manual recruiter entry is not an exception — the UI writes a manual_ui intake row first intake module + database FK
Candidate identity separate from applications candidate and job_application are distinct aggregates in distinct modules; nothing on candidate references a job candidate, application
ATS scores are per-application ApplicationScore keys on application_id, never candidate_id; append-only with superseded_by scoring
Jobs, requirements, scoring configs versioned Immutable version rows with current_version_id pointers and UPDATE revoked; every downstream row pins the version it consumed requisition, scoring
Duplicate detection with manual review and reversible merge Detection is a background scan producing suspected links; merge is additive re-pointing with a per-operation undo log duplicate_review
Recruiter assignment flexible and historical Two concrete interval tables, JobAssignment and JobApplicationAssignment — never one polymorphic (subject_type, subject_id) table, because a polymorphic FK cannot be enforced by the database. [valid_from, valid_to) intervals, valid_to IS NULL = current, a GiST EXCLUDE constraint against overlapping identical (subject, user, role) intervals, and a partial unique index giving exactly one current primary recruiter per job. One assignment service facade presents both assignment
Current state and history Denormalised current column on the aggregate, maintained by trigger, alongside typed history tables; plus the append-only audit.audit_event every domain module + audit
AI explainable, versioned, reviewable Every invocation writes an AiRun (capability, model id + version, prompt template version, input ref, raw output, tokens, cost, latency) before its result is usable; AiReview records verdicts ai_orchestration
AI never auto-rejects intelligence tier is forbidden by an import-linter contract from importing domain modules for write; application.transition() refuses terminal-negative transitions unless actor_kind = 'user' — the exact literal, per _decisions.md RULING-01. actor_kind is ('user','system','integration','ai_agent'); there is no human value, and a guard spelled actor_type != 'human' would compare against a value absent from the CHECK and so fail closed on every legitimate human rejection or throw dependency graph + service guard + DB check
Chatbot never bypasses access control ai_orchestration.invoke() takes the human actor and calls iam.can() with it. There is no service account and no system principal for the assistant. Phase 1 gives the assistant no SQL access at all — only whitelisted, typed, parameterised query intents over the same service layer the UI uses assistant, identity
Not multi-tenant, one database No tenant column, no region column, no tenant module. Retention and residency are per-record, never per-topology schema-wide

Deliberately not introduced in Phase 1

Kubernetes, Kafka or any log broker, microservices, Elasticsearch/OpenSearch, a second database, a separate vector database, a separate AI or parsing service, a BFF, GraphQL, self-hosted model inference, read replicas. Each is either forbidden by constraint or unjustified at 66 named seats and a projected 20k60k applications/year (ASSUMPTION). §10 gives the numeric trigger that would change each answer.


2. Architecture comparison

Three candidates, scored 1 (worst) to 5 (best) against weighted criteria. Weights reflect this project's actual constraints: two developers with one reviewer, 66 users, one mandated database, and a Phase 1 critical path that must be transactional.

# Criterion Weight Modular monolith (2 processes) Microservices (per-module) Hybrid (monolith + extracted AI/parsing service)
1 Fit to team size — 2 devs, 1 reviewer, no ops staff 5 5 — one pipeline, one dashboard set, one runbook 1 — 20+ pipelines and contract versioning for two people is malpractice 3 — two pipelines, two images, two on-call surfaces
2 Fit to existing stack and repo assets 3 5 — no backend exists to preserve (§B); one API for one SPA consumer 3 — nothing to reuse, and the SPA gains an aggregation problem 4 — same as monolith plus one seam
3 Module boundary strength 4 4 — package isolation + service.py facade + CI contracts; strong but bypassable by a determined developer 5 — network boundary is unbypassable 4
4 Deployment complexity 4 5 — one image, one revision pair, one manual promote 1 — service mesh or hand-rolled discovery, N deploy orders, N schema migrations 3 — two images with a version-compatibility matrix
5 Scalability to projected load 3 5 — ~25 peak concurrent users, 200600 docs/day peak (ASSUMPTION) is comfortably inside one 2-vCPU web + one 2-vCPU worker 5 — also sufficient, at far higher cost 5
6 Failure isolation 4 3 — the genuine loss; bought back by the worker already being a separate process and an AI circuit breaker 5 4
7 Security posture — single authorization decision point 5 5 — one iam.can(); directly serves the chatbot constraint 2 — every service re-derives authorization, or a shared library drifts between them 4 — the AI service must be given a scoped, auditable channel back into can()
8 Development speed, Phase 1 5 5 — the intake→score slice is one repository and one transaction 1 — a saga plus compensations across four services 3
9 Operational burden 4 5 — one durable component to back up (Postgres), because the queue lives in it 1 — N services, N queues, distributed tracing mandatory not optional 3
10 Testability 4 4 — each module facade is a test seam; real Postgres in CI; no inter-service mocks 2 — contract tests, test doubles, environment orchestration 3
11 Transactional integrity across intake→parse→candidate→application→score 5 5 — one Postgres transaction 1 — eventual consistency in the one flow that must never lose a document 3 — parse result arrives asynchronously, which is already the design
Weighted total (max 230) 214 105 160

Verdict: modular monolith. Microservices lose on every criterion that this project is actually constrained by, and would drag in forbidden infrastructure (a cluster, or per-service databases against the one-database rule). The hybrid is the only serious alternative, and it is rejected as a Phase 1 starting point but adopted as the Phase 2+ evolution path — the worker/queue split already gives its main benefit, and §10 names the exact triggers that would promote the parsing/AI worker to a separate deployable. Starting hybrid would pay the cost of a second deployable before any evidence justified it.

Honest statement of what is given up: with one app host there is no high availability. A platform-level restart is a few minutes of downtime. That is acceptable for an internal recruiting tool used in business hours, but the six-jurisdiction user spread narrows the maintenance window, and this should be told to the business rather than discovered during the first deploy that overlaps Singapore hours.


3. System context

Diagram 1 — System context

graph TB
  subgraph PEOPLE["People"]
    REC["Recruiter<br/>15 seats"]
    HM["Hiring manager<br/>12 seats"]
    INTV["Interviewer<br/>24 seats"]
    LEAD["Talent lead / HR admin<br/>6 seats"]
    EXEC["Dept head / CEO<br/>3 seats"]
    CAND["Candidate<br/>external, token-gated"]
  end

  subgraph PLATFORM["TalentFlow ATS — one deployable image, two processes"]
    WEBP["web process<br/>REST API v1 + React bundle + streaming assistant"]
    WORKP["worker process<br/>parse, score, poll, sweep"]
    PG[("PostgreSQL 16<br/>domain + queue + audit")]
    OBJ[("Object storage<br/>CV and letter blobs")]
    RDS[("Redis<br/>cache, rate limit, sessions")]
  end

  subgraph EXTERNAL["External systems"]
    ENTRA["Entra ID<br/>SSO, ASSUMPTION"]
    GRAPH["Microsoft Graph<br/>recruiting mailbox"]
    CAREERS["Careers site<br/>public application form"]
    BOARDS["Job boards<br/>Phase 3 in, Phase 4 out"]
    AIP["AI provider API<br/>contracted, under DPA"]
    SMTP["Outbound email<br/>transactional"]
  end

  REC --> WEBP
  HM --> WEBP
  INTV --> WEBP
  LEAD --> WEBP
  EXEC --> WEBP
  CAND --> CAREERS
  CAND -->|"token-gated status page"| WEBP

  WEBP --> PG
  WEBP --> OBJ
  WEBP --> RDS
  WORKP --> PG
  WORKP --> OBJ
  WEBP -.->|"enqueue in the same transaction"| PG
  PG -.->|"LISTEN / NOTIFY"| WORKP

  WEBP --> ENTRA
  CAREERS -->|"HTTPS form post"| WEBP
  WORKP -->|"delta poll, Mail.Read"| GRAPH
  WORKP -->|"scoring, extraction, summarisation"| AIP
  WEBP -->|"assistant streaming only"| AIP
  WORKP --> SMTP
  WORKP -->|"Phase 3 inbound / Phase 4 outbound"| BOARDS

External dependencies and what happens when each is unavailable

System Direction Owner of the risk Failure behaviour
Entra ID Inbound auth (ASSUMPTION — pending confirmation Utopia runs M365) Corporate IT Nobody can log in. Break-glass: a small set of local Django superuser accounts with MFA, used only for incident response, every login audited
Microsoft Graph Outbound poll of the recruiting mailbox Corporate IT — needs an app registration, admin consent for Mail.Read, and a dedicated mailbox Polling backs off; the delta cursor is not advanced, so nothing is lost. Channel health goes amber in the intake UI after 2 missed cycles
Careers site Inbound HTTPS form post Marketing / web Applications stop arriving from that channel. The endpoint is idempotent on submission hash, so a retrying client cannot double-create
AI provider Outbound HTTPS Vendor, under a DPA Circuit breaker opens; scoring jobs park in deferred, not failed; the platform stays fully usable without scores (§9)
Object storage Read/write of CV blobs Cloud platform Intake refuses new attachments rather than recording an intake row with a missing blob; existing text-extracted content still renders
Job boards Phase 3 in, Phase 4 out Vendors Out of the Phase 1 critical path entirely

Trust boundaries

  1. Public internet → web process. Anything arriving here is hostile until proven otherwise: careers-form posts, candidate document uploads, token-gated status pages.
  2. Corporate identity → session. Authenticated but not authorised; every request still passes iam.can().
  3. Attacker-supplied file → parser. The sharpest boundary in the system. Parser libraries running over arbitrary PDF and DOCX files are a real RCE and resource-exhaustion surface. Phase 1 mitigations: hard per-document CPU and wall timeouts, a memory cap, no outbound network from the parse step. Phase 2 promotes this to a worker-untrusted queue running under a restricted OS user. This is the split trigger most likely to fire first (§10).
  4. Stored raw string → rendered pixel. The database deliberately preserves originals unsanitised (address_original, full_name_original, raw_intake.payload, intake_parse_attempt.parsed) so the arrival record is recoverable. That is correct for storage and guarantees attacker-controlled strings reach the rendering layer, so escaping must be an output-side property: JSX escaping in the React app, and the Phase 0 UI.esc() pass plus a CSP without unsafe-inline on the prototype (_repo-findings.md §E).
  5. Human actor → AI provider. No data crosses this boundary that the asking human cannot already see, because invoke() checks the human actor's permissions before any context is assembled.

4. Logical module catalogue

25 logical modules in five tiers, plus one cross-cutting API layer that is not a module. Every module lives in the monolith. The Runtime column states which process executes the module's code: web (request path), worker (queue path), or both.

Dependency rule, enforced by import-linter layered contracts in CI:

surfaces  →  core domain  →  platform
intelligence  →  core domain (READ only)  +  platform

identity, audit and config are ambient dependencies of every module. Core domain modules must never import intelligence or surfaces.

Diagram 2 — Application components and module map

graph TD
  API["API layer<br/>DRF /api/v1, OpenAPI, auth middleware,<br/>one error envelope, request-id, rate limits"]

  subgraph SURFACES["Surfaces tier"]
    S_AN["analytics"]
    S_AS["assistant"]
    S_IN["integrations_inbound"]
    S_OU["integrations_outbound"]
    S_WL["worklist"]
  end

  subgraph INTELLIGENCE["Intelligence tier — read-only into domain"]
    I_AI["ai_orchestration"]
    I_SC["scoring"]
    I_FE["fairness_evaluation"]
    I_DP["document_parsing"]
  end

  subgraph DOMAIN["Core domain tier"]
    D_IK["intake"]
    D_CA["candidate"]
    D_DR["duplicate_review"]
    D_RQ["requisition"]
    D_AP["application"]
    D_PL["pipeline"]
    D_AS["assignment"]
    D_IV["interview"]
    D_AM["assessment"]
    D_OF["offer"]
    D_TP["talent_pool"]
  end

  subgraph PLATFORM["Platform tier — ambient"]
    P_ID["identity"]
    P_AU["audit"]
    P_FI["files"]
    P_CF["config"]
    P_NO["notifications"]
  end

  API --> SURFACES
  API --> DOMAIN
  SURFACES --> DOMAIN
  DOMAIN --> PLATFORM
  INTELLIGENCE --> PLATFORM
  INTELLIGENCE -.->|"read only — no domain writes"| DOMAIN
  D_AP -.->|"accept_suggestion, human actor"| I_SC

  classDef workeronly fill:#6b4a9e,stroke:#3b2a5e,color:#fff
  classDef webonly fill:#1f6f5c,stroke:#0f3f34,color:#fff
  class I_DP,I_FE workeronly
  class S_AS,S_AN webonly

Purple = worker-only execution. Green = web-only execution. Everything unmarked runs in both processes. Note the dotted edges: intelligence reads domain state but never writes it, and the only path from a suggestion into domain state is a domain module calling accept_suggestion() with a human actor. That is what makes "AI never auto-rejects" a property of the dependency graph rather than a policy in a document.

4.1 Platform tier

# Module Responsibility Main entities APIs / interfaces Depends on Permission boundary Background tasks Phase Runtime
1 identity Users, roles, scoped role assignments, sessions, SSO, and the single authorization decision point User, Role, Permission, RoleAssignment (scope: brand / department / requisition), Session authenticate(), can(actor, action, resource), scopes_for(user), DRF permission classes; POST /api/v1/auth/*, GET /api/v1/me Admin-only writes; every user reads own profile Session and token cleanup 01 both
2 audit Append-only record of every state change, access event and AI-influenced decision AuditEvent(actor, actor_kind ∈ user/system/integration/ai_agent, on_behalf_of, action, entity ref, before, after, request_id, ai_run_id, outcome, occurred_at) record(), query(); GET /api/v1/audit (admin) identity Read: admin + compliance. No update or delete path exists — grants revoked, trigger raises, hash chain verifies Nightly hash-chain verification, monthly partition create, 13-month partition detach and archive 1 both
3 files Blob storage, checksums, malware scan hook, signed URLs, retention and deletion StoredFile(sha256, mime, size, storage_key, scan_status, retention_class) store(), signed_url(ttl), delete_for_subject() identity Access derived from the owning domain object. No public blob URLs ever Virus scan, text-extract handoff, retention sweep, orphan-blob reconciliation 1 both
4 config Controlled vocabularies, workspace settings, email and letter templates, branding RefValue, Setting, TemplateVersion values_for(vocab), setting(key), render_template(); GET /api/v1/ref/{vocab} identity configure permission only. Django admin is the Phase 1 UI, which is what lets the Settings screen wait until Phase 4 Cache warm and invalidate 1 both
5 notifications In-app notifications and transactional outbound email to recruiters and candidates Notification, NotificationPreference, OutboundMessage(status, provider_ref) notify(), send_email(); GET /api/v1/notifications identity, config, files Recipient sees own only Digest send, delivery retry with backoff, bounce and complaint handling 1 minimal send / 2 full pipeline — Phase 1 gets the OutboundMessage row, send_email() through the MailProvider port, the idempotency guard and NDR classification, because Phase 1 owns inbound mail and an NDR with no row to attach to cannot be classified (04 §9.1 row 5, 07 §17 divergence 6, 08 §7 finding 10). Everything in the Background-tasks column is Phase 2 both

4.2 Core domain tier

# Module Responsibility Main entities APIs / interfaces Depends on Permission boundary Background tasks Phase Runtime
6 intake The raw layer. Every inbound submission is recorded and triaged before any candidate exists IntakeChannel, RawIntake(channel, external_message_id, received_at, payload jsonb, payload_sha256, state), RawIntakeAttachment → StoredFile, IntakeParseAttempt (append-only), IntakeResolution(kind, decision_mode, decided_by, reason) ingest(channel, payload) — idempotent on (channel, external_message_id) and on (channel, payload_sha256); queue(), resolve(intake, decision), retry(intake); GET/POST /api/v1/intake files, document_parsing, candidate, config Recruiter triages. Nobody can hard-delete a submission. Terminal states rejected_unusable and quarantined carry no candidate Parse dispatch, exponential-backoff retry, stale-queue alert, dead-letter sweep 1 both
7 candidate Person identity, independent of any application Candidate, CandidateEmail, CandidatePhone, CandidateFieldProvenance, CandidateSkill, CandidateDocument, CandidateSearchIndex, consent and retention fields create_from_intake(), apply_parsed_profile(), search(query, filters) — Postgres FTS + trigram, erase(subject); GET/PATCH /api/v1/candidates files, config Recruiter and HR read-write. Interviewer sees only candidates on their own assigned interviews Search-index refresh, retention sweep (pseudonymisation, not deletion), embedding generation from Phase 2 1 both
8 duplicate_review Detect duplicates, mark them, support manual review and a reversible merge DuplicateCandidatePair(a, b, CHECK a < b, score, signals jsonb, detector_version, matching_config_version_id, state ∈ open/confirmed_duplicate/confirmed_distinct/merged)confirmed_distinct is the permanent "not a duplicate" memory the detector must consult and skip, not a dismissed queue item; CandidateMerge(performed_by_user_id NOT NULL, reason NOT NULL), CandidateMergeOperation(op_kind, table, previous_value, seq) find_candidates(candidate), confirm_distinct(), confirm_merge(), undo_merge() — reverse seq order under strict stack discipline; GET/POST /api/v1/duplicate-pairs candidate, audit Merge and unmerge require HR-admin. Recruiters may only flag Nightly rescan of new and updated candidates 1 detect / 2 merge UI both
9 requisition The versioned job of record, with weighted requirements and an approval chain Requisition(current_version_id, status), RequisitionVersion (immutable: title, dept, BU, location, employment type, grade, salary range as amount + ISO-4217 currency, description, effective_from, created_by), RequisitionRequirement (version-scoped, weighted), ApprovalStep create_draft(), publish_version(), current_version(), version_at(ts); GET/POST /api/v1/requisitions config, identity Create and edit: recruiter + HR. Publish and approve: hiring manager or department head Deadline sweep, stale-draft sweep 1 both
10 application The candidate ↔ requisition-version join. Owns stage and status, and is the unit ATS scores attach to Application(candidate_id, requisition_version_id, job_id, raw_intake_id NOT NULL, source_channel, current_stage, status, attempt_no, applied_at), ApplicationStageHistory(from, to, actor, actor_kind, reason, ts), SourceAttribution apply(), transition(app, to_stage, actor)refuses terminal-negative transitions unless actor_kind = 'user' (the exact literal; _decisions.md RULING-01 — there is no human value in the enum), accept_suggestion(suggestion, actor), history(); GET/POST/PATCH /api/v1/applications candidate, requisition, pipeline, audit Recruiter or assigned owner writes; hiring manager approves; interviewer read-only SLA breach detection, cooling-off evaluation 1 both
11 pipeline Stage definitions and per-requisition stage configuration. Owns the rules, not the state StageDefinition, PipelineConfig(requisition_version_id nullable = default), TransitionRule stages_for(requisition_version), is_allowed(from, to) config configure permission only — deliberately not the recruiter edit permission that governs stage changes 1 default / 3 custom web
12 assignment Flexible, historical recruiter and manager ownership Two concrete tables, never one polymorphic table. JobAssignment(job_id, user_id, role_id → ref.assignment_role, valid_from, valid_to NULL = current, assigned_by, reason, allocation_pct) and JobApplicationAssignment(job_application_id, …identical shape…). Both carry EXCLUDE USING gist (<subject> WITH =, user_id WITH =, role_id WITH =, tstzrange(valid_from, valid_to) WITH &&) plus a partial unique index (job_id) WHERE valid_to IS NULL AND role_id = primary_recruiter — exactly one current primary recruiter per job. Roles are a reference table (primary_recruiter, supporting_recruiter, sourcer, coordinator, hiring_manager, interviewer), not an inline enum. Denormalised current_primary_recruiter_id on both subjects, trigger-maintained One assignment service facade over both tables: assign(), unassign(), transfer(), owners_at(subject, ts), workload(user, window); two REST families GET/POST /api/v1/jobs/{id}/assignments and GET/POST /api/v1/applications/{id}/assignments, plus GET /api/v1/assignments/me. No polymorphic POST /api/v1/assignments taking subject_type identity, application, requisition HR-admin reassigns; recruiters view own workload Workload rollup 1 both
13 interview Scheduling and structured scorecards Interview(application_id, type, mode, starts_at utc, scheduling_timezone, local_start_wall, status), InterviewParticipant, Scorecard(interviewer, criteria scores, recommendation, submitted_at, locked), ScorecardTemplateVersion schedule(), reschedule(), submit_scorecard(); GET/POST /api/v1/interviews application, identity, notifications, config Interviewer sees and scores only own interviews. Scorecards lock on submit Reminders, calendar sync, no-show sweep, nightly wall-clock reconciliation 2 both
14 assessment Assign and record structured tests AssessmentTemplateVersion, AssessmentAssignment, AssessmentResult assign(), record_result() application, notifications Recruiter assigns; results visible to recruiter + hiring manager Invite send, expiry sweep 3 both
15 offer Offer lifecycle with mandatory human approval and issue Offer(application_id, version chain, status, base amount + currency, components, dates), OfferApprovalStep, OfferStatusHistory, OfferLetterDocument draft(), submit_for_approval(), approve(), issue()human confirmation required, never automated application, config, files, notifications, identity Draft: recruiter. Approve: department head or HR-admin. Issue: never automated Expiry sweep, approval reminder 3 both
16 talent_pool Re-surface previously sourced candidates Pool, TalentPoolMembership(candidate, pool, reason, added_by), SavedSegment add(), rematch(pool, requisition_version) candidate, application, scoring Recruiter-managed Periodic rematch 3 both

4.3 Intelligence tier

# Module Responsibility Main entities APIs / interfaces Depends on Permission boundary Background tasks Phase Runtime
17 document_parsing Turn an attachment into structured, confidence-scored fields ParsedDocument(file, parser_name, parser_version, text, extracted jsonb, per-field confidence), ParseIssue parse(file) -> ParsedDocument files, config Internal. Output is visible wherever the source document is visible All of it. Runs in worker; Phase 2 moves to the worker-untrusted queue under a restricted OS user with no outbound network 1 worker
18 ai_orchestration The only holder of a model-provider client. Capability registry, prompt and model versioning, run ledger, review surface AiCapability, PromptTemplateVersion, ModelConfigVersion, AiRun(capability, model id + version, prompt version, input_ref, output jsonb, tokens, cost, latency, status), AiSuggestion, AiReview(run, reviewer, verdict, note), AiFeedback invoke(capability, context, actor) — actor is always the human, run(id), status(job_id), review(); GET /api/v1/ai/runs, GET /api/v1/ai/jobs/{id} identity, audit, config Invocation is checked against the human actor's permissions. There is no service account. Run ledger readable by admin + compliance Batch invocation, evaluation runs, cost rollup, circuit-breaker state probe 1 framework + AI-1/2/3; 24 remainder both — worker for batch, web for the one streaming endpoint
19 scoring Per-application ATS match score, reproducible and explainable ScoringConfigVersion(weights, feature set, excluded-attribute list, active_from, evaluation_run_id), ApplicationScore(application_id, config_version, model_version, algorithm_code_version, score 0-100, band, computed_at, superseded_by), ScoreComponent(feature, weight_applied, contribution, evidence_ref), SkillMatch(matched[], missing[]) score(application), explain(score_id), rescore_batch(requisition_version) application (read), requisition (read), document_parsing, ai_orchestration Score visibility is a configurable role setting (BRD OQ-5, unresolved). Config changes are admin-only and gated on a passing evaluation run Batch rescore on requisition-version publish or scoring-config activation 1 worker
20 fairness_evaluation Disparate-impact evaluation per scoring and model version. Gates activation EvaluationRun, EvaluationMetric, EvaluationDataset evaluate(config_version), results(config_version) scoring, audit Results readable by business and legal, not only engineering Evaluation batch 3 — must complete before ranking is released to recruiters worker

4.4 Surfaces tier

# Module Responsibility Main entities APIs / interfaces Depends on Permission boundary Background tasks Phase Runtime
21 assistant Conversational surface over permitted data. Tool-calls only into existing module service facades Conversation, Message, ToolInvocation(tool, args, result_ref, actor) ask(actor, question, screen_context) — streaming; POST /api/v1/assistant/ask (SSE) ai_orchestration, identity, domain service facades Every tool call carries the asking user's identity. No SQL access in Phase 1 — only whitelisted, typed, parameterised query intents. Text-to-SQL against any application role is prohibited None in Phase 1 2 read-only / 4 full tool use web
22 integrations_inbound Channel adapters for the 11 inbound channels, all funnelling into intake.ingest() ChannelConnection(type, credential_ref, delta_cursor, health), IngestionRun(channel, started, items, failures), DeadLetter one fetch() -> [RawSubmission] per adapter; POST /api/v1/inbound/{channel} for push channels intake, files, config Admin configures connections. Credentials live only in the secret store, never in the database Graph delta polling, webhook receipt, OAuth token refresh, dead-letter retry 1: Outlook + careers portal + manual upload. 3: 4 job boards. 4: referral / agency / campus / walk-in forms both
23 integrations_outbound Publish requisition versions to the 8 external platforms JobPosting(requisition_version, platform, external_id, state, cost_band), PublishAttempt publish(), unpublish(), sync_state() requisition, config Publish requires HR-admin — it has cost implications Publish, state reconciliation 4 worker
24 analytics Read-only reporting and dashboard KPIs over materialised read models SavedReport, ReportRun. Owns read-only SQL views, no domain tables kpis(), funnel(), time_to_hire(), source_performance(), recruiter_performance(); GET /api/v1/analytics/* read-model views over application, requisition, assignment, offer Scoped by the viewer's role: recruiter sees own, CEO and department heads see aggregate Nightly materialised-view refresh 2 KPIs / 3 report library web for reads, worker for refresh
25 worklist Recruiter tasks and SLA items, including AI next-best-action suggestions Task(subject_ref, assignee, due_at, status, origin ∈ manual/rule/ai_suggestion) create(), complete(), for_user(); GET /api/v1/worklist/tasks (module-prefixed, per 06 §4.5 — there is no /api/v1/tasks) identity, application, notifications Assignee and their manager Rule evaluation, overdue sweep 2 both

4.5 The API layer — cross-cutting, not a module

DRF under a versioned /api/v1/ namespace. One auth middleware, one error envelope, one pagination and filtering convention, request-id propagation into logs and audit rows, per-user rate limits in Redis, and an OpenAPI schema generated by drf-spectacular from which the TypeScript client is generated — so the contract is written once. Owned by Talha.

4.6 Modules deliberately not created

Tempting module Actual disposition Why
search Postgres FTS + pg_trgm over a trigger-maintained candidate_search_index table, inside candidate Sufficient to ~2M candidate rows; §10 has the numeric trigger
embeddings A pgvector column on the candidate profile version, Phase 2 Keeping vectors in the same database is what makes "no separate AI service in Phase 1" achievable rather than aspirational
workflow_engine State machines live in the owning module's service Speculative until three modules demonstrably need the same engine
calendar (prototype route, FR-16) A read view over interview + worklist Owns no entities. A Calendar table would duplicate interview state
managers (FR-15) Managers are User rows with a role plus assignment rows The prototype keeps a parallel Manager entity with its own name, title and department. Two sources of truth for a person is how permissions drift
recruiterhub (FR-9) assignment (workload, history) + analytics (efficiency, SLA) A dashboard over other modules' data
aistudio (FR-19) ai_orchestration admin surface over AiCapability status It is an activation surface, not a domain
settings (FR-22) Split: security / users / roles → identity; vocabularies / templates / branding → config The current screen is inert chrome (js/settings.js:148-154); splitting puts each setting behind the permission that actually governs it
help (FR-23) Static docs plus client-side search, Phase 4 No entities, no backend
reporting_warehouse Read-model SQL views inside analytics No second datastore at this volume
tenant / region Does not exist Excluded by constraint

5. Component boundaries

5.1 The five rules

  1. One public entry point per module. Every module is a Python package whose only public surface is service.py, with dto.py for the typed data it exchanges. Cross-module imports may touch <module>.service and <module>.dto and nothing else — never models, views, selectors or tasks.
  2. No module reads or writes another module's tables. The sole exception is analytics, which owns read-only SQL views declared in migrations. The exception is auditable because it appears in a schema diff.
  3. Core domain never imports intelligence or surfaces. AI output is attached by a domain module accepting a suggestion. This is what makes "AI never auto-rejects" structurally true.
  4. identity, audit and config are ambient and may be imported by anything.
  5. Violations fail the build. import-linter layered contracts plus a forbidden-import contract. A boundary breach is a red pipeline, not a review argument.

5.2 Code layout

ats/
  api/                      # DRF routers, versioned URLconf, error envelope, schema
  platform_/
    identity/  audit/  files/  config/  notifications/
  domain/
    intake/  candidate/  duplicate_review/  requisition/  application/
    pipeline/  assignment/  interview/  assessment/  offer/  talent_pool/
  intelligence/
    document_parsing/  ai_orchestration/  scoring/  fairness_evaluation/
  surfaces/
    assistant/  integrations_inbound/  integrations_outbound/  analytics/  worklist/
  <each module>/
    service.py    # THE public facade — the only importable symbol set
    dto.py        # typed inputs and outputs
    models.py     # ORM models, mapped to the plain-SQL schema (§12.4)
    selectors.py  # read queries, module-private
    tasks.py      # procrastinate task bodies, module-private
    views.py      # DRF views, module-private
    tests/
db/migrations/            # ordered plain-SQL migration files — the schema authority
web/                      # Vite + TS + React app
  src/styles/styles.css   # git mv of css/styles.css, content-frozen
  src/lib/charts.js       # js/charts.js, retained verbatim

5.3 What the facade guarantees

The facade is not decoration — it is the test seam, the transaction boundary and the authorization checkpoint. Every public facade function:

  • takes an explicit actor and calls iam.can(actor, action, resource) before touching data, so there is exactly one authorization implementation to review;
  • owns its transaction, using transaction.atomic() — callers never open transactions across two facades, because a cross-module operation that needs atomicity belongs behind one facade;
  • returns DTOs, never ORM instances, so a schema change cannot leak across a boundary;
  • is type-annotated and checked by mypy. The facade signature plus mypy is the inter-module contract; in-process contract tests would be ceremony.

5.4 Where boundaries deliberately bend

Bend Justification Containment
analytics reads across boundaries Reporting queries that respect module boundaries would be N round trips or a duplicated warehouse Read-only SQL views declared in migrations, reviewable in a diff, never writes
assistant may call many domain facades It is a surface over the whole product Every call carries the asking human's identity and goes through the same can() the REST API uses
History and audit rows are written by database triggers, not by module code A code path that forgets to write history is invisible; a trigger cannot forget Actor and reason reach the trigger via transaction-local SET LOCAL app.actor_user_id. Named risk: any path that omits the SET LOCAL still writes history but attributes it to system with actor_unknown = true. That count is an alarmed data-quality metric (§11)

6. Deployment boundaries

Diagram 3 — Deployment topology, per environment

graph TB
  subgraph EDGE["Platform edge"]
    CDN["CDN + TLS + WAF<br/>serves the built React bundle"]
  end

  subgraph APPS["Managed container platform — Azure Container Apps, ASSUMPTION"]
    subgraph IMG["ONE container image, two revisions"]
      WEB["revision: web<br/>entrypoint uvicorn<br/>2 vCPU / 4 GB<br/>2 workers x 4 threads<br/>autoscale 1-4 replicas"]
      WRK["revision: worker<br/>entrypoint procrastinate<br/>2 vCPU / 4 GB<br/>concurrency 4<br/>autoscale 1-3 replicas"]
    end
  end

  subgraph DATA["Managed stateful services"]
    PG[("PostgreSQL 16 Flexible Server<br/>2 vCPU / 8 GB<br/>PITR, 14-day backups<br/>pg_trgm, unaccent, btree_gist, pgcrypto<br/>schemas: app ref audit ai staging")]
    BLOB[("Blob storage<br/>CV and letter blobs<br/>per-container policy — see ADR 0003")]
    REDIS[("Redis<br/>cache, rate limit, sessions<br/>NEVER a broker")]
    KV[("Key Vault<br/>Graph creds, AI key, DB password")]
    IMM[("Immutable blob container<br/>closed audit partitions,<br/>write-once")]
  end

  subgraph OBS["Observability"]
    LOGS["Log workspace<br/>structured JSON, request-id correlated"]
    ERR["Error tracking<br/>Sentry, self-hosted or EU region"]
    UPT["External uptime probe<br/>hits /healthz from outside the platform"]
  end

  CDN --> WEB
  WEB --> PG
  WEB --> REDIS
  WEB --> BLOB
  WEB --> KV
  WRK --> PG
  WRK --> BLOB
  WRK --> KV
  PG -.->|"LISTEN / NOTIFY wakes the consumer"| WRK
  PG -->|"nightly closed-partition export"| IMM
  WEB --> LOGS
  WRK --> LOGS
  WEB --> ERR
  WRK --> ERR
  UPT --> CDN

6.1 Process boundaries and why there are exactly two

Boundary What it separates Why it is a real boundary, not taxonomy
web / worker Sub-second I/O-bound request handling from multi-second CPU-bound parsing, model calls, batch rescoring and sweeps Different resource profile, different failure mode, different timeout budget. A scanned CV must never occupy a request thread
worker-default / worker-untrusted (Phase 2) Trusted background work from parsing attacker-supplied files Security, not scale. Same image, same codebase, different queue and different OS user: restricted user, no outbound network, hard CPU and wall timeout, memory cap
Web request path / streaming assistant path Standard request/response from a long-lived SSE connection The streaming endpoint is the one async Django view under uvicorn; everything else is sync

No third deployable exists in Phases 04. It would duplicate the data model, the auth layer and the deploy surface for two developers, and every requirement it would serve (documented versioned API, graceful degradation, retrievable async job status) is already met by an internal module boundary plus a circuit breaker and a job-status endpoint.

6.2 Environments

Environment Composition Data Integrations Promote
local docker compose: web, worker, postgres:16, redis, MinIO (S3-compatible) Anonymised fixture set generated by a script that reads pii_classification; never a production copy Graph and AI provider mocked by default, real credentials opt-in via .env.local (no .env exists in the repo today, _repo-findings.md §B)
staging Same image, one replica each, smaller database tier Anonymised fixtures plus real test-mailbox traffic Real integrations pointed at a dedicated test mailbox and sandbox job-board accounts Auto-deploy on merge to main
production Web 14 replicas, worker 13, PITR enabled Real Real Manual promote, Talha only

No per-developer cloud environment. Two developers do not need six environments; they need one that behaves like production.

6.3 Region and residency

One region for the single database. Which region is a legal decision, not an architectural one — postings span six jurisdictions (BRD OQ-4, unresolved). Retention and deletion are implemented per record, including derived embeddings, not per region. If legal requires in-jurisdiction storage, that conflicts directly with the no-per-region-databases constraint and requires an explicit business exception; it is not solvable with topology.


7. Data flow

7.1 The common write path

Every state-changing request follows the same shape, which is why it is worth stating once:

  1. Edge terminates TLS; the web process assigns a request_id.
  2. Auth middleware resolves the session to a User; SET LOCAL app.actor_user_id is issued on the connection so history triggers can attribute the change.
  3. The DRF view validates input against a serializer and delegates to exactly one module facade.
  4. The facade calls iam.can(actor, action, resource). Denial is a 403 and an audit_event with outcome = 'denied' and a denial reason — denials are as interesting as grants.
  5. Inside transaction.atomic(): domain writes, history rows written by trigger, and any background job enqueued into the same PostgreSQL. Commit is atomic across the domain row and its job.
  6. audit.record() writes the append-only event with request_id and, if applicable, ai_run_id.
  7. Response returns the public UUIDv7 public_id, never a bigint primary key.

Diagram 4 — Email application flow (Outlook, inbound channel #1)

sequenceDiagram
  autonumber
  participant GR as Microsoft Graph
  participant AD as inbound adapter on worker
  participant IK as intake
  participant FI as files
  participant DP as document_parsing on worker
  participant DR as duplicate_review
  participant CA as candidate
  participant AP as application
  participant SC as scoring on worker
  participant WL as worklist
  participant AU as audit

  Note over AD: periodic task on the mail queue, every 2 minutes
  AD->>GR: delta query on the recruiting mailbox
  GR-->>AD: messages plus attachments since the stored cursor
  loop per message
    AD->>IK: ingest channel=outlook, external_message_id, payload
    Note over IK: UNIQUE on channel plus external_message_id makes redelivery a no-op
    IK->>FI: store each attachment, sha256, virus scan pending
    FI-->>IK: StoredFile refs
    IK->>IK: RawIntake state=received, enqueue parse in the SAME transaction
    IK->>AU: record intake received
  end
  AD->>AD: advance delta cursor ONLY after all messages commit

  DP->>FI: fetch blob by storage key
  DP->>DP: extract text and fields, per-field confidence, hard timeout
  DP-->>IK: IntakeParseAttempt appended, status succeeded partial or failed
  alt parse failed or confidence below threshold
    IK->>IK: state=needs_review
    IK->>WL: task for the intake triage queue
    Note over IK: terminal without a candidate is a representable state
  else parse usable
    IK->>DR: find candidates by normalised email, trigram name, phone
    alt confident single match
      DR-->>IK: existing candidate
      IK->>CA: apply_parsed_profile writes candidate_field_provenance rows per field, source parse attempt pinned
    else no match
      IK->>CA: create_from_intake, created_from_raw_intake_id NOT NULL
    else ambiguous
      DR->>DR: DuplicateCandidatePair state=open
      IK->>IK: state=needs_review, human decides
    end
    IK->>AP: apply candidate, current requisition_version, raw_intake_id
    Note over AP: uq_application_live blocks a second live application for the same job
    AP->>SC: enqueue score for this application
    SC->>SC: pin config version, model version, parse attempt, job version
    SC-->>AP: ApplicationScore plus ScoreComponent rows, append-only
    AP->>AU: record application created and scored with ai_run_id
    AP->>WL: recruiter task new scored application
  end

Load-bearing properties. The delta cursor advances only after every message in the batch has committed, so a crash mid-batch replays rather than skips. Idempotency lives in a database unique constraint, not in adapter code. A parse failure produces a reviewable terminal state, not a silent drop and not a ghost candidate — which is exactly the state the prototype's pre-resolved inbox cannot represent (js/data.js:284-300). Nothing on this path can auto-reject: the only transitions taken are forward and neutral.

Diagram 5 — Website application flow (careers portal)

sequenceDiagram
  autonumber
  participant CD as Candidate browser
  participant CDN as Edge
  participant WEB as web process
  participant IK as intake
  participant FI as files
  participant Q as queue in Postgres
  participant DP as document_parsing on worker
  participant NO as notifications

  CD->>CDN: GET careers form for a published requisition_version
  CDN->>WEB: unauthenticated read, rate limited per IP
  WEB-->>CD: form rendered from the PUBLISHED version only
  CD->>WEB: POST application, fields plus CV file
  WEB->>WEB: rate limit, size and MIME allowlist, no rich text accepted
  WEB->>FI: store blob, sha256 computed, scan_status=pending
  WEB->>IK: ingest channel=careers_portal, external_ref=submission_uuid
  Note over IK: UNIQUE on channel plus payload_sha256 makes a double submit idempotent
  IK->>Q: enqueue parse, SAME transaction as the RawIntake insert
  WEB-->>CD: 202 Accepted plus a reference code, NO score and NO status shown
  Note over CD,WEB: the candidate is never told a score exists
  Q-->>DP: LISTEN NOTIFY wakes the consumer
  DP->>DP: parse, then the same resolution path as diagram 4
  IK->>NO: acknowledgement email, template from config
  Note over NO: a candidate-facing status page is gated by candidate_access_token, hashed and expiring, never by public_id

Why 202 and not 201. The response cannot wait on parsing, and the candidate must not learn anything about internal state. The reference code is internal-shaped (APP-30001-style) but the candidate-facing link uses the UUIDv7 public_id behind an expiring hashed token — public_id is an identifier, never a capability.

Diagram 6 — Manual upload flow (recruiter, and bulk import)

sequenceDiagram
  autonumber
  participant RC as Recruiter
  participant WEB as web process
  participant ID as identity
  participant FI as files
  participant IK as intake
  participant DP as document_parsing on worker
  participant CA as candidate
  participant AP as application
  participant AU as audit

  RC->>WEB: upload one or many CVs, optional target requisition
  WEB->>ID: can actor=recruiter action=intake.create
  ID-->>WEB: allow, scoped to the recruiter's brands and departments
  loop per file
    WEB->>FI: store blob, sha256, scan_status=pending
    WEB->>IK: ingest channel=manual_ui, external_ref=upload_batch plus index
    Note over IK: manual entry is NOT an exception, a raw_intake row is created first
    IK->>DP: enqueue parse, same transaction
  end
  WEB-->>RC: batch accepted, N queued, live progress by polling the job status endpoint

  DP-->>IK: IntakeParseAttempt per file
  IK-->>WEB: triage queue populated
  RC->>WEB: review a parsed record, correct low-confidence fields
  Note over RC,WEB: low-confidence fields are left EMPTY, never guessed
  RC->>IK: resolve kind=create_candidate, decision_mode=human
  IK->>CA: create_from_intake
  CA-->>IK: candidate public_id
  opt a target requisition was chosen
    IK->>AP: apply, raw_intake_id carried through
  end
  IK->>AU: record resolution with actor, decision_mode and reason

Why manual entry goes through intake. Exempting it would create a second, unaudited path to candidate creation and would make "raw intake before candidate" a convention rather than a database fact. The cost is one extra row per manual entry. The benefit is that created_from_raw_intake_id can be NOT NULL with no exceptions, which is what makes the guarantee real.

Diagram 7 — AI chatbot flow

sequenceDiagram
  autonumber
  participant U as Recruiter
  participant WEB as web process, async view
  participant AS as assistant
  participant ID as identity
  participant AI as ai_orchestration
  participant SVC as domain service facades
  participant PR as AI provider
  participant AU as audit

  U->>WEB: POST assistant ask, question plus screen_context, SSE opened
  WEB->>AS: ask actor=U, question, context
  AS->>ID: can actor=U action=assistant.use
  ID-->>AS: allow
  AS->>AI: invoke capability=assistant_answer, actor=U
  Note over AI: the HUMAN actor is passed, there is no service account and no system principal
  AI->>AI: resolve PromptTemplateVersion and ModelConfigVersion
  AI->>AI: open AiRun row BEFORE the call, status=running
  AI->>PR: model call with the whitelisted tool schema, streaming
  loop tool calls requested by the model
    PR-->>AI: tool call, one of a whitelisted set of typed query intents
    AI->>ID: can actor=U action=read resource=the requested subject
    alt denied
      ID-->>AI: deny
      AI-->>PR: tool result denied, no data returned
      AI->>AU: audit_event outcome=denied, actor_kind=ai_agent, on_behalf_of=U
    else allowed
      AI->>SVC: the SAME facade the REST API calls, parameterised, never SQL
      SVC-->>AI: DTO, already scoped to U
      AI->>AU: access event actor_kind=ai_agent, on_behalf_of=U, entity ref
    end
  end
  PR-->>AI: final tokens streamed
  AI->>AI: close AiRun, tokens, cost, latency, status=succeeded
  AI-->>AS: answer stream plus ai_run_id
  AS-->>WEB: SSE chunks
  WEB-->>U: answer rendered with a provenance badge linking to the AiRun
  Note over U,AI: the assistant can suggest an action but cannot perform one. Any state change is a separate authorised UI action by U

Load-bearing properties. Four things are true by construction, not by prompt:

  1. The assistant has no capability the UI does not have — same facades, same can(), so there is one authorization implementation to review rather than two.
  2. There is no SQL access in Phase 1. Text-to-SQL against any application role is prohibited; ad-hoc querying, if ever required, arrives in Phase 2 as a dedicated PostgreSQL role with row-level security keyed to current_setting('app.actor_user_id') and column privileges excluding sensitive_personal columns — enforcement by the database, not by prompt engineering.
  3. Every answer is attributable: an AiRun row exists before the result is usable, and the UI links to it.
  4. The assistant cannot write. surfaces → intelligence may not reach domain writes, and any suggestion becomes state only when the human takes the action.

8. Background workers

8.1 Queue conventions

Postgres-backed procrastinate, six named queues. Redis is never a broker.

Queue Work Concurrency Priority Notes
ingest Normalise and land a raw submission 4 high Must never starve — this is the "nothing is silently lost" path
parse document_parsing.parse() 2 normal CPU-bound. Phase 2: splits into parse-untrusted on the restricted worker
score scoring.score(), rescore_batch() 2 normal Queueing lock per requisition version so a rescore batch cannot interleave with itself
ai Non-interactive model invocations 2 normal Subject to the circuit breaker
mail Graph delta poll, outbound send, bounce handling 1 normal Serialised per channel connection by a queueing lock, so two pollers cannot both advance the cursor
maintenance Sweeps, refreshes, verifications, retention 1 low Windowed to off-peak where the six-jurisdiction spread allows

Every task body: idempotent, keyed on the domain row, safe to run twice. Every task declares max_attempts and a backoff. There is no silent drop — the terminal state is failed, and failed surfaces in the intake UI as a reviewable item with the error.

8.2 Task catalogue

Task Queue Trigger Idempotency key Retry On final failure
poll_mail_channel mail periodic, 2 min channel connection lock 5, exponential to 30 min Channel health amber, alert; cursor not advanced
land_submission ingest enqueued by adapter or web POST (channel, external_message_id) unique index 5 Dead letter row, admin-visible
virus_scan ingest on StoredFile insert sha256 3 Blob quarantined, intake quarantined (terminal, no candidate)
parse_document parse on intake landing, or manual retry (file_id, parser_version) 3, then needs_review New IntakeParseAttempt with status=failed; intake to needs_review
detect_duplicates score on candidate create or profile update, plus nightly rescan candidate_id + version 3 Link left absent, alert; never auto-merges
score_application score on application create, requisition-version publish, scoring-config activation (application_id, config_version) 3, deferred while breaker open Application visible without a score; badge reads "not yet scored"
rescore_requisition_batch score on publish or config activation requisition-version queueing lock 2 Partial batch is safe — scores are append-only with superseded_by
evaluate_fairness ai on scoring-config draft config_version 2 Config cannot activate — the gate holds closed
refresh_search_index maintenance trigger-driven plus nightly full pass candidate_id 3 Stale search results, alerted
refresh_read_models maintenance nightly view name 2 Dashboard shows a staleness timestamp rather than wrong-looking numbers
verify_audit_hash_chain maintenance nightly partition 1 Page immediately — a break means tampering or corruption
create_audit_partition maintenance monthly, and 2 months ahead partition name 3 Alert before it can cause insert failures
export_closed_audit_partition maintenance monthly partition name 3 Alert; immutable off-box copy is the only independent tamper check
retention_purge maintenance nightly (policy, subject) 1 Alert. Skips retention_hold rows and candidates in an unreversed merge
sla_sweep maintenance hourly application id 3 Worklist item not created; alerted
send_notification mail on notify() OutboundMessage.id 5 OutboundMessage status=failed, visible to the sender
refresh_oauth_token mail 10 min before expiry connection id 5 Channel health red, alert
retry_dead_letters ingest hourly dead letter id 1 per cycle, 24 cycles Stays in dead letter for manual action

8.3 Ordering and concurrency hazards addressed explicitly

  • Per-candidate dedupe and per-requisition rescore use procrastinate queueing locks declaratively. Without them, two concurrent scans create mirrored DuplicateCandidatePair rows and two rescore batches interleave into inconsistent superseded_by chains.
  • Mail cursor advance is serialised per connection and happens only after commit.
  • Scores are never updated in place. A rescore inserts and marks the predecessor superseded, so a partially completed batch is a consistent state rather than a corrupt one.
  • Deploy-time drain. The worker gets SIGTERM, stops accepting, finishes in-flight tasks up to a 120 s grace period, then exits. Because jobs live in Postgres, anything not finished is simply still queued.

9. Failure handling

9.1 Failure modes

Failure Detection Behaviour Recovery User-visible effect
AI provider down, throttled or slow Error rate and p95 per capability; circuit breaker in ai_orchestration Breaker opens after 5 failures in 60 s; score and ai jobs move to deferred, not failed; interactive assistant returns an explicit "AI unavailable" state Half-open probe every 60 s; deferred jobs resume automatically Everything works except AI features. No scores appear; no score is wrong
Worker crash or OOM Container restart count, task heartbeat In-flight tasks are unacknowledged and re-run — which is why idempotency is mandatory Automatic restart; three crash-loops in an hour pages Delayed parsing and scoring. No data loss
Poison document — a PDF that hangs or exhausts the parser Per-document wall and CPU timeout, memory cap Attempt recorded as failed with the error; after max_attempts the intake goes to needs_review Recruiter reviews and can key manually; parser fix replays historic intake via a new parser_version One document needs review. This is the trigger most likely to promote parsing to its own deployable (§10)
PostgreSQL failover on the managed service Connection errors; readiness probe fails Web returns 503 with a retry hint; connection pool re-establishes; queue is unaffected because it is in the same database Managed failover, typically 60120 s A short outage. No lost writes — anything uncommitted was never acknowledged
Object storage unavailable files.store() errors Intake refuses the attachment rather than recording an intake row pointing at a blob that does not exist Retry with backoff; alert Uploads fail loudly and are retried by the user or the adapter
Graph token expired or consent revoked Token refresh failure Channel health red; polling stops; cursor frozen Corporate IT re-consents; polling resumes from the frozen cursor Email intake pauses; nothing is lost
Queue backlog queue_depth, time_to_start p95 per queue Autoscale worker replicas up to 3 If p95 time-to-start for interactive AI exceeds 10 s while web p95 stays under 300 ms, that is a soft split trigger (§10) Slower parsing and scoring
Careers-form abuse or scripted spam Per-IP rate limit in Redis; submission-hash duplicates Rate limit returns 429; duplicate hashes are idempotent no-ops Tighten limits; add a challenge only if it becomes real Legitimate applicants unaffected
Malformed email that cannot become a candidate candidate_email CHECK regex fails; the deferrable contactability trigger raises at COMMIT The whole transaction aborts. Intake stays needs_review. No ghost candidate is created Recruiter corrects and resolves manually One item in the triage queue
Audit hash-chain break Nightly verifier Immediate page. The break is evidence, not prevention — layers 1 and 2 (revoked grants, raising trigger) are the prevention, and only the off-box immutable export is an independent check Forensic comparison against the immutable export None directly; a compliance incident
History rows attributed to system because SET LOCAL was missing actor_unknown count metric Treated as a data-quality alarm, not noise Fix the code path; back-attribution is generally impossible Attribution gaps in history — invisible without this alarm, which is why it exists
Migration fails mid-deploy Deploy step exit code Deploy aborts before the new revision takes traffic; the old revision keeps serving the old schema Forward-fix migration, plus PITR if data was touched. No down-migrations exist — §12.4 None if the expand/contract discipline was followed

9.2 Degradation matrix

The design goal is that the platform is useful with AI entirely absent (BRD NFR-7).

Component down Still works Stops
AI provider Login, intake landing, manual triage, candidate and requisition CRUD, applications, stages, assignment, interviews, analytics Scoring, AI extraction assistance, assistant, next-best-action suggestions
Worker process All read and write UI operations; intake landing (the row and its job both commit) Parsing, scoring, mail polling, notifications, sweeps — all resume from the queue when the worker returns
Redis Everything, more slowly; sessions fall back to the database-backed backend Rate limiting degrades to a database counter; cache misses become queries
Object storage Browsing all existing records and text already extracted New uploads, CV downloads, letter generation
Graph Careers portal, manual upload, all internal work Email channel intake
PostgreSQL Nothing Everything. It is the single point of failure, accepted deliberately, mitigated by managed HA-capable hosting and PITR

10. Scalability

10.1 Current sizing and headroom

Dimension Phase 1 projection Basis Headroom on the recommended tier
Named seats 66 BRD §4 (2+4+15+12+8+24+1)
Peak concurrent users 2025 ASSUMPTION — 24 of the 66 are interviewers who touch only their own interviews One 2-vCPU web replica is generously provisioned; autoscale to 4
Applications per year 20k60k ASSUMPTION Trivial for PostgreSQL
Documents per day at peak 200600 ASSUMPTION One 2-vCPU worker with concurrency 4 handles this with room; at 30 s per scanned CV, 600 documents is roughly 75 worker-minutes spread over a day
Blob volume, year one well under 1 TB ASSUMPTION
Candidate rows, several years 10^410^5 ASSUMPTION Three to four orders of magnitude below the search-extraction trigger
Queue throughput hundreds of jobs/hour derived from the above Postgres LISTEN/NOTIFY queues handle thousands/hour comfortably

10.2 The scaling ladder — exhaust in order

  1. Query and index tuning. Most "we need X" moments are an unindexed query.
  2. Vertical scale. Web and worker to 4 vCPU / 8 GB; PostgreSQL to 4 vCPU / 16 GB.
  3. Horizontal scale of the web revision. Stateless behind the platform's load balancer; sessions are in Redis or the database.
  4. Horizontal scale of the worker revision, with queueing locks preventing the duplicate-work hazards §8.3 names.
  5. Split the worker by queue. Already planned for Phase 2 as worker-untrusted, on a security argument rather than a scale one.
  6. A read replica for analytics and search.
  7. Only then extract a service.

10.3 Concrete triggers that would justify extracting a service

Reviewed quarterly. Any single hard trigger justifies a separate deployable; soft triggers need two sustained for 2+ weeks. These are the same triggers as _decisions.md Part 1, restated with the metric that measures each.

# Trigger Threshold Metric that fires it Hard? Extract what
T1 Dependency conflict An ML or OCR dependency cannot coexist in the web image, or pushes it past ~2 GB Image size in CI Hard Parsing/AI worker
T2 Hardware profile divergence Parsing or inference needs a GPU, or sustained >4 vCPU / >8 GB Worker CPU and RSS p95 Hard Parsing/AI worker
T3 Runtime isolation Untrusted-file handling needs a sandbox the worker process cannot provide, beyond the Phase 2 restricted queue A parser CVE, or a confirmed sandbox escape in review Hard Parsing service with a hardened runtime
T4 Non-Python runtime A required model runtime is not Python Hard Inference service
T5 Queue starvation p95 time-to-start for interactive AI tasks >10 s while web p95 <300 ms, after vertical scaling of the worker host is exhausted queue_time_to_start_p95{queue="ai"} vs http_request_duration_p95 Soft AI worker
T6 Release cadence conflict Prompt or model changes need >2 deploys/week while domain code is release-gated Deploy log Soft AI orchestration service
T7 Blast radius Worker OOM or crash-loops have taken the shared host down twice in a quarter Container restart count and incident log Soft Worker
T8 Search outgrown Candidate rows >~2,000,000 or indexed text >~50 GB or p95 search >500 ms after tuning and after moving search to a read replica or sustained >50 queries/s degrading write latency search_latency_p95, table sizes Hard on (a); soft on the rest Search service — but exhaust tuning, replica, materialised table and a BM25 extension first
T9 Queue technology outgrown Sustained >20 jobs/second, a need for chord/fan-in semantics, or worker count >4 jobs_completed_rate, replica count Soft Migrate to Celery + Redis with a transactional outbox, to preserve the enqueue guarantee

Explicitly never a trigger: "AI feels like a different concern", org-chart preference, résumé-driven development, or multi-tenant / regional residency (excluded by constraint).

Honest projection. On the labelled volume assumptions, T8(a) sits three to four orders of magnitude away, so PostgreSQL FTS plus trigram will very likely never be outgrown by this system, and a read replica is the realistic ceiling of what will ever be needed. T3 is the trigger most likely to fire early, and it is a security argument, not a scale one.


11. Monitoring

11.1 Service level objectives

SLO Target Measured
API availability, business hours 99.5% monthly External uptime probe against /healthz
Read endpoint p95 < 300 ms Server-side histogram, excluding the streaming endpoint
Write endpoint p95 < 600 ms Same
Intake landing to parse start, p95 < 60 s received_at to first IntakeParseAttempt.started_at
Parse completion, p95 < 5 min for a text PDF, < 15 min for a scanned document Attempt duration by document class
Application created to score available, p95 < 10 min Application applied_at to ApplicationScore.computed_at
Documents lost zero, always Reconciliation: Graph message count vs raw_intake count per channel per day

11.2 Instrumentation

Metrics — OpenTelemetry to the platform's metric store:

  • http_requests_total{route,method,status}, http_request_duration_seconds{route}
  • queue_depth{queue}, queue_time_to_start_seconds{queue}, task_duration_seconds{task}, task_failures_total{task,attempt}, dead_letter_total{channel}
  • intake_landed_total{channel}, intake_state_total{state}, parse_confidence_bucket{field}, parse_failures_total{parser_version,reason}
  • ai_runs_total{capability,status}, ai_tokens_total{capability}, ai_cost_total{capability}, ai_latency_seconds{capability}, ai_breaker_state{provider}
  • authz_denials_total{action,role} — a spike is either a permission bug or an attack
  • history_actor_unknown_total — the SET LOCAL gap alarm from §5.4
  • db_connections_in_use, db_slow_queries_total, pg_table_bytes{table}

Logs — structured JSON, one line per request and per task, always carrying request_id, actor_id, module, outcome. Never the value of a sensitive_personal column. Log volume is not a compliance record — audit.audit_event is.

Traces — sampled at 10%, always-on for the intake and scoring paths, because those are the multi-hop flows where "where did the 40 seconds go" is a real question.

Business dashboards — built by Ahmed on the retained js/charts.js engine: intake funnel by channel and state, triage queue age, parse confidence distribution, score band distribution, time-to-hire, recruiter workload. These double as operational signals: a sudden shift in the score-band distribution usually means a model or config change, not a change in candidate quality.

11.3 Alerts

Alert Condition Severity Why it is worth waking someone
Audit hash-chain break Nightly verifier fails Page Tampering or corruption of the compliance record
Intake reconciliation mismatch Graph message count ≠ landed count for a channel-day Page Violates the one guarantee that must never break
Web 5xx rate >2% over 5 min Page
Database connections >80% of the pool for 10 min Page Precedes total unavailability
Queue starvation time_to_start_p95{ai} >10 s for 15 min Ticket Soft split trigger T5
Parse failure rate >20% of attempts over 1 h Ticket Parser regression or a new document format
AI breaker open Open >15 min Ticket Degradation is working; the vendor needs chasing
AI cost Daily spend >2× the 7-day mean Ticket Prompt bug or a runaway batch
Channel health Any connection red >30 min Ticket Intake pause
actor_unknown history rows Any increase day over day Ticket An unattributed write path exists
Retention purge failure Nightly job failed Ticket Legal obligation
Disk or table growth audit_event partition >2× projection Ticket Sizing assumption is wrong (§14)

Error tracking — Sentry, self-hosted or in an EU region, with a scrubbing config derived from the pii_classification table so no candidate PII enters a third-party error store. Release-tagged, so a regression is attributable to a deploy in one click.


12. Deployment recommendation

Recommended cloud: Azure — Container Apps, Database for PostgreSQL Flexible Server, Blob Storage, Key Vault, Entra ID. ASSUMPTION: Utopia Brands runs Microsoft 365, since Outlook is inbound channel #1. If true, SSO and the Graph app registration land in the same tenant and the identity problem largely disappears — the single largest free reduction in integration risk available. If false, the architecture is unchanged and the equivalent AWS or GCP services substitute directly.

12.1 Local development

docker compose up      # web, worker, postgres:16, redis, minio
make migrate           # applies db/migrations/*.sql in order
make seed              # anonymised fixtures generated from pii_classification
cd web && npm run dev  # Vite dev server, proxying /api to the web container

Rules: never a production data copy — the fixture generator reads pii_classification and applies each column's masking_strategy. Graph and the AI provider are mocked by default; real credentials are opt-in through .env.local, which is gitignored (the repository has no .env today, _repo-findings.md §B). The retained devserver.py stays only for viewing the frozen prototype.

12.2 CI pipeline — one required workflow, GitHub Actions

.github/ does not exist today (_repo-findings.md §B), so this is entirely additive.

Stage Gate Fails the build when
Lint ruff, mypy --strict on facades Style or type error
Boundaries import-linter layered + forbidden contracts A module imports another module's models, or core domain imports intelligence
Schema — gate 1 (ORM) makemigrations --check --dry-run: models vs declared migration state (§12.4) A model changed without a migration declaring the same state
Schema — gate 2 (SQL) Plain-SQL migrations applied to a scratch database, then pg_dump --schema-only --no-owner plus the trigger/column-privilege catalogue query, diffed against the committed expected dump (§12.4) Any database object exists that no reviewed migration accounts for — the gate that covers everything the ORM cannot see
Backend tests pytest against a real PostgreSQL service container Any failure. Never SQLite — the design depends on jsonb, partial and expression unique indexes, FTS, pg_trgm, deferrable constraint triggers, GiST EXCLUDE and LISTEN/NOTIFY
Constraint tests Every trigger and constraint has a test that attempts the forbidden write and asserts the exception A guard silently stops guarding
PII registry Every column on a candidate-touching table has a pii_classification row A new PII column arrives unclassified
Frontend tsc --noEmit, ESLint with react/no-danger as error, stylelint enforcing var(--…) token usage, Vitest A dangerouslySetInnerHTML, or a hardcoded colour
Prototype guard Grep gate failing on a new unescaped ${ inside an HTML template literal Someone reintroduces the §E exposure
Docs — evidence citations python3 tools/check_evidence_citations.py (stdlib only, no dependencies). Checks every path:line citation in docs/architecture/**/*.md at three levels: the file exists; every cited line is in range; the ~30 load-bearing anchors still match an expected pattern; and no sentence cites an anchor belonging to a different claim A citation goes stale because js/*.js moved, or a claim is attached to the wrong line. This package's argument is "verified by direct inspection", so a citation a reviewer can spot-check and find wrong costs more credibility than the fact is worth. Runs in the same job as the other documentation gates because it needs no services and finishes in under a second
E2E 5 Playwright journeys: login; ingest a CV and see it parsed; promote a submission to candidate; create and publish a requisition version; move an application through two stages Any failure
Image Build, scan, size check against the T1 threshold Vulnerabilities, or image >2 GB

Branch protection: no direct pushes to main; every PR requires Talha's review. That is the mandated review checkpoint for the junior's work, enforced by the platform rather than by habit.

12.3 Promotion

graph LR
  PR["Pull request"] --> CI["CI: all gates"]
  CI --> MERGE["Merge to main"]
  MERGE --> IMG["Build one image, tagged with the commit sha"]
  IMG --> STG["Auto-deploy: staging<br/>migrate then roll web then roll worker"]
  STG --> SMOKE["Playwright smoke against staging"]
  SMOKE --> GATE{"Manual promote<br/>Talha"}
  GATE -->|"approve"| PRD["Production<br/>same image, same order"]
  GATE -->|"reject"| STOP["Stop, fix forward"]

Deploy order is always: migrate → roll web → roll worker. The worker rolls last because it must never process a job written by code that assumes a schema the worker does not yet have.

12.4 Migrations

The schema authority is db/migrations/*.sql — ordered, up-only, plain SQL. The ORM maps to that schema; it never generates it. The invariants in this design live in objects Django's migration autogeneration would never write, in two distinct groups that matter for what follows. Group A — partial unique indexes with WHERE clauses, expression indexes on lower(), CHECK constraints with regexes, GiST EXCLUDE constraints — Django cannot autogenerate but can declare, in Meta.constraints and Meta.indexes. Group B — plpgsql triggers, DEFERRABLE INITIALLY DEFERRED constraint triggers, generated columns, RANGE partitions, and the column-level GRANTs that make ats_result and audit_event append-only — Django cannot express at all, at any layer. The distinction is the whole basis of the ruling: Group A stays visible to Django's own drift check, Group B needs a different check entirely.

Reconciling this with the Django choice — flagged as a cross-document dependency in _decisions.md Part 2 (risk on line 463) and resolved here decisively.

The mechanism below is the canonical text. It is the ruling ADR 0017 adopts, and 03-database-design.md §32.1, 04-integrations-and-processing.md §9.1 #2, 05-security-rbac-ai-governance.md §9.1 I-1 and 07-implementation-plan.md §17 #1 quote it verbatim rather than paraphrasing it — four near-identical paraphrases is how the managed flag ended up pointing in two opposite directions across five documents. The decision is a mechanism, not a flag setting:

Migration authority ruling — canonical text (ADR 0017). Quote it; do not paraphrase it.

db/migrations/NNN_*.sql is the schema authority. Every Django migration is SeparateDatabaseAndState(database_operations=[RunSQL(<that file>)], state_operations=[…]), so Django owns ordering and the applied-state ledger and authors no DDL. Every model stays managed = True; managed = False is used on no table, because it would remove exactly the tables that carry invariants from the one gate watching them. makemigrations --check compares models against declared migration state — never against the live database — so it is kept as the model-vs-state gate, and it is kept quiet not by a flag but by declaring every object Django can model in Meta.constraints / Meta.indexes (CheckConstraint, UniqueConstraint(condition=…), Index(Lower(…)), ExclusionConstraint) and mirroring those same declarations in state_operations. Objects Django cannot model at all — triggers, column-level GRANT/REVOKE, RANGE partitions and their attach/detach, generated columns, DEFERRABLE INITIALLY DEFERRED constraint triggers, and procrastinate's vendor-managed migrations — are named in an explicit, reviewed db/schema-ignore.toml, and are covered instead by a second, SQL-level gate: CI builds a database by running every migration, captures pg_dump --schema-only --no-owner plus a catalogue query for triggers and column privileges, and diffs that against the committed expected dump; any difference fails the build, and updating the expected dump is a reviewed part of the migration PR. Two gates, two failure modes, neither one silently lying: the ORM gate catches a model that has drifted from state, the SQL gate catches a database object that no migration created — or that a migration created and nobody reviewed. Signed off as ADR 0017 before migration 001 is written; it restates the mechanism already binding in adr/0002-primary-relational-database.md §3.

Which gate owns which object, so neither developer has to guess:

Object Declared in Meta In state_operations Watched by
Column, type, nullability, FK, plain index yes (implicitly, by the field) yes makemigrations --check
CHECK including regex (CheckConstraint) yes yes makemigrations --check
Partial unique index (UniqueConstraint(condition=…)) yes yes makemigrations --check
Expression index on lower() (Index(Lower(…))) yes yes makemigrations --check
GiST EXCLUDE (ExclusionConstraint) yes yes makemigrations --check
Trigger (history, immutability, audit hash chain, search index) no no pg_dump diff + catalogue query
Column-level GRANT/REVOKE (append-only ats_result, audit_event) no no catalogue query on information_schema.column_privileges
RANGE partition, ATTACH/DETACH no no pg_dump diff
Generated column no no pg_dump diff
DEFERRABLE INITIALLY DEFERRED constraint trigger no no pg_dump diff
procrastinate vendor migrations n/a — vendor-managed n/a ignore-listed explicitly, never by omission

Consequence stated plainly: the team gives up makemigrations autogeneration and pays for a second CI gate and a committed schema dump. In exchange the schema is reviewable as a SQL diff and every object has exactly one gate watching it. With one junior developer and constraints that are the design, that trade is correct.

Expand/contract, mandatory, because there are no down-migrations:

Step Deploy Rule
Expand N Add nullable columns, new tables, new indexes CONCURRENTLY. Never drop, never rename, never tighten
Migrate data N or N+1 Backfill in batched background jobs, never in a migration — a long-running migration blocks the deploy and can lock a table
Contract N+2, at least one deploy later Drop the old column only after no running code references it

Rules that hold without exception: every migration is reviewed by Talha; migrations run as a dedicated migration role, never the application role (which lacks DDL and lacks UPDATE and DELETE on append-only tables); CREATE INDEX CONCURRENTLY on any table with real volume; recovery from a bad migration is forward-fix plus PITR, never a down-migration.

12.5 Backup and recovery

Asset Mechanism Retention RPO RTO
PostgreSQL Managed automated backup + WAL archiving + PITR 14 days PITR, monthly full retained 12 months ≤ 5 min 12 h for a full restore; minutes for a managed failover
Blobs — candidate-documents Blob versioning OFF, deliberately (ADR 0003 §1). The only recovery path is the soft-delete window 7-day soft delete, disclosed in the retention policy Near zero — blobs are write-once Minutes inside the window; unrecoverable after it
Blobs — intake-quarantine, document-derivatives, exports No versioning, no soft delete. Recovery is re-derivation, not restore: quarantine bytes stay re-parseable for 30 days, derivatives are regenerable from the original, an export is re-runnable from its report_run row Lifecycle deletion only — quarantine 30 days after promotion or rejection; derivatives 90 days after last access; exports 7 days, no exceptions n/a — none of these is a system of record Regenerate, do not restore
Blob storage account Geo-redundant replication (GZRS, ASSUMPTION). Protects against region loss only, not against deletion — a deleted blob is deleted in both regions Hours, via region failover — see the region-loss row below
Closed audit partitions (audit-archive) Monthly export to a write-once immutable container (Object Lock, time-based) with the partition's final row_hash recorded 7 years (ASSUMPTION — legal to confirm); container policy ≥13 months then policy-driven 1 month Hours
Secrets Key Vault soft delete + purge protection 90 days Minutes
Code and migrations Git, plus every image tagged by commit sha in the registry Indefinite Minutes

Why the blob rows are deliberately weaker than they could be. Versioning plus a long soft-delete window is the standard safety configuration for object storage, and this design declines it for candidate-documents on purpose — the reasoning is recorded in adr/0003-object-storage-strategy.md §1 and its Consequences, which is the authoritative source for per-container policy. Retention purge (03-database-design.md §31.2) promises to delete CV bytes, and blob versioning or a 90-day soft-delete window would silently keep a recoverable copy of a document we have told a data subject was erased, making REQ-DAT-03 / REQ-DAT-06 false without anyone noticing. The accident-recovery margin is therefore 7 disclosed days, and no more.

Restore across the two stores is non-atomic, and that is accepted. A PostgreSQL PITR to time T does not resurrect blobs deleted after T; the database will hold candidate_document rows whose stored_file.deleted_from_store_at is stamped and whose bytes are gone. The DR runbook states this explicitly rather than letting the operator discover it mid-incident: after any PITR past a purge run, reconcile stored_file against the container and mark unrecoverable documents, do not retry the fetch.

Restore drills. A staging PITR restore to an arbitrary timestamp, quarterly, timed and written up. An untested backup is a belief, not a backup — and with two developers, the person who would perform the restore under pressure is the same person who must have practised it.

Documented recovery scenarios:

Scenario Procedure
Bad deploy Revision rollback (§12.7)
Bad migration, no data loss Forward-fix migration
Bad migration with data corruption PITR to just before it, into a new instance; verify; repoint; replay intake from the frozen delta cursors, which is safe because landing is idempotent
Accidental bulk delete Soft delete means the rows are still there — restore is an UPDATE deleted_at = NULL, audited
Region loss Restore from geo-redundant backup into a second region. Accepted RTO: hours. There is no warm standby, deliberately
Suspected audit tampering Compare live partitions against the immutable export and recompute the hash chain

12.6 Health checks

Endpoint Checks Used by Rule
GET /healthz Process is alive. No dependency checks Liveness probe, external uptime monitor Must never touch the database — a database blip must not trigger a restart storm
GET /readyz Database SELECT 1, Redis ping, blob HEAD, applied-migration version matches the image's expected version Readiness probe, deploy gate Fails → the replica is pulled from rotation but not killed
GET /internal/queue-health Per-queue depth, oldest job age, worker heartbeat age, breaker state Ops dashboard, alerting Admin-authenticated
GET /internal/version Commit sha, image tag, migration version Incident triage Admin-authenticated
Worker heartbeat row Each worker writes a heartbeat every 30 s queue-health, alerting A worker that stops heartbeating is presumed dead; its unacknowledged jobs re-run

12.7 Rollback

Situation Action Preconditions
Bad application code, schema unchanged Shift 100% of traffic back to the previous Container Apps revision. Seconds The previous image is still in the registry — keep the last 10
Bad code after an expand migration Same revision rollback. Safe because expand-only migrations are additive and the previous code ignores the new columns The expand/contract discipline in §12.4 was followed. This is the entire reason it is mandatory
Bad code after a contract migration No clean rollback. Forward-fix, or PITR Which is why contract steps are separated by at least one deploy and reviewed as the riskiest migration class
Bad frontend only Redeploy the previous static bundle from the CDN. The API is untouched Bundles are content-hashed and retained
A single bad AI prompt or model config Deactivate the PromptTemplateVersion or ModelConfigVersion row — no deploy at all. This is why they are versioned data rather than code The versioning pattern in _decisions.md
A bad scoring config Deactivate the ScoringConfigVersion; previously displayed scores do not change, because every ApplicationScore pins its config, model, algorithm and parse versions Append-only scores
A bad merge undo_merge(), in reverse seq order under stack discipline The CandidateMergeOperation log recorded every re-parent

Rollback rule of thumb, written down because it is the one people forget: the previous revision must always be able to run against the current schema. That single rule is what makes expand/contract non-negotiable and what makes rollback a 30-second operation instead of an incident.


13. Architecture Decision Record summary

Full ADRs live in docs/architecture/adr/, one file per decision, numbered NNNN-slug.md. Those files are the authoritative register. This table is the index and the one-line rationale; it deliberately does not duplicate the alternatives analysis, and every row below corresponds to a file that exists.

ADR File Decision Status Consequence if reversed
0001 0001-modular-monolith-versus-microservices.md Modular monolith, one Django 5 / DRF codebase, one image, exactly two processes (web, worker) with named hard and soft split triggers (§10.3), one PostgreSQL Accepted Reversing to services requires either per-service databases (forbidden) or a shared database (the anti-pattern). A third deployable duplicates the data model, auth layer and deploy surface for two developers
0002 0002-primary-relational-database.md PostgreSQL 16+ as the single primary relational database; schemas app/ref/audit/ai/staging; Phase 1 extensions pg_trgm, unaccent, btree_gist, pgcrypto Accepted Every load-bearing invariant — partial unique indexes, DEFERRABLE constraint triggers, GiST EXCLUDE, column-level GRANTs — moves into application code, which is the prototype's demonstrated failure mode
0003 0003-object-storage-strategy.md Document bytes live in object storage; the database holds only the index (object_store_key, sha256, virus_scan_status); upload → quarantine → scan → promote Accepted Bytes in the database inflate every backup and PITR window, and erasure loses the one place where "delete the blob, keep the audit row" is expressible
0004 0004-background-job-queue.md procrastinate, a PostgreSQL-backed queue; Redis is cache and rate-limit only Accepted Losing transactional enqueue reintroduces the dual-write bug class in the one flow that must never lose data
0005 0005-email-integration-method.md Microsoft Graph delta polling of the careers mailbox as inbound channel #1, with IMAP/SMTP as the named fallback Proposed — assumption-dependent on IT confirmation of A1A3; the design is settled, the premise is not Option B (IMAP/SMTP) is promoted with no change to the surrounding intake design; only the identity and co-location benefit is lost
0006 0006-candidate-search-strategy.md PostgreSQL FTS + pg_trgm over a trigger-maintained candidate_search_index in Phase 1; pgvector in the same database for Phase 2 semantic search Accepted A search cluster adds a permanent dual-write and reindex-drift cost, ~4 orders of magnitude before it is needed
0007 0007-job-and-scoring-versioning.md Jobs, requirements and scoring configs are immutable version rows with UPDATE revoked; ats_result pins six versions per score Accepted A displayed historical score drifts when a config is edited, and "same candidate, same role, same score" (BRD §6.2) becomes unverifiable
0008 0008-candidate-duplicate-resolution-strategy.md Duplicate detection with six signals, mandatory human review, additive-re-pointing merge and a reversible per-operation undo log Accepted Merges become destructive; a wrong merge is unrecoverable and the losing candidate's public_id, already in sent email, stops resolving
0009 0009-permission-enforcement-strategy.md Application service layer (iam.can() / iam.scope()) as the primary authorization boundary; RLS as Phase 2 defence in depth Accepted Authorization re-derives per view, and the constraint that the chatbot never bypasses access control loses its single chokepoint
0010 0010-chatbot-controlled-query-architecture.md Allowlisted typed tool intents with a required_permission_key; no generated SQL, no chatbot service account Accepted Text-to-SQL or a service account makes the access-control bypass structural rather than merely possible
0011 0011-ai-provider-abstraction-and-versioning.md One narrow AiProvider port; all model access through ai_orchestration; the invocation row is written before its result is usable; human actor propagated; suggestions never write domain state Accepted, assumption-dependent on A5 (BRD OQ-1 unresolved) "AI never auto-rejects" and "the chatbot never bypasses access control" revert to policy statements
0012 0012-deployment-topology.md One image, two revisions, one managed container platform, one managed PostgreSQL. Azure Container Apps + Flexible Server + Blob + Key Vault + Entra ID Accepted, assumption-dependent on A1 Cloud-portable; the topology is unchanged and only the M365 co-location benefit is lost
0013 0013-frontend-strangler-migration.md Frontend strangler migration to Vite + TS + React 18; css/styles.css content-frozen as the design contract; js/charts.js retained behind one wrapper Accepted Retaining the prototype rendering layer keeps the _repo-findings.md §E XSS exposure as a per-line discipline forever, and the complex forms still to be built stall a two-person team
0014 0014-phase-0-xss-csp-hardening.md Phase 0 XSS/CSP hardening of the prototype, independent of the migration, 23 developer-days Accepted Decouples the security deadline from the migration schedule; reversing bets that the migration never slips and that nobody ever points the prototype at real data
0015 0015-module-boundary-enforcement.md import-linter layered contracts plus a forbidden-import contract as the boundary enforcement mechanism, failing the build Accepted Boundaries become convention; the prototype's window namespace is the demonstrated failure mode, and "AI never auto-rejects" loses its structural guarantee
0016 0016-real-postgres-in-ci.md Real PostgreSQL service container in CI, never SQLite; constraint tests assert that forbidden writes raise Accepted The team starts avoiding the Postgres features the design depends on, and the suite stops telling the truth
0017 0017-plain-sql-migrations-as-schema-authority.md Plain-SQL migrations under db/migrations/ are the schema authority; Django runs them via SeparateDatabaseAndState(RunSQL + state_operations); managed = True everywhere; two drift gates — makemigrations --check for models vs state, a pg_dump/catalogue diff for everything the ORM cannot express, with an explicit db/schema-ignore.toml. Adopts the canonical text in §12.4 verbatim Proposed — resolves the Part 1 / Part 2 collision in §15 (I1). Must be signed off before migration 001 Either the invariants move into ORM escape hatches and drift, or Django's migration idioms are fought every sprint. Reverting to managed = False on invariant-bearing tables would blind both gates on exactly the tables that matter
0018 0018-backend-language-and-framework.md Python 3.12 / Django 5 / DRF, psycopg3, drf-spectacular; async only for the model-streaming endpoint. Language decided by the CV-parsing and fairness-evaluation ecosystems; framework decided by team shape Accepted Loses first-class PDF/DOCX/OCR, rapidfuzz, pgvector and the free admin back-office; the 7 controlled vocabularies would need a Phase 1 Settings UI, and auth, permissions and the migration workflow become a month of hand-built platform work

One residual gap, stated rather than papered over. It has no file of its own and should get one before Phase 1 closes:

Decision Where it is currently argued Why it still needs its own ADR
The four cross-cutting persistence patterns — versioning, history, append-only scores, money-plus-currency — as one binding ruling ADR 0007 covers the versioning third; the rest live in _decisions.md Part 2 and 03-database-design.md Retrofitting history means backfilling data that was never captured — the one thing that cannot be fixed later — so the pattern, not just its first application, deserves a decision record

Numbering rule (binding). The adr/NNNN-slug.md filenames are the register. There is no parallel ADR-001ADR-013 series — an earlier revision of this section carried one, and four of its thirteen entries pointed at documents that did not exist while five real ADRs were absent from the index. A new decision takes the next free file number; a decision cited as an ADR anywhere in the package must have a file (07 T-08).


14. Assumptions register

Every item is an ASSUMPTION, not a finding. Each names what would change if it is wrong.

# Assumption If wrong
A1 Utopia Brands runs Microsoft 365, making Entra ID SSO and Graph co-located ADR 0012 (deployment topology) changes cloud and ADR 0005 (email) falls back to IMAP/SMTP; SSO becomes a separate integration. Architecture unchanged
A2 Peak concurrency 2025 of 66 seats Web replica count and database tier change. No structural change below roughly 10×
A3 20k60k applications/year; 200600 documents/day at peak; <1 TB blobs in year one Worker sizing and the T2 hardware trigger re-derive. Bulk job-board feeds could be 12 orders higher
A4 10^410^5 candidate rows over several years The T8 search threshold re-derives; a replica may become necessary earlier
A5 Model hosting is a contracted API provider under a DPA (BRD OQ-1, unresolved) If legal requires self-hosting, Phase 1 gains GPU infrastructure, model serving and MLOps that two developers cannot absorb. The single decision most likely to break this design
A6 Audit retention of 7 years satisfies legal Partition archive policy changes; sizing re-derives
A7 Business-hours-only availability is acceptable, so a single app host with no HA is acceptable HA means a second replica plus session affinity review, and a database HA tier. Cost roughly doubles
A8 A recruiter review step for low-confidence parsed fields is acceptable (BRD §11 asks for no re-keying; BRD OQ-6 unresolved) If unqualified accuracy is demanded, the expectation cannot be met by engineering and must be renegotiated
A9 Historic hiring outcome data exists for fairness evaluation (BRD OQ-2, unresolved) The Phase 3 activation gate could block the ranking release with no engineering fix available
A10 A single region satisfies six jurisdictions (BRD OQ-4, unresolved) Direct conflict with the one-database constraint; needs an explicit business exception, not an architectural workaround
A11 Roughly 25 named seats need write access; interviewers are read-plus-scorecard only Permission model unchanged, sizing changes

15. Inconsistencies with _decisions.md to reconcile

Recorded rather than silently resolved, per the instruction at the top of _decisions.md.

The binding resolution for all seven is in _open-items.md. This table is the evidence and the position this document took; the ruling is there, and where the two differ the ruling wins. I1 → RULING-02 (whose ADR was written as adr/0017-plain-sql-migrations-as-schema-authority.md; RULING-02's own text proposed 0013, which was already taken by the frontend strangler migration, so 0017 stands and §13 is the authoritative index), I2 → RULING-07, I3 → C-02, I4 → RULING-03 (_glossary.md is the published glossary this row asks for), I5 → C-03, I6 → C-09 (a risk, carried as 03 §32.2 R8), I7 → RULING-04.

# Inconsistency Where Severity Position taken in this document
I1 Migration tooling. Part 1 selects Django partly because "migrations are built in" and puts makemigrations --check in CI. Part 2 mandates ordered plain-SQL migrations, states the ORM "never generates" the schema, and explicitly rejects Django autogenerated migrations. Part 2's own risk list (line 463) flags this as a cross-document dependency Part 1 §"Backend language", §"Testing, CI"; Part 2 §"Schema tooling" High — it changes daily developer workflow and who owns the schema Settled by the canonical migration authority ruling in §12.4, which every other document in this package quotes verbatim: plain SQL is the authority, Django is the runner via SeparateDatabaseAndState, managed = True on every model (managed = False on nothing), and drift is caught by two gates — makemigrations --check for models-vs-state, and a pg_dump/catalogue diff for the triggers, grants, partitions and generated columns the ORM cannot express. The flag was never the decision; the second gate is. Needs ADR 0017 (adr/0017-plain-sql-migrations-as-schema-authority.md, currently Proposed) signed off before migration 001
I2 pgvector phase. Part 1's deployment topology enables pgvector on the Phase 1 database. Part 2 lists pgvector as "Phase 2 only" Part 1 §"Deployment topology"; Part 2 §"Database engine" Low Read as compatible: the extension may be installed in Phase 1 provisioning, but no Phase 1 feature uses it. Worth one sentence in the provisioning runbook so nobody assumes semantic search is available
I3 Module count. The Part 1 summary paragraph says "26 modules with a junior"; the module tables and the consolidation section both land on 25 plus one API layer Part 1 §"Backend language" vs §"Modules consolidated" Low, cosmetic 25 modules plus one cross-cutting API layer, used consistently in §4
I4 Entity naming. Part 1 uses InboundSubmission, ProcessingAttempt, Requisition/RequisitionVersion, MergeOperation. Part 2 uses raw_intake, intake_parse_attempt, job/job_version, candidate_merge_operation Part 1 module tables vs Part 2 throughout Medium — two vocabularies in one design will produce two sets of names in code §4 uses Part 2's physical table names for entities and keeps Part 1's module names. A single glossary should be published before Phase 1 code is written; requisition vs job is the one most likely to cause real confusion, since Part 2's uniqueness rules key on job_id
I5 Search index ownership. Part 1 gives candidate.search() "Postgres FTS + trigram"; Part 2 requires a separate trigger-maintained candidate_search_index table because a generated tsvector column cannot see child tables Part 1 module 7; Part 2 §"Phase 1 search strategy" Low §4.2 lists CandidateSearchIndex as a candidate-owned entity. No conflict once stated
I6 RLS timing. Part 1's AI boundary implies the human-actor check is the whole enforcement. Part 2 defers RLS to Phase 2 and flags that Phase 1 PII protection therefore rests entirely on a brand-new authorization layer with no database backstop Part 1 §"AI boundary"; Part 2 §"Chatbot isolation", risk line 464 Medium, and it is a risk, not a contradiction Carried forward as a risk. The mitigation is concrete: one centralised authorization module, no direct repository access from views, and a test asserting every candidate-reading endpoint passes through iam.can()
I8 CandidateProfileVersion does not exist — resolved, not open. Part 1's module 7 lists CandidateProfileVersion ("from a parsed doc") among the candidate module's entities, and diagram 4 in §7 of this document had the step "apply_parsed_profile as a new CandidateProfileVersion". 03-database-design.md §30.3 lists it under "Named but deliberately not created": a whole-profile version row would duplicate intake_parse_attempt.parsed, which already is the immutable parsed profile, and the question the version was actually for — "where did this candidate's job title come from, and how confident were we" — is per-field, not per-profile Part 1 module 7; this document §4.2 module 7 and §7 diagram 4; 03 §13.2, §30.3 Resolved — was a stale entity in two places Part 2 wins. §4.2 module 7 now names CandidateFieldProvenance (app.candidate_field_provenance, 03 §13.2) and diagram 4's step reads "apply_parsed_profile — writes candidate_field_provenance rows per field, source parse attempt pinned". 03 §32.1 row 3 anticipated exactly this failure — "confirm the Part 1 module entity list is updated, or the API document will describe an entity that does not exist" — and this document was the one that had not been updated; 06-api-boundaries.md already correctly avoids the entity. Recorded here so a reader can see it was decided rather than overlooked
I7 Assignment table shape — resolved, not open. Part 1's module table defines a single polymorphic Assignment(subject_type ∈ requisition/application, subject_id, …). Part 2 explicitly rejects it and mandates two concrete tables, job_assignment and job_application_assignment, because a polymorphic subject FK cannot be enforced by the database at all — and unenforceable references to jobs and candidates are precisely what this design exists to eliminate Part 1 module 12; Part 2 §"Which entities get current-plus-history". Also 00-scope-classification.md §10 row 2 ("Part 2 wins"), 03-database-design.md §16 and §30.3, 06-api-boundaries.md §9.1 #3 Resolved — was a direct contradiction, now closed Part 2 wins. §1 and §4.2 module 12 in this document have been corrected to name JobAssignment and JobApplicationAssignment as two concrete tables with the GiST EXCLUDE overlap constraint and the partial unique index on the current primary recruiter. The module boundary is unaffected: one assignment service facade and one permission surface present both tables, so the module inventory still stands at 25. Earlier revisions of this document carried Part 1's polymorphic shape verbatim, which is how the stale definition survived three other documents rejecting it — recorded here so a reader can see it was decided rather than overlooked

Nothing in this document diverges from _decisions.md Part 2 on schema shape. Against Part 1, the divergences are exactly I1I8 above; I7 and I8 are closed in Part 2's favour and the rest are open. Outside those, nothing here diverges from Part 1 on architecture style, process count, queue technology, module inventory, phasing, or the deployment topology.