24 KiB
24 KiB
ADR 0012 — Deployment topology: one image, two revisions, one managed container platform, one managed PostgreSQL
Status: Accepted — 2026-07-29 Assumption-dependent: the cloud recommendation rests on assumption A1 (Utopia Brands runs Microsoft 365). The topology does not — AWS or GCP equivalents substitute directly with no structural change.
Deciders: Talha Ahmed (infrastructure definition, the CI pipeline, secret store wiring,
production promotion, restore drills). Ahmed Mujtaba (the local docker compose stack and the
pii_classification-driven fixture generator, /healthz and /healthz/integrations, the
Playwright smoke journeys, the orphan-blob sweep report) — each independently demonstrable, each
with a Talha review checkpoint.
Context
The repository provides no deployment inputs at all
| Absent | Evidence |
|---|---|
Dockerfile, docker-compose.yml, Makefile |
findings §B |
Any CI configuration — .github/ does not exist |
findings §B |
.env, .env.example — none present; .gitignore merely anticipates them |
findings §B |
Any build tooling (tsconfig.json, vite.config.js, webpack.config.js) |
findings §B |
| Any server-side code, API route, controller or service | findings §B |
| Any migration directory or migration tool, any ORM or database driver | findings §B |
The only server-shaped artefact is devserver.py, 36 lines of no-cache static file server
added for local preview. .gitignore contains a Python section and .env rules but no Node
section; findings §H states explicitly that this postdates devserver.py and is
weak/ambiguous evidence that must not be read as stack intent. It is therefore not a decision
input here.
So this is entirely greenfield. Nothing constrains the topology except the actual population, the two-developer team, and the hard constraints.
The population this has to serve
| Dimension | Figure | Basis |
|---|---|---|
| Named seats | 66 | BRD §4 (2+4+15+12+8+24+1) |
| Peak concurrent users | 20–25 | ASSUMPTION A2 — 24 of the 66 are interviewers who touch only their own interviews |
| Applications per year | 20k–60k | ASSUMPTION A3 |
| Documents per day at peak | 200–600 | ASSUMPTION A3 |
| Blob volume, year one | well under 1 TB | ASSUMPTION A3 |
| Candidate rows over several years | 10^4–10^5 | ASSUMPTION A4 |
| Queue throughput | hundreds of jobs/hour | derived |
These are small numbers. The honest consequence is that almost every interesting deployment question here is about operability by two people, not about scale.
Constraints that bind the topology
No Kubernetes. One relational database, no per-region databases, not multi-tenant. No separate
AI service in Phase 1. Two developers, one junior, one reviewer. Two runtime processes are
already required by ADR 0001: web is I/O-bound and sub-second, worker is CPU-bound and
multi-second, and a scanned CV must never occupy a request thread. The queue is Postgres-backed
(procrastinate), so the queue is the database and the worker is woken by LISTEN/NOTIFY —
which has a direct consequence for autoscaling.
Options considered
Option A — Managed container platform, one image, two revisions (chosen)
Azure Container Apps (or the AWS/GCP equivalent): two revisions from one image, differing only by entrypoint.
| Pros | Cons |
|---|---|
Independently scalable web and worker with rolling deploys and log aggregation, and no cluster to operate — which respects the no-Kubernetes constraint while keeping the process split |
Platform-specific revision and scaling semantics; moving cloud is roughly a week of work, not a day |
| One image means the two processes cannot drift in dependencies, and a deploy is atomic across both | Scale-to-zero must be explicitly disabled for the worker, or a LISTEN/NOTIFY consumer idles out and the queue silently stops draining — an easy and expensive mistake |
| Managed TLS, CDN and platform ingress; no load-balancer tier to own | Single app host in the recommended sizing means no HA (see the accepted limitation below) |
| Priced per replica at a small footprint; the whole environment is cheap at this population | Vendor-managed autoscaling is a black box when it misbehaves |
Phase 2's worker-untrusted split is a third revision of the same image — no new infrastructure |
Option B — Kubernetes (AKS/EKS/GKE)
| Pros | Cons |
|---|---|
| Genuinely the industry standard, with per-workload resource isolation, mature secret and config handling, real pod security contexts (which would suit untrusted parsing well), and portability across clouds | Forbidden by constraint, and independently unjustifiable: a cluster is an operational product that needs upgrades, node pools, ingress controllers, RBAC and observability wiring |
| A future parsing sandbox (gVisor, seccomp profiles, dedicated node pool) is natural | Two developers, one junior, one reviewer. The cluster becomes the senior developer's unpaid second job, competing directly with the Phase 1 critical path |
| Horizontal scaling and rollout primitives are better than any PaaS | Nothing in the measured population needs it — 25 concurrent users and hundreds of jobs/hour |
Option C — One VM (or one PaaS app service) running both processes
docker compose or systemd on a single host, or an App Service instance running web and worker
side by side.
| Pros | Cons |
|---|---|
Cheapest option, and the simplest mental model — one machine, ssh, docker ps |
A worker OOM or a runaway OCR job takes the web tier down with it. That is the precise failure isolation the two-process split exists to provide, and this option gives it back |
| No platform semantics to learn; every debugging technique is the familiar one | Deploys are not rolling: a restart is visible downtime on every deploy, not just on platform maintenance |
| Total control over process resource limits and OS users, which suits untrusted parsing | Patching, OS upgrades, TLS renewal and log shipping all become manual work owned by one person |
Scaling web and worker independently is impossible without splitting the host anyway |
Option D — Serverless: functions or container jobs for the worker, PaaS for web
| Pros | Cons |
|---|---|
| Scale-to-zero economics, no idle worker cost — attractive given genuinely bursty document volume | Cold starts on interactive AI paths, and an execution ceiling (commonly ~10 min) that is wrong for OCR batches and rescore fan-outs |
| No process supervision to own | A Postgres LISTEN/NOTIFY consumer is a long-lived connection holder — fundamentally at odds with an ephemeral invocation model. The queue would have to become polling, losing the wake-up latency benefit |
| Per-job isolation is a real security benefit for untrusted parsing | No shared connection pool; connection churn against a 2-vCPU database is a real risk |
| Two runtime models to debug instead of one; a junior would own neither confidently |
Option E — Self-managed PostgreSQL on a VM
| Pros | Cons |
|---|---|
Cheaper, and full control over extensions, version pinning and configuration — including pgvector and any BM25 extension without waiting for managed-service support |
Backup verification, PITR, patching, failover and connection-pooling operations become the senior developer's second job |
| No managed-service version lag, which matters because UUIDv7 generation differs by major version | The database is the only stateful component in this design and the single point of total failure. Managed PITR is the difference between a recovery command and a runbook the team has to write and rehearse |
| An unrehearsed restore is not a backup |
Rejected. The one stateful component is exactly the one to buy rather than build.
Decision
Per environment
| Component | Sizing / configuration |
|---|---|
| Managed container platform | ONE image, two revisions. web: uvicorn/ASGI, 2 vCPU / 4 GB, 2 workers × 4 threads, autoscale 1–4 replicas. worker: procrastinate consumer, 2 vCPU / 4 GB, concurrency 4, autoscale 1–3 replicas, minimum replicas 1 — scale-to-zero disabled so the LISTEN/NOTIFY consumer never idles out |
| Managed PostgreSQL 16 (target 17) | 2 vCPU / 8 GB, PITR, 14-day backups. Extensions Phase 1: pg_trgm, unaccent, btree_gist, pgcrypto; Phase 2: pgvector, partition management. Schemas app, ref, audit, ai, staging. Server and application role timezone = 'UTC' |
| Object storage | CV and letter blobs; versioning and soft delete on; lifecycle rules aligned to retention classes |
| Immutable / write-once container | Closed audit_event partitions exported nightly with the partition's final row_hash recorded — the one genuinely independent tamper check |
| Managed Redis | Cache, rate limiting, sessions, and the HMAC replay-nonce cache. Never a broker, never a store of record |
| Secret store | Graph credentials, AI provider key, database password. No secret in an image or an environment file in the repository |
| CDN / platform edge | TLS, WAF, serves the built React bundle |
| Observability | Structured JSON log workspace, request-id correlated; Sentry self-hosted or EU region with a scrubbing config derived from pii_classification; an external uptime probe hitting /healthz from outside the platform |
No read replicas. No load-balancer tier beyond the platform's own. No Kubernetes. No third deployable in Phases 0–4.
Recommended cloud
Azure — Container Apps, Database for PostgreSQL Flexible Server, Blob Storage, Key Vault, Entra ID for SSO. ASSUMPTION A1: Utopia Brands runs M365, since Outlook is inbound channel #1. If true, Entra ID SSO and the Graph app registration land in the same tenant and the identity problem largely disappears — the single largest free reduction in integration risk available. If false, the architecture is unchanged and AWS or GCP services substitute directly.
Storage portability is handled at the port, not the platform: ObjectStore is a Protocol whose
semantics are the S3 subset every provider supports, with AzureBlobObjectStore,
S3ObjectStore (AWS or MinIO) and FakeObjectStore adapters. Azure Blob is not S3-API
compatible, and that is fine because nothing above the adapter knows.
Environments — three, not six
| Environment | Composition | Data | Integrations | Promote |
|---|---|---|---|---|
| local | docker compose: web, worker, postgres:16, redis, MinIO/Azurite |
Anonymised fixtures generated by a script that reads pii_classification. Never a production copy |
Graph and AI mocked by default; real credentials opt-in via .env.local |
— |
| staging | Same image, one replica each, smaller database tier | Anonymised fixtures plus real test-mailbox traffic | Real integrations against a dedicated test mailbox and sandbox job-board accounts | Auto-deploy on merge to main |
| production | web 1–4, worker 1–3, PITR enabled | Real | Real | Manual promote, Talha only |
No per-developer cloud environment. Two developers do not need six environments; they need one that behaves like production.
Process boundaries
| Boundary | Separates | Why it is real, not taxonomy |
|---|---|---|
web / worker |
Sub-second I/O-bound requests from multi-second CPU-bound parsing, model calls, batch rescoring and sweeps | Different resource profile, failure mode and timeout budget |
worker-default / worker-untrusted (Phase 2) |
Trusted background work from parsing attacker-supplied files | Security, not scale. Same image, same codebase; different queue, restricted OS user, no outbound network, hard CPU and wall timeout, memory cap |
| Request path / streaming assistant | Request/response from a long-lived SSE connection | The streaming endpoint is the one async Django view under uvicorn; everything else is sync |
Region and residency
One region for the single database. Which region is a legal decision, not an architectural one — postings span six jurisdictions (BRD OQ-4, unresolved). Retention and deletion are implemented per record, including derived embeddings, not per region. If legal requires in-jurisdiction storage, that conflicts directly with the no-per-region-databases constraint and requires an explicit business exception. It is not solvable with topology, and attempting to solve it with topology would violate a hard constraint.
Delivery, health and rollback
- CI on GitHub Actions, one required pipeline: ruff, mypy,
import-lintercontracts, pytest against a real Postgres service container (never SQLite — the design depends on jsonb, partial unique indexes, FTS,pg_trgmand LISTEN/NOTIFY), a migration drift gate, ESLint withreact/no-dangeras an error, stylelint enforcing design-token usage,tsc --noEmit, Vitest, and 5 Playwright smoke journeys. Ephemeral MinIO/Azurite alongside Postgres; no external provider is contacted from CI — mail, AI, storage and scanner are faked at the port boundary. - Health checks distinguish liveness from readiness, plus
/healthz/integrationsreporting per-channel health, subscription expiry, scanner signature age and AI circuit-breaker state — one page a recruiter's admin can read before escalating. - Rollback is a revision pin, not a migration reversal. Down-migrations are not written; recovery is forward-fix plus PITR. Therefore every migration must be expand/contract-safe: additive deploy, then backfill, then a later contract migration, so pinning the previous revision is always safe. This is the operational consequence of the forward-only migration decision and the most likely thing to be forgotten under pressure.
Justification
- Sized to the actual population. One 2-vCPU web replica is generously provisioned for 20–25 concurrent users. 600 documents/day at ~30 s per scanned CV is roughly 75 worker-minutes spread over a day, on one worker with concurrency 4. Anything larger would be paying for a projection nobody has evidence for.
- The two-process split is bought without a cluster. Option A is the cheapest topology that keeps CPU-bound parsing off the request path, gives rolling deploys, and adds Phase 2's untrusted-parsing isolation as a third revision rather than new infrastructure.
- Co-locating with the identity provider and the mail source is free risk reduction. Outlook is inbound channel #1 and Entra ID is the SSO source; putting them in the same tenant removes an entire class of integration and credential problem at zero cost.
- The one stateful component is managed. Recovery is PITR rather than a runbook two developers have to write, rehearse and keep current.
- Three environments, not six. Every environment is a thing to configure, secure, pay for and keep in sync. Staging pointed at real integrations with a dedicated test mailbox is worth more than four half-maintained sandboxes.
Consequences
Positive
- Rolling deploys and independent scaling of
webandworkerwith no cluster to operate. - A worker OOM or a parser crash-loop cannot take the web tier down.
- Queue/worker outage loses nothing: work accumulates durably in Postgres and drains on recovery, because the queue is the database.
- Phase 2 untrusted-parsing isolation is a configuration change to a third revision of the same image.
- Portable in principle: the coupling surface is the
ObjectStoreadapter and Entra SSO, both behind ports. - CI is fully self-contained, so the pipeline is deterministic and costs nothing in provider calls.
Negative — the costs being accepted
| Cost | Detail |
|---|---|
| No high availability | A single app host means a platform-level restart is a few minutes of downtime. Accepted deliberately for an internal recruiting tool used in business hours (assumption A7) — but the six-jurisdiction user spread narrows the maintenance window, and this should be stated to the business rather than discovered during the first deploy that overlaps Singapore business hours |
| The database is a single point of total failure | When it is down, everything stops: web, worker, queue, sessions-in-Postgres. PITR is the recovery path. This follows directly from the one-database constraint and is accepted, not solved |
| Cloud coupling is real even behind ports | Container Apps revision semantics, Flexible Server parameters, Key Vault references and Entra SSO are Azure-shaped. A cloud migration is roughly a week, not a day |
| A frontend deploy is a backend deploy | The built React bundle is served as static files by the web process behind the CDN. A CSS-only change ships a new image. Acceptable at two developers; it would be wrong at ten |
| Forward-only migrations make rollback conditional | Pinning the previous revision is only safe if the migration was expand/contract-safe. The discipline is now load-bearing for rollback, and a non-additive migration silently removes the rollback path |
| Scale-to-zero is a trap that must stay disabled | A cost-optimisation reflex on the worker revision stops the queue draining, and the symptom (intake stuck in received) looks like a parser bug rather than a scaling setting |
| Talha is the only person who can promote to production | Deliberate, and a bus-factor cost stated plainly rather than papered over |
| Sentry scrubbing is load-bearing for PII | The scrubbing config is derived from pii_classification; if that derivation breaks, candidate PII flows to a third-party error store. It needs a test, not a convention |
Risks
| # | Risk | Likelihood | Mitigation |
|---|---|---|---|
| R1 | A1 wrong — Utopia Brands is not on M365 | Low-medium | Topology unchanged; the cloud changes and SSO becomes a separate integration with its own Phase 1 cost. Confirm A1 in Phase 0 before the first storage or identity migration is written |
| R2 | A7 wrong — availability outside business hours is required | Medium | HA means a second web replica with a session-affinity review plus a database HA tier: cost roughly doubles. Raise as a business decision, not an engineering surprise |
| R3 | A10 / OQ-4 — legal requires in-jurisdiction storage | Medium | Direct conflict with the one-database constraint. Requires an explicit written business exception. Do not attempt a topology workaround |
| R4 | Microsoft Graph dependency on corporate IT: Entra app registration, admin consent for Mail.Read, a dedicated recruiting mailbox | High — outside the team's control, can block the Phase 1 critical path for weeks | Start the request in Phase 0, ahead of any code that needs it |
| R5 | Untrusted-file parsing runs in our own worker process in Phase 1 — a real RCE and resource-exhaustion surface | Medium-high; the split trigger most likely to fire early | Phase 1: timeouts, memory caps, restricted OS user, no outbound network from the parse step. Phase 2: worker-untrusted revision. These are honestly weaker than a sandbox and are stated as such |
| R6 | Worker autoscaling misconfigured, or scale-to-zero enabled as a cost saving | Medium | Minimum replicas 1 asserted in infrastructure configuration; a queue-age alert (any intake unresolved > 48h) catches it regardless of cause |
| R7 | Cost overrun from an always-on worker plus Redis plus managed Postgres at a 66-seat scale | Low-medium | Small footprint by design; monthly spend reviewed against the baseline; V8 below is the trigger |
| R8 | An unrehearsed restore. PITR configured is not PITR proven | Medium | A restore drill into a scratch environment once per phase, timed, with the result recorded. An untested backup is a belief |
| R9 | Staging drifts from production (tier, extensions, parameters) and a migration passes staging then fails production | Medium | Same image, same extensions, same PostgreSQL major version; database tier is the only sanctioned difference |
| R10 | Audit archive growth: monthly partitions exported to write-once storage accumulate indefinitely | Low | Partitions older than 13 months are detached, compressed and archived; audit retention set independently of candidate retention; alert if a partition exceeds 2× projection |
Revisit conditions
| # | Trigger | Threshold | What changes |
|---|---|---|---|
| V1 | Web saturation | web CPU p95 > 70% or read endpoint p95 > 300 ms at 4 replicas after query and index tuning |
Vertical scale to 4 vCPU / 8 GB, then horizontal beyond 4 replicas. Follow the ladder: tuning → vertical → horizontal → worker split → replica → extract |
| V2 | Database saturation | PostgreSQL CPU p95 > 75%, or connections > 80% of the pool for 10 min, sustained a week | Vertical to 4 vCPU / 16 GB; then a connection pooler review; then a read replica for analytics and search |
| V3 | Volume beyond the sizing assumptions | Peak concurrency > 100, or documents/day > 2,000 sustained for a week, or blob volume > 5 TB, or applications/year > 200k | Re-derive the whole sizing table (A2/A3/A4) rather than inheriting it. This is the trigger that invalidates the arithmetic, not just the tier |
| V4 | Any hard split trigger fires (T1–T4) | Image > ~2 GB or a dependency conflict; parsing/inference needs a GPU or sustained > 4 vCPU / > 8 GB; untrusted-file handling needs a sandbox the worker cannot provide; a required model runtime is not Python | A third deployable becomes justified for that workload only. It does not reopen Kubernetes by itself |
| V5 | Blast radius (T7) | Worker OOM or crash-loops have taken the shared host down twice in a rolling quarter | Split the worker to its own host/plan ahead of schedule; if it recurs after the split, re-evaluate Option B for resource isolation |
| V6 | Availability requirement changes | A written requirement for availability outside business hours, or an RTO < 1 hour, or an RPO < 5 minutes | HA: second web replica with session-affinity review, database HA tier, and a rehearsed failover. Costs roughly double — treat as a funded change |
| V7 | Residency ruling | Legal decides in-jurisdiction storage is mandatory (OQ-4) | Escalate for a documented business exception to the one-database constraint. Do not shard by region |
| V8 | Cost | Monthly platform spend exceeds 2× the first full month's steady-state baseline without a corresponding volume increase from V3 | Audit replica counts, worker idle time, log retention and Redis tier before changing topology. Most cost surprises here are retention settings, not compute |
| V9 | Deploy friction | Deploys blocked or rolled back more than twice in a month, or any incident where a revision pin was unsafe because a migration was not expand/contract | Enforce expand/contract in CI with a migration-shape check; if that is insufficient, reconsider forward-only migrations in a superseding ADR |
| V10 | A second consumer needs database connectivity | Any BI tool, spreadsheet connector or external reporting product | Provision ats_report_reader with RLS on analytics views before enabling the connection (ADR 0009 V2). This is a topology change with an authorization precondition |
| V11 | Managed-service capability gap | A required extension or PostgreSQL major version is unavailable on the managed tier (e.g. a BM25 extension needed by ADR 0006's search escalation ladder, or native uuidv7()) |
Re-evaluate Option E for that environment only, with backup and patching responsibilities named and owned before the move |
Related
_decisions.mdPart 1 — deployment topology, environments and hosting; the two-process decision and its split triggers; the Postgres-backed queue; testing and CI._decisions.mdPart 2 — engine, extensions, schemas, UTC timezone, audit partitioning and the write-once archive.02-system-architecture.md§6, §10, §11, §12 — the topology diagram, the scaling ladder, triggers T1–T9 with the metric that fires each, SLOs, alerts and the promotion pipeline.04-integrations-and-processing.md§6–§8 — theObjectStoreport and adapters, encryption posture, retention classes, and the degradation ladder per external dependency.- ADR 0009 — the Phase 2 database roles (
ats_ai_reader,ats_report_reader,ats_support_readonly) are provisioned here. - ADR 0011 — worker sizing, the
aiqueue and the circuit breaker that keeps an AI outage from becoming a platform outage.