HR-ATS-Portal/docs/architecture/adr/0001-modular-monolith-versus-microservices.md

# ADR 0001 — Modular Monolith With a Separate Worker Process, Not Microservices

| | |
|---|---|
| **Status** | **Accepted** — 2026-07-29 |
| **Scope** | Backend runtime architecture for the Utopia Brands internal HR Recruitment & ATS platform |
| **Owner** | Talha Ahmed (senior) |
| **Consistent with** | `_decisions.md` Part 1 → *Architecture style*, *Module boundary enforcement and dependency rules*, *Which processes are separate*, *Deployment topology* |
| **Related ADRs** | 0002 (one PostgreSQL), 0004 (in-database queue — the mechanism that makes the two-process split safe) |

---

## Context

**The repository imposes no backend constraint at all.** Direct inspection (`_repo-findings.md` §B) confirms the absence of `package.json`, `requirements.txt`, `pyproject.toml`, `go.mod`, `Dockerfile`, `docker-compose.yml`, any migration directory, any ORM or database driver, any API route or controller, any test file, and any CI configuration. What exists is a static browser-only prototype: `index.html` (287 lines), `css/styles.css` (1269 lines) and 22 unbundled scripts totalling ~4,780 lines, with zero network calls anywhere in `js/` or `index.html` and all data generated in-browser by a seeded LCG (`js/data.js:8-10`).

The consequence is precise and worth stating so nobody re-litigates it later: **the "do not replace the existing stack" rule constrains only the frontend and design system, not the backend.** There is no backend stack to preserve. This ADR is therefore a greenfield choice bounded by three things — team size, actual load, and the platform's hard constraints.

### Forces

| Force | Evidence | Consequence for this decision |
|---|---|---|
| Two developers, one of them junior, one reviewer | `_repo-findings.md` §I | Operational surface per deployable is the dominant cost, not code elegance |
| 66 named seats; ~20–25 realistic peak concurrency (**assumption**) | BRD §4 (2+4+15+12+8+24+1); 24 of the 66 are interviewers who touch only their own interviews | No scale argument for distribution exists |
| ONE master data model, ONE relational database, not multi-tenant | Platform constraint | Microservices would either share a database (anti-pattern) or split the model against an explicit constraint |
| The Phase 1 critical path crosses four module boundaries transactionally | `intake → document_parsing → candidate → application → scoring`; `job_application.raw_intake_id` and `candidate.created_from_raw_intake_id` are both `NOT NULL` against non-deferrable FKs (see `_decisions.md` Part 2) | In-process this is one `COMMIT`; across services it is a saga with compensations |
| Two workloads with genuinely different runtime properties | Parsing a scanned CV is CPU-bound and multi-second; a recruiter request is I/O-bound and sub-second. 200–600 documents/day at peak (**assumption**) | They must not share a request thread — this is the one split that is earned on day one |
| Unenforced modularity has already failed once in this repo | 22 ordered `<script>` tags, everything on `window`, any file can reach any other (`_repo-findings.md` §C, `index.html:264-285`); the flat candidate array carrying `jobId`, `stage`, `aiScore` (`js/data.js:117-127`) | Boundaries must be machine-enforced, not conventional |
| Authorization must exist exactly once | Today the RBAC matrix is a display widget; clicking a cell mutates an in-memory array and nothing reads it (`js/rbac.js:78`, `js/rbac.js:83-85`); security settings are inert chrome (`js/settings.js:148-154`) | A single `iam.can()` chokepoint is a security requirement, especially for "the chatbot must never bypass access controls" |
| Forbidden without demonstrated need | Kubernetes, Kafka, microservices, Elasticsearch, multiple databases, a separate AI service in Phase 1 | The architecture must be defensible without any of them |

---

## Options considered

### Option A — Modular monolith: one codebase, one image, two processes

One Django project, 25 logical modules in five tiers, one container image, two runtime revisions differing only by entrypoint (`web`, `worker`), one PostgreSQL database.

**Pros**
- The four-module Phase 1 write path is a single database transaction. The raw-intake-before-candidate invariant is enforced by a `NOT NULL` FK rather than by a saga that must not be interrupted.
- One deploy pipeline, one log stream, one metrics dashboard, one backup story. Two people can actually hold this in their heads.
- One authorization implementation. `ai_orchestration.invoke()` calls the same `iam.can()` the REST API calls, with the human actor propagated — so the chatbot cannot see what the asking user cannot.
- Refactoring across module boundaries is a compile/type-check away, which matters because splitting candidate from application (`js/data.js:117-127`) will churn entity shapes through Phase 1.
- pgvector, FTS and trigram all live inside the same database, which is what makes "no separate AI service in Phase 1" practically achievable rather than aspirational.

**Cons (stated, not minimised)**
- Shared fate on deploy: a bad migration or a bad release stops recruiters *and* intake.
- One runtime and one dependency set for everything. An OCR or ML library that conflicts with a web-tier dependency blocks both.
- Horizontal scaling is coarse — you scale the whole image, not the hot module.
- Boundary discipline depends on CI tooling (`import-linter`) that somebody has to maintain and that a determined shortcut can be talked past in review.
- No independent module release cadence; prompt/model tweaks ride the domain release train.

### Option B — Microservices (a service per domain area)

**Pros — the real ones, not a strawman**
- Genuine failure isolation: an OOM in parsing cannot degrade the recruiter UI at all.
- Independent scaling and independent release cadence per service; the AI service could deploy daily while domain code is release-gated.
- Hard, unbypassable module boundaries — a network call cannot accidentally import another module's `models.py`.
- Heterogeneous runtimes become possible (a non-Python model runtime, for example).
- Onboarding a third and fourth engineer later has a clean ownership story.

**Cons**
- Operationally disqualifying at this team size: 15–20 pipelines, 15–20 dashboards and alert routes, inter-service contract versioning, distributed tracing, and a local development story that involves running most of the estate. Two developers cannot own that and also ship 25 modules.
- Directly collides with the platform constraints. One master data model plus one relational database means the services share a database — the textbook anti-pattern — or the data model is fragmented against an explicit rule. There is no third answer.
- Transactional integrity on the critical path degrades into sagas plus compensating transactions. With one senior reviewer, the realistic outcome is orphaned candidates and applications with no raw intake, which is the exact invariant the brief calls non-negotiable.
- Authorization gets re-derived in every service. Twenty places to get `can()` wrong is a worse security posture than one.
- It drags in the forbidden infrastructure (an orchestrator) as a near-inevitability.
- Zero load justification: 66 seats, ~25 peak concurrent users.

### Option C — Hybrid: monolith plus one extracted AI/parsing service

**Pros**
- Isolates the one genuinely different workload (CPU-bound, untrusted-file-handling, potentially GPU-bound later) behind a network boundary.
- Lets the AI/parsing component take a different dependency set and, eventually, different hardware.
- Keeps the domain model in one place, so it does not violate the one-database rule as long as the extracted service is stateless.

**Cons**
- The isolation it buys is *already available* as a process split inside one image and one codebase — same failure isolation for the web tier, without a second deploy surface, a second dependency graph, or an HTTP contract to version.
- It fractures the audit trail unless carefully engineered: `AiRun` rows must join to applications in the same database (BRD §6.2, §7.3), so an extracted AI service either writes to the shared database anyway or ships events that can be lost.
- Doubles the deploy and secret-management surface for two developers in exchange for a boundary CI can already enforce in-process.

**Verdict:** correct eventually, premature now. Kept as the *named first split* with measurable triggers (see Revisit conditions), which is materially better than either doing it now or leaving it undefined.

### Option D — Serverless functions per module

**Pros**
- No servers to size or patch; scale-to-zero is genuinely cheap for a system idle outside business hours across six jurisdictions.
- Per-function scaling suits bursty CV batches well.

**Cons**
- Cold starts land directly on the interactive AI/chatbot path, which is the most latency-visible feature in the product.
- No shared connection pool against a single managed Postgres — connection exhaustion is the standard failure, and this design's connection budget is already tight (see ADR 0004).
- Execution ceilings (typically ~10 minutes) are wrong for OCR batches and full-requisition rescoring.
- Debuggability is poor for a junior developer, and local development diverges sharply from production.
- Fragments the transactional write path exactly as microservices do.

### Option E — Single-process monolith with in-process background threads

**Pros**
- The simplest possible operational footprint: one process, one revision, nothing to coordinate. Cheapest hosting.
- No queue technology to learn.

**Cons**
- Puts multi-second OCR CPU load on the request path, degrading interactive latency unpredictably.
- No retries, no visibility, no job status. Work in flight is lost on every deploy — unacceptable when BRD §6.3 requires that no document is ever silently lost and BRD §8.3 requires retrievable async job status.
- No path to running untrusted-file parsing under a restricted OS user with no outbound network, which is a Phase 2 security requirement.

---

## Decision

**Modular monolith. One Django 5 / DRF codebase, one container image, exactly two runtime processes, one PostgreSQL 16 database.** Module boundaries are logical — Python packages with a public service facade — not network boundaries.

```mermaid
graph LR
  subgraph IMAGE["ONE container image, ONE codebase"]
    WEB["web — uvicorn ASGI<br/>/api/v1/* + static"]
    WORKER["worker — queue consumer<br/>ingest · parse · score · ai · mail · maintenance"]
  end
  CDN["Platform CDN<br/>built React bundle"] --> WEB
  WEB --> PG[("PostgreSQL 16<br/>one logical database")]
  WORKER --> PG
  WEB --> REDIS[("Redis<br/>cache · rate limit · sessions")]
  WEB --> BLOB[["Object storage<br/>CV blobs"]]
  WORKER --> BLOB
  WORKER --> EXT["Microsoft Graph · AI provider · job boards"]
```

### 1. Processes

| Process | Entrypoint | Contains | Sizing (per environment) |
|---|---|---|---|
| `web` | uvicorn ASGI | `/api/v1/*`, the built React bundle as static files, one async streaming endpoint for model output | 2 vCPU / 4 GB, 2 uvicorn workers × 4 threads |
| `worker` | queue consumer | document parsing, all model invocations, batch rescoring, Graph mail polling, fairness evaluation, periodic/scheduled tasks | 2 vCPU / 4 GB, concurrency 4 |

In Phase 2 the worker splits **by queue, not by codebase**: `worker-default` and `worker-untrusted`. The untrusted process runs parsing under a restricted OS user with no outbound network, a hard per-document CPU/wall timeout, and a memory cap. Same image, same repository, third revision.

### 2. Module tiers and the one-way dependency rule

Five tiers: `surfaces → core domain → platform`, and `intelligence → core domain (read) + platform`.

| # | Rule | Enforcement |
|---|---|---|
| 1 | Every module is a Python package whose only public entry point is `service.py`; cross-module imports may touch only `<module>.service` and `<module>.dto` — never `models`, `views` or `selectors` | `import-linter` forbidden-import contract |
| 2 | No module reads or writes another module's tables. Sole exception: `analytics`, which owns read-only SQL views declared in migrations | Code review + `import-linter`; view ownership is explicit in migration files |
| 3 | Core domain modules must never import `intelligence` or `surfaces`. AI results are *attached* by the domain module accepting a suggestion | `import-linter` layered contract |
| 4 | `identity`, `audit` and `config` are ambient dependencies of every module | Layered contract exempts them |
| 5 | A violation is a failed build, not a review argument | Required CI job |

**Rule 3 is the load-bearing one.** Because `intelligence` cannot import `application`, an AI module is *physically unable* to call `application.transition()` with a rejection. Combined with the guard that rejects any terminal-negative transition whose `actor_kind <> 'user'` — the exact literal, per `_decisions.md` RULING-01; `actor_kind` is `('user','system','integration','ai_agent')` and has no `human` member — "AI must never auto-reject" becomes a property of the dependency graph and a database constraint rather than a paragraph in a policy document.

### 3. What is explicitly not built

No service mesh, no orchestrator, no per-module deployable, no separate AI service, no separate search service, no message broker outside Postgres (ADR 0004), no read replica in Phase 1.

---

## Justification

**Team size decides this, and it is not close.** The cost of a deployable is not its code — it is its pipeline, its dashboards, its alert routing, its secrets, its contract versions and its local-dev story. Two developers with one reviewer have a budget for one or two of those, not fifteen. Every hour spent on inter-service plumbing is an hour not spent on the versioned requisitions, reversible merge and explainable scoring that are the actual product.

**Load provides no counter-argument.** 66 named seats, ~20–25 peak concurrent (**assumption**), 20k–60k applications/year and 200–600 documents/day at peak (**assumptions**). A single 2-vCPU web process is generously provisioned; a single 2-vCPU worker handles the document volume with headroom.

**The hard constraints make microservices actively worse, not merely unnecessary.** One master data model plus one relational database is incompatible with independent service datastores. Any microservice topology here converges on a shared database — which loses the isolation that was the entire point while keeping all the operational cost.

**Transactional integrity is where the brief's invariants live.** Raw intake must exist before a candidate; identity must be separate from applications; scores are per-application and pin exact versions. In-process these are foreign keys, `CHECK` constraints, partial unique indexes and one `COMMIT`. Distributed, they become eventual consistency plus compensations — and the failure mode is precisely the orphaned/duplicated state the brief exists to prevent.

**Security improves rather than degrades.** One `iam.can()` chokepoint, one audit writer, one place where the human actor is propagated into AI invocations. Twenty services each re-deriving authorization is twenty chances to leak candidate PII.

**The one real loss — failure isolation — is bought back cheaply.** The worker is already a separate process, so OCR and model calls cannot take the web tier down. The AI boundary carries a circuit breaker so the platform degrades gracefully with the model provider absent (BRD NFR-7, §8.3). That covers the overwhelming majority of the isolation microservices would have bought.

**Judgement call, stated:** we are choosing lower blast-radius isolation in exchange for correctness guarantees and operational tractability. For an internal, business-hours, 66-seat recruiting system where a few minutes of downtime costs a rescheduled screening call — not revenue — that trade is clearly right. It would be the wrong trade for a customer-facing system with an availability SLA, and this ADR should be reread if the platform is ever exposed externally.

---

## Consequences

### Positive

| Consequence | Why it matters here |
|---|---|
| The Phase 1 critical path is one transaction | The raw-intake-before-candidate invariant is a `NOT NULL` FK, not a saga |
| One CI pipeline, one image, one deploy | The junior can deploy on day one without learning an orchestrator |
| One authorization decision point | Directly satisfies "the chatbot must never bypass access controls" |
| Cross-module refactoring stays cheap | Essential while the candidate/application split churns entity shapes |
| Facade-per-module is a natural test seam | Service-level tests per module give the junior varied, independently demonstrable work |
| AI governance is structural | Dependency direction + a transition guard, not a prompt instruction |
| pgvector, FTS and trigram in one engine | No separate AI or search service is needed to hit Phase 2 goals |

### Negative — costs we are accepting

| Cost | Accepted because |
|---|---|
| **Shared deploy fate.** A bad release or migration stops recruiters and intake together | Business-hours internal tool; manual promote to production; PITR available |
| **No HA.** One app host means a platform restart is a few minutes of downtime | Explicitly accepted; the six-jurisdiction spread narrows the maintenance window and we schedule accordingly |
| **One dependency graph.** OCR/ML libraries share the image with the web tier; image size grows toward the ~2 GB trigger | Monitored as a *hard* split trigger; the split is pre-planned rather than emergent |
| **Coarse scaling.** You scale the image, not the module | At 66 seats this is theoretical; vertical scaling is the answer for years |
| **Boundary enforcement is a tool, and tools rot.** If `import-linter` contracts are weakened to unblock a PR, the boundaries silently stop existing | The contracts are a required check with no override path; weakening them requires a reviewed change to the contract file, which is visible in the diff |
| **No independent release cadence for AI.** Prompt/model changes ride the domain train | Named as a *soft* split trigger (>2 deploys/week needed) |
| **A monolith invites accidental coupling under deadline pressure** | Rules 1–3 are checkable mechanically because discipline does not scale with one reviewer |

---

## Risks

| # | Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|---|
| R1 | The modular monolith degenerates into a ball of mud within ~6 months — the exact failure the prototype already shows (`index.html:264-285`) | Medium | High | `import-linter` layered + forbidden-import contracts as a required CI job; one `service.py` per module; no module touches another's tables |
| R2 | A worker crash loop or OOM destabilises the shared host | Medium | Medium | Separate revision with its own memory cap; per-document CPU/wall timeout; queue-level isolation in Phase 2; "worker OOM took the host down twice in a quarter" is a named soft split trigger |
| R3 | An untrusted-file parser is exploited and, being in the same image, has the web tier's credentials | Low–Medium | High | Phase 2 `worker-untrusted` queue under a restricted OS user, no outbound network, magic-byte allowlist, malware-scan gate before any parse attempt (ADR 0003). Named as a *hard* split trigger if a real sandbox is required |
| R4 | An ML/OCR dependency becomes incompatible with the web stack | Medium | Medium | Hard split trigger at image >~2 GB or an irreconcilable conflict; the split is a new entrypoint, not a rewrite, because the codebase is already module-partitioned |
| R5 | Someone later argues "we should have used microservices" from taste rather than evidence | High | Low | The Revisit conditions below are numeric; "AI feels like a different concern" and org-chart preference are explicitly *not* triggers |
| R6 | Headcount grows and the single codebase becomes a merge-contention point | Low in Phase 1–2 | Medium | Trigger T7 below; module facades mean extraction is mechanical when it is genuinely needed |

---

## Revisit conditions

Reviewed quarterly by Talha. **Any single hard trigger justifies a separate deployable. Soft triggers require two sustained for 2+ weeks.**

| # | Trigger | Threshold | Hard? |
|---|---|---|---|
| T1 | Dependency conflict | An ML/OCR dependency cannot coexist in the web image, or the image exceeds ~2 GB | **Hard** |
| T2 | Hardware profile divergence | Parsing or inference requires a GPU, or >4 vCPU / >8 GB steady-state | **Hard** |
| T3 | Runtime isolation | Untrusted-file handling requires a sandbox the worker process cannot provide, beyond the Phase 2 restricted queue | **Hard** |
| T4 | Non-Python runtime | A required model runtime is not Python | **Hard** |
| T5 | Queue starvation | p95 time-to-start for interactive AI tasks >10 s while web p95 <300 ms, after vertical scaling of the worker host is exhausted | Soft |
| T6 | Release cadence conflict | Prompt/model changes require >2 deploys/week while domain code is release-gated | Soft |
| T7 | Blast radius | Worker OOM or crash loops have taken the shared host down twice in one quarter | Soft |
| T8 | Web tier saturation | Sustained web p95 >300 ms at 4 vCPU / 8 GB after query tuning, or CPU >70% for 2+ weeks in business hours | Soft |
| T9 | Team shape | Engineering headcount ≥6 organised into two or more independently releasing teams | Soft |
| T10 | Merge contention | >2 developer-days per month lost to cross-team merge conflicts or release-train blocking | Soft |

**Explicitly never a trigger:** "AI feels like a different concern"; org-chart preference; résumé-driven architecture; multi-tenancy or regional data residency (both excluded by constraint — residency is answered by one region plus per-record retention and escalated to legal, not solved with topology).

**If a trigger fires, the expected move is the cheapest one that clears it** — in order: vertical scaling → a third revision from the *same* image with a different entrypoint/queue → extraction of `document_parsing` + `ai_orchestration` as one stateless service that still writes `AiRun` rows to the same database. Full domain decomposition remains off the table while the one-database and two-developer constraints hold.