Bulk ATS Scoring Engine

One job description in, many resume PDFs in, a score-sorted leaderboard out.

Resumes are extracted with pypdf, evaluated concurrently against the job description with the OpenAI Responses API, validated against a strict schema, and returned as a single JSON response. A failure on one resume never fails the batch.

CLAUDE.md is the specification this implements and remains the source of truth for design decisions.

Setup

Requires Python 3.11+.

python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # macOS / Linux

pip install -e ".[dev]"

copy .env.example .env          # Windows
# cp .env.example .env          # macOS / Linux

Put an OPENAI_API_KEY in .env. .env is gitignored; never commit it.

On Windows, write .env as UTF-8 without a BOM. PowerShell 5.1's Set-Content -Encoding utf8 adds one, and a BOM becomes part of the first variable's name — that setting then silently reads as empty. The app parses .env as utf-8-sig so it tolerates this, but other tools reading the same file will not.

Run the service:

uvicorn app.main:create_app --factory --reload

There is deliberately no module-level app object. Building one at import time would read settings — and fail on a bad ANTHROPIC_MODEL — merely because something imported the module.

Verify the request shape before trusting it

Unit tests use fakes, so they cannot prove the real API accepts the request. One live check does, and it costs a few cents:

python scripts/smoke_structured_output.py

It confirms the schema derived from ATSScore is accepted, that output_parsed comes back valid, and that the second call reports non-zero cached_tokens. Run it whenever you change the model or upgrade the SDK.

Last verified against gpt-5.4-mini: both calls parsed, and call 2 served 2304 of 2649 input tokens from cache.

API

POST /api/v1/score

multipart/form-data:

Field	Type	Notes
job_description	text	Required, non-blank, MAX_JD_CHARS ceiling
resumes	file[]	Required, .pdf only, MAX_RESUMES_PER_REQUEST / MAX_PDF_SIZE_MB ceilings

curl -X POST http://localhost:8000/api/v1/score \
  -F "job_description=Backend engineer. Required: Python, FastAPI, Docker." \
  -F "resumes=@candidate-a.pdf" \
  -F "resumes=@candidate-b.pdf"

{
  "request_id": "2ce31ea9-29b2-4cad-a916-1a18cfc69c20",
  "total": 2,
  "succeeded": 1,
  "failed": 1,
  "results": [
    {
      "filename": "candidate-a.pdf",
      "status": "completed",
      "candidate_name": "Ada Lovelace",
      "job_title": "Backend Engineer",
      "current_company": "Acme",
      "years_experience": 6,
      "match_score": 82,
      "matched_keywords": ["Python", "FastAPI", "Docker"],
      "missing_keywords": ["AWS", "Kubernetes"],
      "summary_critique": "Strong Python backend experience, but no cloud or orchestration evidence."
    },
    {
      "filename": "candidate-b.pdf",
      "status": "failed",
      "error_code": "PDF_TEXT_UNAVAILABLE",
      "error_message": "No usable text could be extracted from the PDF."
    }
  ]
}

Completed results come first, sorted by match_score descending. Failures follow, in upload order. Ties keep upload order.

The profile fields (candidate_name, job_title, current_company, years_experience) are extracted from the resume by the model and are null whenever the resume does not state them. years_experience uses the total stated in the resume when there is one, otherwise it is computed from explicitly stated dates — never guessed. matched_keywords are verified server-side against the resume text; a keyword the resume never mentions is dropped rather than shown as evidence.

Status codes

Code	Meaning
200	Batch processed — including batches where every candidate failed
400	Malformed multipart request, or blank job description
413	Too many files, or a file over the size limit
415	A file is not a PDF
422	Structurally valid request with an out-of-range field value
500	Unexpected internal error

Error responses are {"request_id", "error_code", "error_message"}. Stack traces, provider response bodies, prompts, and document content never appear in them.

Per-candidate error codes

INVALID_PDF, PDF_ENCRYPTED, PDF_TEXT_UNAVAILABLE, MODEL_RATE_LIMITED, MODEL_TIMEOUT, MODEL_REFUSED, MODEL_RESPONSE_INVALID, MODEL_UNAVAILABLE, INTERNAL_ERROR.

Configuration

See .env.example for the full list. Three settings are easy to get wrong:

OPENAI_MODEL must support structured outputs. Validated at startup as a prefix check over known families — gpt-5*, gpt-4.1*, o3*, o4* — rather than an exact list, so a new point release isn't rejected on arrival. Two deliberate exclusions: gpt-4o (snapshots before 2024-08-06 lack structured outputs, and aliases hide which you get) and any -chat-latest variant (tracks the ChatGPT product surface, no reasoning effort). gpt-4.1 is allowed but is not a reasoning model — the adapter detects that and omits the reasoning parameter instead of sending a 400.

OPENAI_MAX_OUTPUT_TOKENS covers reasoning tokens and the response together. A small cap truncates mid-JSON and the candidate fails with MODEL_RESPONSE_INVALID. The enforced floor is 2048 and the tested baseline is 4000.

Lower OPENAI_EFFORT, not OPENAI_MAX_OUTPUT_TOKENS, to cut cost. The token budget is a truncation guard, not a spend dial; effort is the spend dial.

There is no temperature / top_p setting. Reasoning models reject them; the model is steered by the system prompt and structured outputs instead.

How cost is controlled

OpenAI caches automatically on an exact prompt prefix match — there is no breakpoint to place, so block ordering is the entire strategy. The instructions and job description are byte-identical across every candidate in a batch and go first; the resume goes second. On the live smoke test this served 87% of input tokens from cache (2304 of 2649) on the second call.

Two supporting details:

• prompt_cache_key is sent as a routing hint, derived from a hash of the job description. It is stable for a whole batch and never per-candidate — a high-cardinality key would defeat the purpose.
• A cache entry only becomes readable once the first response exists. If all 50 candidates launched at once, every one would pay full price — so score_batch awaits the first candidate alone to prime the prefix, then fans the rest out under the concurrency semaphore.

This is why nothing volatile may ever enter the job-description block. A timestamp, request id, or filename there moves the divergence point to the front of the prompt and the whole batch stops hitting the cache. test_llm.py fails if that happens.

Caching has a 1024-token minimum, so short job descriptions will not cache at all.

Development

ruff check .
ruff format --check .
mypy app
pytest -q

No test makes a live API call. Tests inject either FakeScorer (replacing the whole adapter) or a fake responses resource (to exercise the adapter itself), and an autouse fixture strips OPENAI_* from the environment so a real key cannot leak in.

Swapping providers is a contained change: the Scorer protocol in app/services/llm.py is the only seam that touches a vendor SDK. Models, PDF handling, orchestration, routing, and logging are provider-agnostic.

Data handling

Resumes contain personal data.

• Uploads are held in memory and parsed from io.BytesIO. Nothing is written to disk.
• Resume text, job-description text, prompts, and full model responses are never logged. The logger emits only an explicit allowlist of keys, and exceptions are recorded as a type plus file:line:func frames — never a formatted message, because provider errors can echo request content.
• Nothing is persisted between requests. There is no database and no queue, so retention is bounded by process lifetime. Adding any storage means writing a retention and deletion policy first.

Limitations

• Not a hiring decision. The score is decision support. The prompt forbids inferring or scoring protected characteristics, and candidates are never compared against each other — each is scored independently against the same job description.
• Scanned and image-only PDFs fail with PDF_TEXT_UNAVAILABLE. There is no OCR.
• Multi-column layouts extract in reading-order-ish, not exact, order. The system prompt tells the model this is an extraction artifact and not to penalise it.
• No authentication or rate limiting. Both are required before public deployment.