AI-Pricing-Agent/ARCHITECTURE-DETAIL.md

Utopia Pricing Agent — Architecture (engineering detail)

The technical reference: data sources, decision cascade, formulas, file map, backtests. For the plain-English overview see ARCHITECTURE.md.

A one-page Streamlit dashboard (Utopia/CRAI design system) that turns live COSMOS data into price recommendations a human can Approve / Modify / Reject. Read-only: nothing is written back to COSMOS or Amazon.

---

1. High-level view

flowchart LR
    U["🧑 User<br/>(same-network browser)"] --> APP

    subgraph APP["app.py — presentation (Streamlit, CRAI theme)"]
        SIDE["Sidebar<br/>Single product / Product line<br/>+ filters, kill switch"]
        QUEUE["Recommendation queue<br/>tiles · pills · rows"]
        SECT["Per-SKU sections<br/>price · inventory · scenarios<br/>competitors · PPC · costs · AI"]
    end

    subgraph DASH["dashboard/ package"]
        THEME["theme.py<br/>CRAI tokens + plotly template"]
        LIVE["live_data.py<br/>adapter + decision engine<br/>+ scenario economics"]
    end

    subgraph CORE["src/pricing_agent — analysis core"]
        AN["analyze.py"]
        MARGIN["margin_engine.py"]
        ELAST["elasticity.py"]
        PERF["performance.py"]
        SVC["cosmos/service.py"]
        CLIENT["cosmos/client.py"]
    end

    COSMOS[("COSMOS API")]
    APIFY[("Apify — optional")]

    APP --> THEME
    APP --> LIVE
    LIVE --> AN
    AN --> MARGIN & ELAST & PERF
    AN --> SVC --> CLIENT --> COSMOS
    LIVE --> SVC
    AN -.optional.-> APIFY

---

2. Layers

Layer	Files	Responsibility
Presentation	app.py	All rendering, zero pricing logic. Session state (approve/modify/reject, filters, per-SKU section + window), staged progress loader, session-state cache.
Design system	dashboard/theme.py, .streamlit/config.toml	CRAI palette (cream #f4f0e8, teal #0c8276, coral #df4f33, navy #22304e), Inter font, plotly template.
Adapter + engine	dashboard/live_data.py	Builds the per-SKU dict; decides the action; computes scenario economics (elasticity projection → bulk reconciliation → calibration); exposes scenarios_for_window().
Competitive state	src/pricing_agent/competitive_state.py	The one competitive fact the cascade may read. Adapts either scraper into a typed WON/LOST_PRICE/LOST_ELIGIBILITY/SUPPRESSED state with a source and a timestamp, gates it on age, and logs disagreement between sources.
Analysis core	src/pricing_agent/analyze.py	Orchestrates one SKU: fees → trend → bulk → ad cost → elasticity → actual-profit evidence.
Money math	tools/margin_engine.py	Pure: break-even, MAP, contribution margin, suggested price.
Statistics	elasticity.py, performance.py	Log-log elasticity fit, profit-optimal sweep, actual-profit aggregation.
Data access	cosmos/{client,service,models}.py	Auth + retry client; endpoint calls + response flattening; typed pydantic models.

---

3. Data sources — what each COSMOS endpoint feeds

flowchart TB
    subgraph COSMOS["COSMOS API"]
        TH["/sales-insight/takehome-calculator<br/>nested fees.breakdown · cost.breakdown"]
        INVP["/invp-insight<br/>trend + inventory + dateMap PROJECTIONS"]
        BULK["/sales-insight/bulk-calculator<br/>storage + total take-home"]
        SI["/sales-insight (daily, 6-month)<br/>price · units · revenue · profit · ad spend"]
        PROD["/products<br/>brand · marketplace"]
        CAMP["/api/campaigns · /adsApi<br/>budget · ACoS · ad sales (not yet wired)"]
    end

    TH -->|"_flatten_takehome()"| FEES["Fee model<br/>referral% · FBA · landed · returns"]
    INVP --> TREND["Velocity + cover days"]
    INVP --> INVPROJ["Inventory Outlook tab<br/>real weekly units/value/cover/arrivals"]
    BULK --> STORAGE["Storage + take-home (scenarios)"]
    SI --> HIST["180-day daily series<br/>(window filter + calibration)"]
    SI --> ADS["Ad spend / TACoS (PPC tab)"]
    PROD --> META["Brand / marketplace"]

Three response quirks handled:

• Fees come nested (fees.breakdown["Referral Fee"], "$ 9.28" strings). service._flatten_takehome() normalises them — without it every fee parsed to 0 (the old "$0.99 / break-even $0" bug).
• INVP dateMap holds COSMOS's own forward inventory projection (weekly units, value, cover days, warehouse arrivals). The Inventory tab renders this directly — not a locally-invented forecast.
• SKU lookups are CONTAINS/relevance searches, not equality. See §3.1 — this one silently bound the wrong product's data to a SKU.

3.1 Exact-SKU joins (service._exact_row)

Neither /api/products nor /api/invp-insight has an equality filter, and both used to fall back to data[0] "if COSMOS returned a fuzzy match set". That bound one product's ASIN, cost, brand and inventory to a different product's SKU.

Observed live: get_product("UBCFKFITTEDSHEETWHITECALKING") — a SKU COSMOS does not carry at all — returned UBMICROFIBERGUSSETPILLOWWHITEQUEEN / B08DTH86Q2.

Two independent problems, and each guard is necessary:

1. Wrong parameter. get_product queried q= (relevance across the whole catalogue). For UBMICROFIBERDUVETTWINWHITE the correct row sat on page 2 of 100-row pages, behind 100 unrelated products — a 20-row lookup never saw it. sku= returns it first.
2. sku= is still a CONTAINS filter. sku=UBCFKMATTRESSPROTECTORTWIN88 returns three rows: the real one (B00MRH9NCK), the ...BOX variant (B09K7HXJ4M), and a WAL... Walmart row whose "ASIN" (8946709597) is not an ASIN. All three are marketplace: AMAZON_USA, so the marketplace check alone does not separate them — only the exact SKU test does.

_exact_row() requires an exact SKU match and the right marketplace, and returns None otherwise, logging what it rejected. Blast radius of the old behaviour: analyze_price assigns asin = product.asin whenever INVP has none, so the competitive scrape would have run against an unrelated listing; and get_invp's skuPrefix matches every colour variant, so a sibling's inventory and cover days fed LOW_STOCK / EXCESS_STOCK directly — a sibling reading 12 units / 3 days would fire a stockout raise on a SKU holding 4,000 units.

Genuinely not in this COSMOS integration → shown as "—", never faked: ad-attributed sales / ACoS / campaign budget (live in /api/campaigns + /adsApi, not yet wired), and historical competitor prices (Apify gives a current snapshot only).

Competitor data is no longer display-only. COSMOS has no Buy Box, no rival price and no third-party offer anywhere in it — that gap is filled by a scrape, and two of its facts (our Buy Box being suppressed, and a rival materially undercutting us) reach the verdict. They are the only two, they are bounded by the guardrails, and their absence changes nothing. See §6.

---

4. Per-SKU pipeline (one "Analyze")

sequenceDiagram
    participant U as User
    participant A as app.py
    participant L as live_data.build_live_sku
    participant AN as analyze.py
    participant S as CosmosService
    U->>A: Single product / Product line
    A->>L: get_live_data(skus, progress_cb)
    Note over A: staged progress bar (3→10→45→82→94→100%)
    L->>S: get_current_price
    L->>S: get_sales_history (180d, parallel windows)
    L->>AN: analyze_price (fees, trend, bulk, elasticity, evidence)
    L->>S: get_invp (real inventory projection)
    L->>S: bulk_quote (storage)
    L->>L: decide action + scenario economics + 30d/6mo calibration
    L-->>A: {summary, details, errors}  (session-cached)
    A-->>U: queue + expandable per-SKU analysis

---

5. Scenario economics (the heart of the Scenarios tab)

For each candidate price, one consistent chain:

flowchart LR
    W["Window filter<br/>7/14/30/90d · 6mo"] --> BASE["Baseline velocity<br/>= avg units/day in window"]
    BASE --> DEMAND["Units(p) = units × (p/cur)^elasticity"]
    DEMAND --> REV["Revenue = units × p × 30"]
    REV --> AD["Ad spend = TACoS × revenue"]
    DEMAND --> TH["Take-home (bulk calculator fee model)"]
    TH --> GROSS["Gross = take-home − storage − ad"]
    GROSS --> CAL["× realization factor<br/>(actual booked ÷ modeled at current)"]
    CAL --> NET["Net profit / 30d"]

Key rules:

• Current row = FACT, not a projection: real units, real revenue, real ad spend, real booked profit. Its price is the average price sold (revenue ÷ units) so price × units × 30 = revenue reconciles — this is below list when promos ran, and changes with the window because the avg selling price differed period to period. The list price is fixed.
• Calibration: raw bulk-calculator profit over-states reality (prices at list, ignores real returns/promos). A realization factor = actual booked profit ÷ modeled profit at the current price scales every projected row.
• The elasticity used for projections is gated (projection_elasticity()). It honours the same actionable test the decision does: a slope whose 95% CI spans zero — or a positive slope, which estimate_elasticity can return since actionable requires e < 0 but the value is not clamped — falls back to FALLBACK_ELASTICITY. Ungated, a positive slope projected that raising price sells more, and that number drives the 30-day impact tile, the portfolio opportunity total and the queue sort. elasticity_is_fitted records which was used.
• ⭐ marks the highest-net-profit price in the current view.

---

6. Decision engine (deterministic, first match wins)

flowchart TD
    S([signals]) --> R0{cost data = 0?}
    R0 -- yes --> INV["🔍 INVESTIGATE · NO_COST_DATA"]
    R0 -- no --> RB{our Buy Box suppressed?}
    RB -- yes --> INVB["🔍 INVESTIGATE · BUYBOX_SUPPRESSED"]
    RB -- no --> R1{price < break-even?}
    R1 -- yes --> UP1["↑ raise to safe floor · BELOW_BREAK_EVEN"]
    R1 -- no --> R2{losing money after ads?}
    R2 -- yes --> UP2["↑ raise · LOSING_MONEY"]
    R2 -- no --> R3{cover ≤ 35d?}
    R3 -- yes --> UP3["↑ +5% · LOW_STOCK"]
    R3 -- no --> R4{−30% sales, no cause?}
    R4 -- yes --> INV2["🔍 INVESTIGATE · UNEXPLAINED_DROP"]
    R4 -- no --> R5{cover ≥ 90d?}
    R5 -- yes --> DN["↓ −5% · EXCESS_STOCK"]
    R5 -- no --> RC{rival ≥3% below AND corroborated?}
    RC -- yes --> DNC["↓ toward rival · COMPETITOR_UNDERCUT"]
    RC -- no --> R6{profit-optimal ≠ current?}
    R6 -- yes --> MOVE["↑/↓ toward optimal · PROFIT_OPTIMAL"]
    R6 -- no --> R7{a price we ran booked more?}
    R7 -- yes --> BO["↑/↓ toward it · BEST_OBSERVED"]
    R7 -- no --> HOLD["→ MAINTAIN · NO_SIGNALS"]

Guardrails: floor = highest of four break-evens × 1.02, ceiling = current × 1.25; the recommended move is capped at ±5% per step; the kill switch pauses all approvals.

AD_SPIRAL — applied after the cascade

When ad cost per unit climbs almost as fast as price, each extra $1 of price buys only cents of contribution and no price reaches break-even. That verdict (AD_SPIRAL, Investigate-and-hold) is applied after the cascade and overrides whatever fired — so what it must not override is named explicitly:

AD_SPIRAL_YIELDS_TO = frozenset({"NO_COST_DATA", "BUYBOX_SUPPRESSED", "LOW_STOCK"})

Yields to	Why
NO_COST_DATA	With costPerUnit/fbaFee at 0, fixed is understated, which makes the contribution <= 0 test easier to hit. A missing-COGS SKU would be sent to the ad console when the fix is a data-entry field.
BUYBOX_SUPPRESSED	A listing nobody can buy from has no meaningful ad economics.
LOW_STOCK	The only one that changes a price, not just a label. A shelf about to empty gets +5% whatever the ad slope does — those units sell regardless, so the only question is what we get for them. Holding sells the last of the stock cheap.

Deliberately narrow: EXCESS_STOCK does not outrank it, because cutting price to clear stock is exactly the move that cannot work when ads eat the contribution.

Competitor rules — the two that can move a price, and what bounds them

Both branches read a single CompetitiveState (competitive_state.py), never a raw scrape:

Rule	Fires when	Effect
BUYBOX_SUPPRESSED	Amazon is not showing our offer	Investigate, hold. Placed directly under NO_COST_DATA: the only two states where the answer is "go and fix something" rather than "set a price".
COMPETITOR_UNDERCUT	cheapest rival ≥ competitor_undercut_material_pct below us, and the basis qualifies (below)	Decrease toward the rival, floored and step-capped like every other branch.

Basis decides what "qualifies" means, because competitor_min can be a price of two different things:

• same-asin-buybox (Apify) — another seller's offer on our own listing. Only LOST_PRICE fires; a rival holding the Buy Box above us is LOST_ELIGIBILITY, where cutting donates margin. Losing the Buy Box on price is the corroboration.

• like-for-like-sheet — a rival brand's equivalent variant, matched on size + colour. This additionally requires corroboration: either demand has materially dropped, or we are not actually winning the Buy Box.

  Why: on its own a sheet row says only "a different brand is cheaper". True, reportable, but not evidence the gap is costing us anything — we can sit 3% dearer, hold our own Buy Box and sell fine on brand, reviews or the Prime badge. And because this branch sits above PROFIT_OPTIMAL, an uncorroborated cut could overrule an elasticity fit that wanted a raise. Switch: competitor_sheet_requires_corroboration (default true).

  The documented UNEXPLAINED_DROP fall-through is unaffected — UBMICROFIBERDUVETKINGPURPLE and UBMICROFIBERBS4PCFULLGREY arrive here with a velocity drop, which is the first form of corroboration.

  An uncorroborated material undercut is still reported in the root cause ("Competitor undercut not acted on"), so it never looks like missing data.

A third signal, a competitor premium while we hold the Buy Box, is a narrative note only. It never sets a price and never changes an action.

Ordering is deliberate: inventory risk outranks competitor position, which outranks profit-optimal. Chasing a rival down while the shelf is emptying pays margin to sell out faster.

Three properties make this safe to ship:

1. Fail-safe. Absent, failed, stale (> competitor_state_max_age_hours) and "ownership unknown" all collapse to one flag, and the cascade then computes exactly the verdict it computed before competitor data existed. Competitor data can only ever add a verdict.
2. Never below break-even. The rival price is a candidate (comp_match), not a decision.
3. One named reason per verdict. No blended scores — every fired rule is traceable to a single reason code, and logger.info names the SKU, the rule, the state and the source.

Thresholds live in config/pricing_rules.yaml (competitor_undercut_material_pct: 0.03, competitor_premium_material_pct: 0.10, competitor_state_max_age_hours: 6.0, competitor_sheet_requires_corroboration: true), not in code. The 3% floor sits above the ~2% band our own realized price already swings through as coupons toggle.

Inventory cover matches COSMOS Inventory Planning

cover_days is COSMOS's own coverDays, so the dashboard and the INVP grid never quote two different numbers for one SKU. COSMOS counts inbound stock against a 7-day velocity, so it reads longer than what is on the shelf — UBMICROFIBERDUVETTWINWHITE is 78 days on (3,999 on hand + 1,030 inbound) ÷ 64/day, against 63 on-hand-only. Both are reported: the tile leads with the matched figure and appends 63 d on hand, rest inbound.

The on-hand figure remains the fallback, because COSMOS returns coverDays: 0 on some very low-velocity SKUs that hold months of stock (UBMICROFIBERBS4PCFULLGREY: 167 units, 334 real days, COSMOS said 0). Zero satisfies neither inventory rule, so taken literally it silences both. The reconciled figure is computed before _decide and passed in, so the tile and the rule that fired read the same number by construction.

Trade-off, accepted deliberately: stockout risk is now judged partly on stock that has not landed. Measured over the 56-SKU covered line, matching COSMOS moved 7 verdicts — LOW_STOCK 7 → 4, EXCESS_STOCK 18 → 22. The one to watch is UBMICROFIBERBS4PCKINGWHITE: 12 days on the shelf, 84 with inbound, so it no longer raises. If that shipment slips, nothing protects it.

Display bands are COSMOS's Alpha/Beta scheme (theme.COVER_BANDS, Alpha 20/40/70/100). The pricing triggers are separate and live in pricing_rules.yaml (low_cover_days: 35, high_cover_days: 90) — COSMOS's pink at 70 days is a replenishment warning, while crossing a trigger here spends margin on a 5% move.

Coverage: the comparison sheet gates competitor data, one product line at a time

The competitor workbook currently covers one product line, so the engine reads it as the first competitor source and gates on coverage:

SKU	Competitor state
In the sheet	Priced from the sheet — real like-for-like rival prices, basis=like-for-like-sheet
Not in the sheet	N/A, naming what the sheet does cover. No rule fires; the verdict is byte-identical to the competitor-blind one

Coverage is the exact SKU set in the sheet, not a line prefix. Measured against the real workbook, a prefix gate would be wrong in both directions: the UBMICROFIBERDUVET run contains 49 UBMICROFIBERDUVET* SKUs and 7 UBMICROFIBERBS4PC* ones, while the line has 139 SKUs in COSMOS of which only 56 reached a comparison row. So the sheet's own SKU list is the authority, and "not in the sheet" is a coverage hole, never a claim that the SKU has no competitors.

competitor_sheet_only: true (default while one line is under test) means an uncovered SKU gets N/A rather than falling through to a per-ASIN Apify scrape. Config: competitor_sheet_path (blank = auto-discover the newest Competitor_Price_Comparison_*.xlsx), competitor_sheet_dirs, competitor_sheet_max_age_hours: 168 (the sheet is a 25–35 min batch run, not a live feed).

Two sheet-driven refinements, both from real rows:

• A rival whose own Buy Box is suppressed is excluded from the band. Their price is not buyable, so undercutting it donates margin for nothing.
• A material undercut can explain a velocity drop (subject to the corroboration rule above), converting an UNEXPLAINED_DROP Investigate into an actionable verdict — exactly as the existing stockout branch already did.

Two scrapers, one state

Source	Authoritative for	Why
Apify (tools/amazon/apify.py)	Buy Box state read by the engine	The only source carrying a seller id, so the only one that can tell WON from LOST_PRICE from LOST_ELIGIBILITY. Those lead to opposite actions.
Playwright (../scraper/)	The workbook: like-for-like size/colour matching, BSR, demand buckets, SKU gaps	Apify cannot produce any of it. Its Buy Box field knows only whether a price rendered, not whose it was.

The split is by question, not preference. reconcile() cross-checks the authoritative state against the Playwright run's own cache (scraper/.scrape_cache.json, keyed ASIN@ZIP) and logs any disagreement. A disagreement never changes the verdict.

---

6.1 Verdict backtest

scripts/backtest_competitor_rules.py re-runs the captured real cascade inputs (real AnalysisResult, real 180-day history, real fee stack, real scenario grid) through the same _decide, varying only the arm. Nothing is reconstructed by hand.

Competitor rules blind vs live (7 requested SKUs, 6 analysed — see §9):

Arm	Verdicts changed
Real competitor state	0 / 6 — the fail-safe working
Counterfactual 8% undercut	4 / 6 — the others blocked by LOW_STOCK / LOSING_MONEY
Counterfactual suppression	6 / 6 → Investigate

Policy delta, old vs new (the two cascade changes), same SKUs:

Case	Changed	Notes
SHEET_UNDERCUT_WE_WIN	2 / 6	Both were being cut while holding the Buy Box with flat demand: UBMICROFIBERDUVETTWINWHITE Decrease $17.06 → Increase $17.94 (BEST_OBSERVED), UBMICROFIBERGUSSETPILLOWWHITEQUEEN Decrease $25.01 → Increase $27.37. The two corroborated SKUs were unchanged.
AD_SPIRAL_LOW_STOCK	4 / 6	Investigate/AD_SPIRAL → Increase/LOW_STOCK
AD_SPIRAL_NO_COST	6 / 6	Investigate/AD_SPIRAL → Investigate/NO_COST_DATA
Targets below break-even, any arm	0

The invariant is measured on the TARGET, not the step-capped first move. A SKU already selling under its own ad-inclusive floor cannot be lifted over it in one 5% step, and reporting that deliberate multi-cycle climb as a breach buries any real one. Two SKUs are below floor and stepping up by design; both are reported separately.

---

7. Key formulas

Quantity	Formula
Take-home / unit	p·(1 − referral% − returns%) − landed − FBA − other
Break-even	(landed + FBA + returns + other) / (1 − referral%)
Elasticity	OLS on ln(units/day) = a + e·ln(price) over 6 months
Scenario demand	units × (p / p₀)^e
Realization factor	actual booked profit (window) ÷ modeled net at current price
TACoS	ad spend ÷ total revenue (window)
Avg sold price	revenue ÷ units (window) — reconciles the Current row
Best observed price	avg_price of the best-earning band — the price actually charged, never the $0.50-rounded price_band key

---

8. Repository map

pricing_agent/
├── app.py                     # dashboard (presentation only)
├── legacy_app.py              # previous analyst UI (still runnable)
├── dashboard/
│   ├── theme.py               # CRAI design tokens + plotly template
│   └── live_data.py           # COSMOS adapter, decision + scenario engine
├── src/pricing_agent/
│   ├── analyze.py             # per-SKU orchestration → AnalysisResult
│   ├── competitive_state.py   # canonical Buy Box state + two-scraper reconciliation
│   ├── elasticity.py          # demand model + profit optimizer
│   ├── performance.py         # actual-profit evidence
│   ├── tools/margin_engine.py # pure fee/break-even math (golden-tested)
│   └── cosmos/
│       ├── client.py          # auth + retry HTTP
│       ├── service.py         # endpoints, _exact_row, _flatten_takehome, INVP projections
│       └── models.py          # typed COSMOS responses (pydantic)
├── config/                    # settings + pricing_rules.yaml (incl. competitor thresholds)
├── scripts/
│   └── backtest_competitor_rules.py   # verdict delta + old-vs-new policy delta
├── .streamlit/config.toml     # CRAI theme
└── tests/                     # margin-engine golden values, test_competitor_rules.py
                               # (fail-safe + ordering invariants), test_pricing_safety.py,
                               # test_cosmos.py (exact-SKU joins)

---

9. Known data gaps

• Ad-attributed sales / ACoS / campaign budget — live in /api/campaigns + /adsApi, not yet wired. Shown as "—".
• Historical competitor prices — Apify returns a current snapshot only.
• UBCFKFITTEDSHEETWHITECALKING is not in COSMOS at all. sku= returns zero rows on any marketplace and the fee endpoint replies "Product not found". Likely delisted or renamed. It now fails loudly (get_product → None) rather than adopting another product's identity. A catalogue question, not a code one.

---

10. Principles

1. Deterministic core, narrative shell — every number is a formula over COSMOS data; language models only phrase explanations.
2. Read-only — the agent proposes; a human approves; nothing writes back. submit_price_approval remains a stub with no callers.
3. Honest gaps — missing upstream data shows "—" or an explicit investigation, never a fabricated number. A wrong number is worse than a blank one.
4. Facts vs projections are labeled — the Current row is real booked history; other prices are clearly modeled.
5. Everything reconciles — one averaging window drives units, revenue, ads and profit so price × units = revenue always holds.