AI-Pricing-Agent/ARCHITECTURE.md

# Utopia Pricing Agent — Architecture

A one-page Streamlit dashboard (Utopia/CRAI design system) that turns **live COSMOS
data** into price recommendations a human can Approve / Modify / Reject. Read-only:
nothing is written back to COSMOS or Amazon.


---

## 1. High-level view

```mermaid
flowchart LR
    U["🧑 User<br/>(same-network browser)"] --> APP

    subgraph APP["app.py — presentation (Streamlit, CRAI theme)"]
        SIDE["Sidebar<br/>Single product / Product line<br/>+ filters, kill switch"]
        QUEUE["Recommendation queue<br/>tiles · pills · rows"]
        SECT["Per-SKU sections<br/>price · inventory · scenarios<br/>competitors · PPC · costs · AI"]
    end

    subgraph DASH["dashboard/ package"]
        THEME["theme.py<br/>CRAI tokens + plotly template"]
        LIVE["live_data.py<br/>adapter + decision engine<br/>+ scenario economics"]
    end

    subgraph CORE["src/pricing_agent — analysis core"]
        AN["analyze.py"]
        MARGIN["margin_engine.py"]
        ELAST["elasticity.py"]
        PERF["performance.py"]
        SVC["cosmos/service.py"]
        CLIENT["cosmos/client.py"]
    end

    COSMOS[("COSMOS API")]
    APIFY[("Apify — optional")]

    APP --> THEME
    APP --> LIVE
    LIVE --> AN
    AN --> MARGIN & ELAST & PERF
    AN --> SVC --> CLIENT --> COSMOS
    LIVE --> SVC
    AN -.optional.-> APIFY
```

---

## 2. Layers

| Layer | Files | Responsibility |
|---|---|---|
| **Presentation** | `app.py` | All rendering, zero pricing logic. Session state (approve/modify/reject, filters, per-SKU section + window), staged progress loader, session-state cache. |
| **Design system** | `dashboard/theme.py`, `.streamlit/config.toml` | CRAI palette (cream `#f4f0e8`, teal `#0c8276`, coral `#df4f33`, navy `#22304e`), Inter font, plotly template. |
| **Adapter + engine** | `dashboard/live_data.py` | Builds the per-SKU dict; **decides** the action; computes **scenario economics** (elasticity projection → bulk reconciliation → calibration); exposes `scenarios_for_window()`. |
| **Competitive state** | `src/pricing_agent/competitive_state.py` | The one competitive fact the cascade may read. Adapts either scraper into a typed `WON`/`LOST_PRICE`/`LOST_ELIGIBILITY`/`SUPPRESSED` state with a source and a timestamp, gates it on age, and logs disagreement between sources. |
| **Analysis core** | `src/pricing_agent/analyze.py` | Orchestrates one SKU: fees → trend → bulk → ad cost → elasticity → actual-profit evidence. |
| **Money math** | `tools/margin_engine.py` | Pure: break-even, MAP, contribution margin, suggested price. |
| **Statistics** | `elasticity.py`, `performance.py` | Log-log elasticity fit, profit-optimal sweep, actual-profit aggregation. |
| **Data access** | `cosmos/{client,service,models}.py` | Auth + retry client; endpoint calls + response flattening; typed pydantic models. |

---

## 3. Data sources — what each COSMOS endpoint feeds

```mermaid
flowchart TB
    subgraph COSMOS["COSMOS API"]
        TH["/sales-insight/takehome-calculator<br/>nested fees.breakdown · cost.breakdown"]
        INVP["/invp-insight<br/>trend + inventory + dateMap PROJECTIONS"]
        BULK["/sales-insight/bulk-calculator<br/>storage + total take-home"]
        SI["/sales-insight (daily, 6-month)<br/>price · units · revenue · profit · ad spend"]
        PROD["/products<br/>brand · marketplace"]
        CAMP["/api/campaigns · /adsApi<br/>budget · ACoS · ad sales (not yet wired)"]
    end

    TH -->|"_flatten_takehome()"| FEES["Fee model<br/>referral% · FBA · landed · returns"]
    INVP --> TREND["Velocity + cover days"]
    INVP --> INVPROJ["Inventory Outlook tab<br/>real weekly units/value/cover/arrivals"]
    BULK --> STORAGE["Storage + take-home (scenarios)"]
    SI --> HIST["180-day daily series<br/>(window filter + calibration)"]
    SI --> ADS["Ad spend / TACoS (PPC tab)"]
    PROD --> META["Brand / marketplace"]
```

**Two response quirks handled:**
- **Fees come nested** (`fees.breakdown["Referral Fee"]`, `"$ 9.28"` strings). `service._flatten_takehome()` normalises them — without it every fee parsed to 0 (the old "$0.99 / break-even $0" bug).
- **INVP `dateMap`** holds COSMOS's own **forward inventory projection** (weekly units, value, cover days, warehouse arrivals). The Inventory tab renders this directly — not a locally-invented forecast.

Genuinely **not** in this COSMOS integration → shown as "—", never faked: ad-attributed
sales / ACoS / campaign budget (live in `/api/campaigns` + `/adsApi`, not yet wired), and
historical competitor prices (Apify gives a current snapshot only).

**Competitor data is no longer display-only.** COSMOS has no Buy Box, no rival price and no
third-party offer anywhere in it — that gap is filled by a scrape, and as of the competitor
wiring two of its facts (our Buy Box being suppressed, and a rival materially undercutting us)
reach the verdict. They are the only two, they are bounded by the guardrails, and their
absence changes nothing. See §6.

---

## 4. Per-SKU pipeline (one "Analyze")

```mermaid
sequenceDiagram
    participant U as User
    participant A as app.py
    participant L as live_data.build_live_sku
    participant AN as analyze.py
    participant S as CosmosService
    U->>A: Single product / Product line
    A->>L: get_live_data(skus, progress_cb)
    Note over A: staged progress bar (3→10→45→82→94→100%)
    L->>S: get_current_price
    L->>S: get_sales_history (180d, parallel windows)
    L->>AN: analyze_price (fees, trend, bulk, elasticity, evidence)
    L->>S: get_invp (real inventory projection)
    L->>S: bulk_quote (storage)
    L->>L: decide action + scenario economics + 30d/6mo calibration
    L-->>A: {summary, details, errors}  (session-cached)
    A-->>U: queue + expandable per-SKU analysis
```

---

## 5. Scenario economics (the heart of the Scenarios tab)

For each candidate price, one consistent chain:

```mermaid
flowchart LR
    W["Window filter<br/>7/14/30/90d · 6mo"] --> BASE["Baseline velocity<br/>= avg units/day in window"]
    BASE --> DEMAND["Units(p) = units × (p/cur)^elasticity"]
    DEMAND --> REV["Revenue = units × p × 30"]
    REV --> AD["Ad spend = TACoS × revenue"]
    DEMAND --> TH["Take-home (bulk calculator fee model)"]
    TH --> GROSS["Gross = take-home − storage − ad"]
    GROSS --> CAL["× realization factor<br/>(actual booked ÷ modeled at current)"]
    CAL --> NET["Net profit / 30d"]
```

Key rules:
- **Current row = FACT**, not a projection: real units, real revenue, real ad spend, real
  booked profit. Its price is the **average price sold** (revenue ÷ units) so
  `price × units × 30 = revenue` reconciles — this is *below* list when promos ran, and
  changes with the window because the avg selling price differed period to period. The
  **list price is fixed**.
- **Calibration:** raw bulk-calculator profit over-states reality (prices at list, ignores
  real returns/promos). A **realization factor** = actual booked profit ÷ modeled profit at
  the current price scales every projected row, anchoring net profit to what the SKU truly
  earns.
- **Suggested price** (teal callout) is computed from the **full 6-month** window always —
  stable — independent of the display-window filter.
- **⭐** marks the highest-net-profit price in the current view.

---

## 6. Decision engine (deterministic, first match wins)

```mermaid
flowchart TD
    S([signals]) --> R0{cost data = 0?}
    R0 -- yes --> INV["🔍 INVESTIGATE · NO_COST_DATA"]
    R0 -- no --> RB{our Buy Box suppressed?}
    RB -- yes --> INVB["🔍 INVESTIGATE · BUYBOX_SUPPRESSED"]
    RB -- no --> R1{price < break-even?}
    R1 -- yes --> UP1["↑ raise to safe floor · BELOW_BREAK_EVEN"]
    R1 -- no --> R2{losing money after ads?}
    R2 -- yes --> UP2["↑ raise · LOSING_MONEY"]
    R2 -- no --> R3{cover < 35d?}
    R3 -- yes --> UP3["↑ +5% · LOW_STOCK"]
    R3 -- no --> R4{−30% sales, no cause?}
    R4 -- yes --> INV2["🔍 INVESTIGATE · UNEXPLAINED_DROP"]
    R4 -- no --> R5{cover > 90d?}
    R5 -- yes --> DN["↓ −5% · EXCESS_STOCK"]
    R5 -- no --> RC{lost Buy Box AND rival ≥3% below?}
    RC -- yes --> DNC["↓ toward rival · COMPETITOR_UNDERCUT"]
    RC -- no --> R6{profit-optimal ≠ current?}
    R6 -- yes --> MOVE["↑/↓ toward optimal · PROFIT_OPTIMAL"]
    R6 -- no --> HOLD["→ MAINTAIN · NO_SIGNALS"]
```

Guardrails: floor = break-even × 1.05, ceiling = current × 1.25; the recommended move is
capped at ±5% (bigger steps need elevated approval); the kill switch pauses all approvals.

### Competitor rules — the two that can move a price, and what bounds them

Both branches read a single `CompetitiveState`
([competitive_state.py](src/pricing_agent/competitive_state.py)), never a raw scrape:

| Rule | Fires when | Effect |
|---|---|---|
| `BUYBOX_SUPPRESSED` | Amazon is not showing our offer | **Investigate, hold.** Placed directly under `NO_COST_DATA`: those are the only two states where the answer is "go and fix something" rather than "set a price". A suppressed variant sells nothing at any price, so its margin and its modelled optimum both describe a listing nobody can buy from. |
| `COMPETITOR_UNDERCUT` | `LOST_PRICE` **and** cheapest rival ≥ `competitor_undercut_material_pct` below us | **Decrease toward the rival**, floored and step-capped like every other branch. |

A third signal, a **competitor premium** while we hold the Buy Box, is a narrative note only.
It never sets a price and never changes an action — the elasticity fit is the thing with
evidence behind it, and a premium is not grounds to overrule it.

Ordering is deliberate: **inventory risk still outranks competitor position, which outranks
profit-optimal.** Chasing a rival down while the shelf is emptying pays margin to sell out
faster. This is visible in the backtest below — two of five real SKUs did not move under a
counterfactual undercut precisely because `LOW_STOCK` and `LOSING_MONEY` fired first.

Three properties make this safe to ship:

1. **Fail-safe.** Absent, failed, stale (> `competitor_state_max_age_hours`) and
   "ownership unknown" all collapse to one flag, and the cascade then computes exactly the
   verdict it computed before competitor data existed. Nothing waits on a scrape; nothing is
   blocked by one. Competitor data can only ever *add* a verdict.
2. **Never below break-even.** The rival price is a *candidate* (`comp_match`), not a
   decision. Verified against real SKUs with a counterfactual rival 60% below us: against a
   $14.88 floor the shipped recommendations were $15.19–$21.84, because the ±5% step cap
   binds first. Zero violations.
3. **One named reason per verdict.** No blended scores — every fired rule is traceable to a
   single reason code, and `logger.info` names the SKU, the rule, the state and the source.

Thresholds live in [config/pricing_rules.yaml](config/pricing_rules.yaml)
(`competitor_undercut_material_pct: 0.03`, `competitor_premium_material_pct: 0.10`,
`competitor_state_max_age_hours: 6.0`), not in code. The 3% floor sits above the ~2% band our
own realized price already swings through as coupons toggle.

### Inventory cover matches COSMOS Inventory Planning

`cover_days` **is COSMOS's own `coverDays`**, so the dashboard and the INVP grid never quote
two different numbers for one SKU. COSMOS counts **inbound** stock against a **7-day**
velocity, so it reads longer than what is on the shelf — `UBMICROFIBERDUVETTWINWHITE` is
78 days on (3,999 on hand + 1,030 inbound) ÷ 64/day, against 63 on-hand-only. Both are
reported: the tile leads with the matched figure and appends `63 d on hand, rest inbound`.

The on-hand figure remains the **fallback**, because COSMOS returns `coverDays: 0` on some
very low-velocity SKUs that hold months of stock (`UBMICROFIBERBS4PCFULLGREY`: 167 units,
334 real days, COSMOS said `0`). Zero satisfies neither inventory rule, so taken literally it
silences both.

**Trade-off, accepted deliberately:** stockout risk is now judged partly on stock that has not
landed. Measured over the 56-SKU covered line, matching COSMOS moved 7 verdicts —
`LOW_STOCK` 7 → 4, `EXCESS_STOCK` 18 → 22. The one to watch is
`UBMICROFIBERBS4PCKINGWHITE`: **12 days on the shelf, 84 with inbound**, so it no longer
raises. If that shipment slips, nothing protects it.

Display bands are COSMOS's Alpha/Beta scheme (`theme.COVER_BANDS`, Alpha 20/40/70/100). The
pricing **triggers** are separate and live in `pricing_rules.yaml`
(`low_cover_days: 35`, `high_cover_days: 90`) — COSMOS's pink at 70 days is a *replenishment*
warning, while crossing a trigger here spends margin on a 5% move. Adopting the band edges
(40/70) would put six more SKUs on a discount; the measured table is in the config beside the
values.

### Coverage: the comparison sheet gates competitor data, one product line at a time

The competitor workbook currently covers **one product line**, so the engine reads it as the
first competitor source and **gates on coverage**:

| SKU | Competitor state |
|---|---|
| In the sheet | Priced from the sheet — real like-for-like rival prices, `basis=like-for-like-sheet` |
| Not in the sheet | **`N/A`**, naming what the sheet *does* cover. No rule fires; the verdict is byte-identical to the competitor-blind one |

**Coverage is the exact SKU set in the sheet, not a line prefix.** Measured against the real
workbook, a prefix gate would be wrong in both directions: the `UBMICROFIBERDUVET` run contains
49 `UBMICROFIBERDUVET*` SKUs **and 7 `UBMICROFIBERBS4PC*`** ones (variants come off the Amazon
parent twister, and COSMOS maps those ASINs to whatever SKU codes they carry), while the line
has 139 SKUs in COSMOS of which only 56 reached a comparison row. So the sheet's own SKU list is
the authority, and "not in the sheet" is reported as a **coverage hole**, never as a claim that
the SKU has no competitors.

`competitor_sheet_only: true` (default while one line is under test) means an uncovered SKU gets
N/A rather than falling through to a per-ASIN Apify scrape — so every verdict either rests on
the sheet or says it has no competitor data. Set it `False` once coverage is broad enough for
Apify to be a sensible fallback. Config: `competitor_sheet_path` (blank = auto-discover the
newest `Competitor_Price_Comparison_*.xlsx`), `competitor_sheet_dirs`,
`competitor_sheet_max_age_hours: 168` (the sheet is a 25–35 min batch run, not a live feed, so
it gets a longer limit than the 6 h single-ASIN one).

**Two bases, never conflated.** `competitor_min` can be a price of two different things, and the
`basis` field records which:

- `same-asin-buybox` (Apify) — another seller's offer on **our own listing**. Only `LOST_PRICE`
  fires the undercut rule; a rival holding the Buy Box *above* us is `LOST_ELIGIBILITY`, where
  cutting donates margin.
- `like-for-like-sheet` — a **rival brand's** equivalent variant, matched on size + colour. A
  cheaper one fires the rule **regardless of who owns our Buy Box**: we can hold ours perfectly
  well while a different product undercuts us. This is what the comparison tool exists to
  report. It is never labelled a Buy Box loss.

Two sheet-driven refinements, both from real rows:

- **A rival whose own Buy Box is suppressed is excluded from the band.** Their price is not
  buyable, so undercutting it donates margin for nothing.
- **A material undercut now explains a velocity drop.** Previously a drop with no *own* price
  change and no ad collapse was filed `UNEXPLAINED_DROP` even when the sheet held the
  explanation — observed on `UBMICROFIBERDUVETKINGPURPLE` (rival 15.6% below) and
  `UBMICROFIBERBS4PCFULLGREY` (35.2% below). A named cause now converts the Investigate into an
  actionable verdict, exactly as the existing stockout branch already did. A stockout still
  explains a drop first; an immaterial rival explains nothing.

Finally, a **suppressed listing has no selling price** (it sells nothing, so COSMOS records no
sales), which used to fail with a bare "no current selling price found in COSMOS". That error now
names the cause, so the most actionable rows in the sheet stop looking like a data problem.

### Two scrapers, one state

| Source | Authoritative for | Why |
|---|---|---|
| **Apify** (`tools/amazon/apify.py`) | **Buy Box state read by the engine** | The only source carrying a seller id, so the only one that can tell `WON` from `LOST_PRICE` from `LOST_ELIGIBILITY`. Those lead to opposite actions (cut price vs. fix fulfilment eligibility). |
| **Playwright** (`../scraper/`) | The workbook: like-for-like size/colour matching, BSR, demand buckets, SKU gaps | Apify cannot produce any of it. Its Buy Box field knows only whether a price *rendered*, not whose it was. |

The split is by **question**, not preference, so neither source is redundant. `reconcile()`
cross-checks the authoritative state against the Playwright run's own cache
(`scraper/.scrape_cache.json`, keyed `ASIN@ZIP`) and **logs any disagreement** rather than
letting a workbook and a recommendation contradict each other in front of a stakeholder. A
disagreement never changes the verdict. Where the authoritative source has nothing usable but
the other has a `SUPPRESSED`, that fact is promoted and the provenance recorded — discarding
it to preserve a hierarchy would be choosing the hierarchy over the fact.

Backtest (`scripts/backtest_competitor_rules.py`, 5 real SKUs, real COSMOS data):

| Arm | Verdicts changed |
|---|---|
| Real competitor state (both cached entries stale: 66h / 146h) | **0 / 5** — the fail-safe working |
| Counterfactual 8% undercut | 3 / 5 — the other 2 blocked by `LOW_STOCK` / `LOSING_MONEY` |
| Counterfactual suppression | 5 / 5 → Investigate |
| Prices shipped below break-even, any arm | **0** |

---

## 7. Key formulas

| Quantity | Formula |
|---|---|
| Take-home / unit | `p·(1 − referral% − returns%) − landed − FBA − other` |
| Break-even | `(landed + FBA + returns + other) / (1 − referral%)` |
| Elasticity | OLS on `ln(units/day) = a + e·ln(price)` over 6 months |
| Scenario demand | `units × (p / p₀)^e` |
| Realization factor | `actual booked profit (window) ÷ modeled net at current price` |
| TACoS | `ad spend ÷ total revenue` (window) |
| Avg sold price | `revenue ÷ units` (window) — reconciles the Current row |

---

## 8. Repository map

```
pricing_agent/
├── app.py                     # dashboard (presentation only)
├── legacy_app.py              # previous analyst UI (still runnable)
├── dashboard/
│   ├── theme.py               # CRAI design tokens + plotly template
│   └── live_data.py           # COSMOS adapter, decision + scenario engine
├── src/pricing_agent/
│   ├── analyze.py             # per-SKU orchestration → AnalysisResult
│   ├── competitive_state.py   # canonical Buy Box state + two-scraper reconciliation
│   ├── elasticity.py          # demand model + profit optimizer
│   ├── performance.py         # actual-profit evidence
│   ├── tools/margin_engine.py # pure fee/break-even math (golden-tested)
│   └── cosmos/
│       ├── client.py          # auth + retry HTTP
│       ├── service.py         # endpoints, _flatten_takehome, INVP projections
│       └── models.py          # typed COSMOS responses (pydantic)
├── config/                    # settings + pricing_rules.yaml (incl. competitor thresholds)
├── scripts/
│   └── backtest_competitor_rules.py   # verdict delta, competitor rules blind vs live
├── .streamlit/config.toml     # CRAI theme
└── tests/                     # incl. margin-engine golden values and
                               # test_competitor_rules.py (fail-safe + ordering invariants)
```

---

## 9. Principles

1. **Deterministic core, narrative shell** — every number is a formula over COSMOS data; language models only phrase explanations.
2. **Read-only** — the agent proposes; a human approves; nothing writes back.
   `submit_price_approval` remains a stub with no callers.
3. **Honest gaps** — missing upstream data shows "—" or an explicit investigation, never a fabricated number.
4. **Facts vs projections are labeled** — the Current row is real booked history; other prices are clearly modeled.
5. **Everything reconciles** — one averaging window drives units, revenue, ads and profit so `price × units = revenue` always holds.