utopia-ai
/
OOB-Dashboard
5
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: utopia-ai:main
Branches Tags
utopia-ai:main
utopia-ai:feat/hosted-auth-mysql-docker
..
compare: utopia-ai:feat/hosted-auth-mysql-docker
Branches Tags
utopia-ai:main
utopia-ai:feat/hosted-auth-mysql-docker

No commits in common. "main" and "feat/hosted-auth-mysql-docker" have entirely different histories.
main ... feat/hoste

11 changed files with 26 additions and 364 deletions
Show Stats Expand all files Collapse all files

18
.gitignore vendored
View File

@ -28,29 +28,13 @@ htmlcov/
# OS
.DS_Store
.AppleDouble
.LSOverride
._*
Thumbs.db
ehthumbs.db
Desktop.ini
# Editor / IDE
# Editor
.idea/
.vscode/
.cursor/
*.swp
*~
*.orig
*.bak
*.tmp
# Local launchers (macOS double-click scripts)
start.command
*.command
# Logs
*.log
# Excel lock / temp files
~$*.xlsx

37
README.md
View File

@ -12,14 +12,6 @@ makes it available to a team.
### The dashboard
Double-click **start.command** in this folder. That is the whole thing — it
checks Python, installs `openpyxl` if it is missing, picks a free port, starts
the server and opens your browser. Keep the Terminal window it opens; closing it
(or Ctrl+C) stops the dashboard. Double-clicking again while it is already
running just reopens the tab instead of starting a second copy.
From a terminal instead:
```bash
python3 serve.py
```
@ -154,31 +146,6 @@ which is precisely backwards. Only the delivery half of that change type
The window is always the span the data actually covers, never what you asked
Amazon for. One day of export can only ever say "no action in 1 day".
## When an export is unusable
The tool refuses a file rather than guessing, and says exactly why. The common
causes, all seen in the wild:
**No timestamps.** If `Date and time` and `Date and time (ISO)` are both empty,
there is no way to know *when* a campaign went in or out of budget, so hours
cannot be measured at all. Nothing can be salvaged from that file — re-run the
extraction. (If only the ISO column is empty, the human-readable one is used
instead and the file works fine.)
**Exporter warnings.** The export may carry an "Errors and Warnings" sheet.
`PARTIAL_PAGE_HARVEST` means the extension collected fewer rows than the page
reported, so changes are missing and every duration becomes a lower bound. That
is surfaced in Data Quality rather than swallowed.
**Mixed marketplaces.** One file can end up holding rows from two consoles —
`advertising.amazon.de` and `advertising.amazon.es`, say — if two extraction runs
were merged. Account spend and ROAS come from a single metadata block, so money
would be attributed to the wrong marketplace. Export each marketplace separately.
Loading several files at once? Any file that cannot be used is named in a red
banner at the top of the dashboard with the reason, and the rest are still
analysed. A dropped file is never silent.
## The diagnoses
| Label | Means |
@ -295,11 +262,11 @@ python3 tests/test_golden.py # the analysis
python3 tests/test_auth_smoke.py # the hosted app
```
`test_golden.py` is 38 checks: frozen totals from the reference export,
`test_golden.py` is 34 checks: frozen totals from the reference export,
structural invariants, overlapping-export handling, action classification, and
edge cases. The important ones are `test_chain_breaks_canary` and
`test_amazon_pacing_rows_are_not_actions`. It needs the reference export in
`data/`; the synthetic-fixture checks run without it.
`data/`; the 15 synthetic-fixture checks run without it.
`test_auth_smoke.py` is 53 checks covering the whole account lifecycle plus
upload, analyse and export. It runs against in-memory SQLite with mail captured

12
app/services/analysis.py
View File

@ -41,21 +41,15 @@ def run_analysis(history: Sequence[Path], perf: Path | None,
try:
evs, meta, qa = load_history(path)
except (ValueError, KeyError, OSError) as exc:
message = str(exc)
skipped.append(message if path.name in message
else f"{path.name}: {message}")
skipped.append(f"{path.name}: {exc}")
continue
events.extend(evs)
metas.append(meta)
qas.append(qa)
if not events:
if skipped:
# The per-file reason is the useful part; do not bury it behind a
# generic "could not be read".
raise ValueError(" ".join(skipped) if len(skipped) == 1
else "No file could be used. " + " ".join(skipped))
raise ValueError("The file has no readable change rows.")
detail = " ".join(skipped) or "no readable rows"
raise ValueError(f"None of the files could be read as a change-history export. {detail}")
events, overlap_rows = dedupe_events(events)
days = score_all(events, merge_gap_min=int(settings_in.get("merge_gap", 5)))

14
ppcbudget/excelout.py
View File

@ -556,20 +556,6 @@ def _sheet_quality(wb: Workbook, qas: list[QaReport], days: list[CampaignDay],
if m.status and m.status != "completed":
check(f"Extraction status: {qa.path.name[:28]}", False, m.status, "completed",
"A partial extraction can be missing whole campaigns, not just rows.")
if qa.warning_categories:
check(f"Exporter warnings: {qa.path.name[:28]}", False,
", ".join(f"{n}x {c}" for c, n in qa.warning_categories.items()), "none",
"The export recorded its own failures. PARTIAL_PAGE_HARVEST means it "
"collected fewer rows than the page reported, so changes are missing and "
"every duration here is a lower bound. Re-run the extraction.")
if qa.mixed_sources:
check(f"Mixed sources: {qa.path.name[:28]}", False,
f"{len(qa.source_urls)} consoles / {len(qa.row_marketplaces)} marketplaces",
"1 / 1",
"This file holds rows from more than one console or marketplace ("
+ ", ".join(sorted(qa.row_marketplaces)[:4])
+ "). Account spend and ROAS come from one metadata block, so money would be "
"attributed to the wrong marketplace. Export each marketplace separately.")
if len(qas) > 1:
check("Overlapping exports", True,

117
ppcbudget/ingest.py
View File

@ -57,8 +57,6 @@ class WorkbookMeta:
rows_exported: int | None = None
duplicates_skipped: int | None = None
pages_processed: int | None = None
started_at: str = ""
completed_at: str = ""
spend: float | None = None
sales: float | None = None
roas: float | None = None
@ -75,22 +73,10 @@ class QaReport:
rows_unparsable_time: int = 0
rows_no_campaign: int = 0 # account- or portfolio-level rows
rows_blank: int = 0
# Warnings the exporter itself recorded, and provenance of the rows. A file
# holding two marketplaces or two extraction runs was stitched together.
extraction_warnings: list[str] = field(default_factory=list)
warning_categories: dict = field(default_factory=dict)
source_urls: list[str] = field(default_factory=list)
row_marketplaces: list[str] = field(default_factory=list)
row_run_ids: list[str] = field(default_factory=list)
distinct_campaigns: int = 0
campaigns_with_budget_events: int = 0
date_keys: list[str] = field(default_factory=list)
@property
def mixed_sources(self) -> bool:
"""True when one file holds rows from more than one console or run."""
return len(self.source_urls) > 1 or len(self.row_marketplaces) > 1
@property
def rows_seen(self) -> int:
"""Every data row in the sheet, including ones we deliberately drop."""
@ -132,39 +118,6 @@ class QaReport:
return " = ".join([parts[0], " + ".join(parts[1:])])
_WHEN_FORMATS = (
"%d %b %Y %H:%M:%S.%f", "%d %b %Y %H:%M:%S", "%d %b %Y %H:%M",
"%m/%d/%Y %H:%M:%S.%f", "%m/%d/%Y %H:%M:%S", "%m/%d/%Y %H:%M",
"%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M",
)
def parse_when(*candidates) -> datetime | None:
"""First readable timestamp among the given cells.
The ISO column is preferred, but some exports fill only the human-readable
one, so both are tried before a row is written off.
"""
for raw in candidates:
if raw is None or raw == "":
continue
if isinstance(raw, datetime):
return raw
text = str(raw).strip()
if not text or text == "None":
continue
try:
return datetime.fromisoformat(text.replace("Z", "+00:00"))
except ValueError:
pass
for fmt in _WHEN_FORMATS:
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
return None
def _num(value) -> float | None:
if value is None or value == "":
return None
@ -215,8 +168,6 @@ def _read_meta(wb, path: Path) -> WorkbookMeta:
meta.date_range = str(pairs.get("Date range", "") or "")
meta.run_id = str(pairs.get("Extraction run ID", "") or "")
meta.status = str(pairs.get("Status", "") or "")
meta.started_at = str(pairs.get("Started at", "") or "")
meta.completed_at = str(pairs.get("Completed at", "") or "")
for attr, key in (
("rows_expected", "Rows expected"),
("rows_exported", "Rows exported"),
@ -238,37 +189,6 @@ def _read_meta(wb, path: Path) -> WorkbookMeta:
return meta
def _read_warnings(wb, qa: QaReport) -> None:
"""The exporter records its own failures on an 'Errors and Warnings' sheet.
Ignoring it means silently inheriting whatever it could not collect, so the
categories and a sample of messages are carried through to Data Quality.
"""
sheet = next((n for n in wb.sheetnames if "error" in n.lower()
or "warning" in n.lower()), None)
if sheet is None:
return
rows = list(wb[sheet].iter_rows(values_only=True))
if len(rows) < 2:
return
header = [str(h).strip().lower() if h else "" for h in rows[0]]
def idx(name):
return header.index(name) if name in header else None
i_cat, i_msg, i_pg = idx("category"), idx("message"), idx("page number")
for row in rows[1:]:
if not row or all(v is None for v in row):
continue
cat = str(row[i_cat]).strip() if i_cat is not None and row[i_cat] else "WARNING"
msg = str(row[i_msg]).strip() if i_msg is not None and row[i_msg] else ""
page = row[i_pg] if i_pg is not None else None
qa.warning_categories[cat] = qa.warning_categories.get(cat, 0) + 1
if len(qa.extraction_warnings) < 8:
qa.extraction_warnings.append(
f"{cat}" + (f" (page {page})" if page else "") + (f": {msg}" if msg else ""))
def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]:
"""Parse one change-history workbook."""
path = Path(path)
@ -278,7 +198,6 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
meta = _read_meta(wb, path)
qa = QaReport(path=path, meta=meta)
_read_warnings(wb, qa)
rows = wb["History"].iter_rows(values_only=True)
header = next(rows, None)
@ -312,8 +231,10 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
campaign = str(campaign).strip()
campaigns.add(campaign)
when = parse_when(cell(row, "Date and time (ISO)"), cell(row, "Date and time"))
if when is None:
iso = cell(row, "Date and time (ISO)")
try:
when = datetime.fromisoformat(str(iso))
except (TypeError, ValueError):
qa.rows_unparsable_time += 1
continue
@ -327,13 +248,6 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
if in_b[0] != in_b[1]:
qa.crossover_violations += 1
for name, bucket in (("Source URL", qa.source_urls),
("Marketplace", qa.row_marketplaces),
("Run ID", qa.row_run_ids)):
v = cell(row, name)
if v and str(v).strip() not in bucket:
bucket.append(str(v).strip())
date_key = when.date().isoformat()
dates.add(date_key)
events.append(
@ -356,29 +270,6 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
wb.close()
# No timestamps means no timeline, and no timeline means out-of-budget hours
# cannot be measured at all. Say exactly that, plus whatever the exporter
# already admitted, instead of a generic "could not be read".
if not events and qa.rows_unparsable_time:
detail = [
f"{path.name}: none of the {qa.rows_unparsable_time:,} rows have a usable "
"timestamp — 'Date and time' and 'Date and time (ISO)' are both empty. "
"Without a time on each change there is no way to reconstruct when a "
"campaign was in or out of budget, so hours cannot be measured."
]
if qa.warning_categories:
worst = ", ".join(f"{n}x {c}" for c, n in qa.warning_categories.items())
detail.append(f"The export also recorded its own problems ({worst}), "
"so rows are missing as well.")
if not qa.row_accounting_ok and meta.rows_expected:
detail.append(f"Its row counts do not reconcile either: "
f"{meta.rows_expected:,} expected - "
f"{meta.duplicates_skipped:,} duplicates != "
f"{meta.rows_exported:,} exported.")
detail.append("Re-run the extraction in the Amazon Ads console and let it "
"finish before downloading.")
raise ValueError(" ".join(detail))
qa.rows_parsed = len(events) + qa.rows_unparsable_time
qa.distinct_campaigns = len(campaigns)
qa.campaigns_with_budget_events = len(

18
ppcbudget/payload.py
View File

@ -106,24 +106,6 @@ def _quality(qas: list[QaReport], days: list[CampaignDay], totals: Totals,
if m.status and m.status != "completed":
add(f"Extraction status - {qa.path.name}", False, m.status,
"A partial extraction can be missing whole campaigns, not just rows.")
if qa.warning_categories:
worst = ", ".join(f"{n}x {c}" for c, n in qa.warning_categories.items())
add(f"Exporter warnings - {qa.path.name}", False, worst,
"The export recorded its own failures on an 'Errors and Warnings' sheet. "
"PARTIAL_PAGE_HARVEST means it collected fewer rows than the page reported, so "
"changes are missing and every duration here is a lower bound. Re-run the "
"extraction and let it finish. Sample: "
+ " | ".join(qa.extraction_warnings[:3]))
if qa.mixed_sources:
add(f"Mixed sources - {qa.path.name}", False,
f"{len(qa.source_urls)} consoles, {len(qa.row_marketplaces)} marketplaces",
"This one file holds rows from more than one Amazon console or marketplace: "
+ ", ".join(sorted(qa.row_marketplaces)[:4])
+ ". Account-level spend and ROAS come from a single metadata block, so the "
"money figures would be attributed to the wrong marketplace. Export each "
"marketplace separately.")
if qa.crossover_violations:
add("State machines crossed", False, qa.crossover_violations,
"A 'Campaign status' row mixed budget and delivery vocabularies, so splitting "

91
serve.py
View File

@ -16,7 +16,6 @@ from __future__ import annotations
import argparse
import json
import re
import shutil
import tempfile
import threading
@ -36,75 +35,7 @@ MAX_UPLOAD = 200 * 1024 * 1024
MIME = {".html": "text/html; charset=utf-8", ".css": "text/css; charset=utf-8",
".js": "text/javascript; charset=utf-8", ".svg": "image/svg+xml",
".ico": "image/x-icon", ".woff2": "font/woff2"}
def _web_file(url_path: str) -> Path | None:
"""Map a URL path to a file under web/, mirroring the hosted /static mount."""
rel = url_path.removeprefix("/static/").lstrip("/")
if url_path in ("/", ""):
rel = "index.html"
target = (WEB / rel).resolve()
if not str(target).startswith(str(WEB.resolve())) or not target.is_file():
return None
return target
_DISPOSITION = re.compile(
r'content-disposition:\s*form-data;\s*(.*)',
re.IGNORECASE,
)
_FIELD = re.compile(r'name="([^"]+)"')
_FILENAME = re.compile(r'filename="([^"]*)"')
def _parse_multipart(body: bytes, content_type: str) -> tuple[str, bytes, str]:
"""Extract (filename, file_bytes, kind) from a multipart upload."""
if "multipart/form-data" not in content_type:
raise ValueError("Expected a multipart upload.")
boundary = None
for part in content_type.split(";"):
part = part.strip()
if part.startswith("boundary="):
boundary = part[9:].strip().strip('"')
break
if not boundary:
raise ValueError("Upload is missing a boundary.")
filename = "upload.xlsx"
kind = "history"
file_data = b""
for section in body.split(f"--{boundary}".encode()):
if not section or section in (b"--", b"--\r\n"):
continue
chunk = section.lstrip(b"\r\n")
if not chunk:
continue
header_end = chunk.find(b"\r\n\r\n")
if header_end < 0:
continue
headers = chunk[:header_end].decode("latin-1", errors="replace")
payload = chunk[header_end + 4:]
if payload.endswith(b"\r\n"):
payload = payload[:-2]
disp = _DISPOSITION.search(headers)
if not disp:
continue
name_match = _FIELD.search(disp.group(1))
if not name_match:
continue
name = name_match.group(1)
if name == "kind":
kind = payload.decode("utf-8", errors="replace").strip() or "history"
elif name == "file":
file_data = payload
fn = _FILENAME.search(disp.group(1))
if fn and fn.group(1):
filename = fn.group(1)
if not file_data:
raise ValueError("That file was empty.")
return filename, file_data, kind
".ico": "image/x-icon"}
class Session:
@ -118,12 +49,8 @@ class Session:
self.lock = threading.Lock()
def add(self, name: str, data: bytes, kind: str) -> Path:
# One subfolder per upload keeps path.name the user's real filename, so
# error messages and the Data Quality sheet never show a "0_" prefix.
safe = Path(name).name.replace("/", "_") or "upload.xlsx"
slot = self.dir / f"{len(self.history)}{'p' if kind == 'perf' else ''}"
slot.mkdir(parents=True, exist_ok=True)
target = slot / safe
target = self.dir / f"{len(self.history)}_{safe}"
target.write_bytes(data)
if kind == "perf":
self.perf = target
@ -192,9 +119,8 @@ class Handler(BaseHTTPRequestHandler):
if path == "/api/state":
self._json({
"mode": "local",
"history": [p.name for p in SESSION.history],
"perf": SESSION.perf.name if SESSION.perf else None,
"history": [p.name.split("_", 1)[-1] for p in SESSION.history],
"perf": SESSION.perf.name.split("_", 1)[-1] if SESSION.perf else None,
})
return
@ -202,8 +128,9 @@ class Handler(BaseHTTPRequestHandler):
self._export(parse_qs(route.query).get("format", ["xlsx"])[0])
return
target = _web_file(path)
if target is None:
rel = "index.html" if path in ("/", "") else path.lstrip("/")
target = (WEB / rel).resolve()
if not str(target).startswith(str(WEB.resolve())) or not target.is_file():
self._send(HTTPStatus.NOT_FOUND, b"Not found", "text/plain; charset=utf-8")
return
self._send(HTTPStatus.OK, target.read_bytes(),
@ -213,10 +140,6 @@ class Handler(BaseHTTPRequestHandler):
path = urlparse(self.path).path
try:
if path == "/api/upload":
ctype = self.headers.get("Content-Type", "")
if "multipart/form-data" in ctype:
name, data, kind = _parse_multipart(self._body(), ctype)
else:
name = self.headers.get("X-Filename", "upload.xlsx")
kind = self.headers.get("X-Kind", "history")
data = self._body()

33
tests/test_golden.py
View File

@ -137,39 +137,6 @@ def test_row_accounting_explains_unscoreable_rows():
assert "UNACCOUNTED" in broken.accounting_detail
# ------------------------------------------------------ broken exports
def test_human_readable_timestamp_is_a_valid_fallback():
"""Some extractions fill only 'Date and time', not the ISO column."""
from ppcbudget.ingest import parse_when
from datetime import datetime
assert parse_when(None, "5 Aug 2026 23:59") == datetime(2026, 8, 5, 23, 59)
assert parse_when("", "5 Aug 2026 14:17:18.440").hour == 14
assert parse_when("2026-08-05T14:17:18", "ignored").minute == 17
assert parse_when("2026-08-05T14:17:18Z") is not None
# The ISO column wins when both are present and disagree.
assert parse_when("2026-08-05T01:00:00", "5 Aug 2026 23:59").hour == 1
def test_no_timestamp_anywhere_is_unusable_and_says_why():
"""A timeline needs a time on every row. Failing quietly would be worse."""
from ppcbudget.ingest import parse_when
assert parse_when(None, None) is None
assert parse_when("", "") is None
assert parse_when("None", "not a date") is None
def test_exporter_warnings_are_never_ignored():
"""The reference file is clean, but the reader must expose the fields the
Data Quality panel relies on."""
qa = load()["qa"]
assert qa.warning_categories == {}
assert qa.extraction_warnings == []
assert not qa.mixed_sources
assert len(qa.source_urls) == 1, "one export should come from one console"
assert len(qa.row_marketplaces) == 1
# ------------------------------------------------------- last meaningful action
def test_amazon_pacing_rows_are_not_actions():

18
web/app.js
View File

@ -133,16 +133,6 @@ function render() {
row.unt = row.act.unt;
}
// A file that could not be read must never disappear quietly.
const skipped = d.skipped || [];
$('skipped').hidden = skipped.length === 0;
$('skipped-list').innerHTML = skipped.map((line) => {
const cut = line.indexOf(': ');
const name = cut > 0 ? line.slice(0, cut) : 'A file';
const why = cut > 0 ? line.slice(cut + 2) : line;
return `<div><b>${escapeHtml(name)}</b>${escapeHtml(why)}</div>`;
}).join('');
$('grain').hidden = t.days < 2;
renderAnswer(t, m);
renderKpis(t, m);
@ -823,13 +813,7 @@ $('settings').addEventListener('close', (ev) => {
// expired session redirects to the sign-in page rather than silently rendering
// an empty dashboard.
A.api('/api/state').then((r) => r.json()).then((s) => {
if (s.mode === 'local') {
const note = $('upload-note');
if (note) {
note.textContent = 'Everything stays on this machine. Nothing is uploaded anywhere.';
}
} else if (s.user) {
$('btn-signout').hidden = false;
if (s.user) {
$('whoami').textContent = s.user.name || s.user.email;
$('whoami').title = s.user.email;
$('link-admin').hidden = !s.user.is_admin;

9
web/index.html
View File

@ -25,7 +25,7 @@
<!-- Outside the set that stage() toggles, so these stay visible throughout. -->
<span id="whoami" class="muted" style="align-self:center;font-size:12px"></span>
<a id="link-admin" href="/admin" hidden><button type="button" class="ghost">Accounts</button></a>
<button id="btn-signout" class="ghost" hidden>Sign out</button>
<button id="btn-signout" class="ghost">Sign out</button>
</div>
</header>
@ -44,7 +44,7 @@
<h2>Drop your change-history exports here</h2>
<p>One file or a whole week of them. <button type="button" class="linklike" id="pick">Choose files</button>
&mdash; or drop a folder.</p>
<p class="fineprint" id="upload-note">Your files are uploaded to this server, analysed, and deleted
<p class="fineprint">Your files are uploaded to this server, analysed, and deleted
when you sign out or go idle. Nobody else using this dashboard can see them.</p>
<input type="file" id="file-input" multiple accept=".xlsx,.xlsm" hidden>
</div>
@ -84,11 +84,6 @@
<!-- --------------------------------------------------------- dashboard -->
<section id="stage-dash" hidden>
<div class="panel skipped-panel" id="skipped" hidden>
<h2>Some files were not used</h2>
<div id="skipped-list"></div>
</div>
<div class="answer panel" id="answer">
<h2>A typical campaign's day</h2>
<p class="lede" id="answer-lede"></p>

11
web/styles.css
View File

@ -469,17 +469,6 @@ table.grid th small { display: block; font-size: 9px; font-weight: 500; opacity:
.axis { display: flex; justify-content: space-between; font-size: 10px; color: var(--ink-3); margin-top: 4px; }
.drawer h3 { margin: 18px 0 8px; }
/* ------------------------------------------------------------ skipped files */
.skipped-panel { border-left: 4px solid var(--red); }
.skipped-panel h2 { color: var(--red); }
#skipped-list { margin-top: 8px; display: grid; gap: 8px; }
#skipped-list div {
background: var(--red-soft); border-radius: 8px; padding: 10px 13px;
font-size: 12.5px; color: var(--ink); word-break: break-word;
}
#skipped-list b { display: block; margin-bottom: 3px; }
/* ------------------------------------------------------- action log (drawer) */
.act-none, .act-yes {