Compare commits
No commits in common. "main" and "feat/hosted-auth-mysql-docker" have entirely different histories.
main
...
feat/hoste
|
|
@ -28,29 +28,13 @@ htmlcov/
|
||||||
|
|
||||||
# OS
|
# OS
|
||||||
.DS_Store
|
.DS_Store
|
||||||
.AppleDouble
|
|
||||||
.LSOverride
|
|
||||||
._*
|
|
||||||
Thumbs.db
|
Thumbs.db
|
||||||
ehthumbs.db
|
|
||||||
Desktop.ini
|
|
||||||
|
|
||||||
# Editor / IDE
|
# Editor
|
||||||
.idea/
|
.idea/
|
||||||
.vscode/
|
.vscode/
|
||||||
.cursor/
|
|
||||||
*.swp
|
*.swp
|
||||||
*~
|
*~
|
||||||
*.orig
|
|
||||||
*.bak
|
|
||||||
*.tmp
|
|
||||||
|
|
||||||
# Local launchers (macOS double-click scripts)
|
|
||||||
start.command
|
|
||||||
*.command
|
|
||||||
|
|
||||||
# Logs
|
|
||||||
*.log
|
|
||||||
|
|
||||||
# Excel lock / temp files
|
# Excel lock / temp files
|
||||||
~$*.xlsx
|
~$*.xlsx
|
||||||
|
|
|
||||||
37
README.md
37
README.md
|
|
@ -12,14 +12,6 @@ makes it available to a team.
|
||||||
|
|
||||||
### The dashboard
|
### The dashboard
|
||||||
|
|
||||||
Double-click **start.command** in this folder. That is the whole thing — it
|
|
||||||
checks Python, installs `openpyxl` if it is missing, picks a free port, starts
|
|
||||||
the server and opens your browser. Keep the Terminal window it opens; closing it
|
|
||||||
(or Ctrl+C) stops the dashboard. Double-clicking again while it is already
|
|
||||||
running just reopens the tab instead of starting a second copy.
|
|
||||||
|
|
||||||
From a terminal instead:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 serve.py
|
python3 serve.py
|
||||||
```
|
```
|
||||||
|
|
@ -154,31 +146,6 @@ which is precisely backwards. Only the delivery half of that change type
|
||||||
The window is always the span the data actually covers, never what you asked
|
The window is always the span the data actually covers, never what you asked
|
||||||
Amazon for. One day of export can only ever say "no action in 1 day".
|
Amazon for. One day of export can only ever say "no action in 1 day".
|
||||||
|
|
||||||
## When an export is unusable
|
|
||||||
|
|
||||||
The tool refuses a file rather than guessing, and says exactly why. The common
|
|
||||||
causes, all seen in the wild:
|
|
||||||
|
|
||||||
**No timestamps.** If `Date and time` and `Date and time (ISO)` are both empty,
|
|
||||||
there is no way to know *when* a campaign went in or out of budget, so hours
|
|
||||||
cannot be measured at all. Nothing can be salvaged from that file — re-run the
|
|
||||||
extraction. (If only the ISO column is empty, the human-readable one is used
|
|
||||||
instead and the file works fine.)
|
|
||||||
|
|
||||||
**Exporter warnings.** The export may carry an "Errors and Warnings" sheet.
|
|
||||||
`PARTIAL_PAGE_HARVEST` means the extension collected fewer rows than the page
|
|
||||||
reported, so changes are missing and every duration becomes a lower bound. That
|
|
||||||
is surfaced in Data Quality rather than swallowed.
|
|
||||||
|
|
||||||
**Mixed marketplaces.** One file can end up holding rows from two consoles —
|
|
||||||
`advertising.amazon.de` and `advertising.amazon.es`, say — if two extraction runs
|
|
||||||
were merged. Account spend and ROAS come from a single metadata block, so money
|
|
||||||
would be attributed to the wrong marketplace. Export each marketplace separately.
|
|
||||||
|
|
||||||
Loading several files at once? Any file that cannot be used is named in a red
|
|
||||||
banner at the top of the dashboard with the reason, and the rest are still
|
|
||||||
analysed. A dropped file is never silent.
|
|
||||||
|
|
||||||
## The diagnoses
|
## The diagnoses
|
||||||
|
|
||||||
| Label | Means |
|
| Label | Means |
|
||||||
|
|
@ -295,11 +262,11 @@ python3 tests/test_golden.py # the analysis
|
||||||
python3 tests/test_auth_smoke.py # the hosted app
|
python3 tests/test_auth_smoke.py # the hosted app
|
||||||
```
|
```
|
||||||
|
|
||||||
`test_golden.py` is 38 checks: frozen totals from the reference export,
|
`test_golden.py` is 34 checks: frozen totals from the reference export,
|
||||||
structural invariants, overlapping-export handling, action classification, and
|
structural invariants, overlapping-export handling, action classification, and
|
||||||
edge cases. The important ones are `test_chain_breaks_canary` and
|
edge cases. The important ones are `test_chain_breaks_canary` and
|
||||||
`test_amazon_pacing_rows_are_not_actions`. It needs the reference export in
|
`test_amazon_pacing_rows_are_not_actions`. It needs the reference export in
|
||||||
`data/`; the synthetic-fixture checks run without it.
|
`data/`; the 15 synthetic-fixture checks run without it.
|
||||||
|
|
||||||
`test_auth_smoke.py` is 53 checks covering the whole account lifecycle plus
|
`test_auth_smoke.py` is 53 checks covering the whole account lifecycle plus
|
||||||
upload, analyse and export. It runs against in-memory SQLite with mail captured
|
upload, analyse and export. It runs against in-memory SQLite with mail captured
|
||||||
|
|
|
||||||
|
|
@ -41,21 +41,15 @@ def run_analysis(history: Sequence[Path], perf: Path | None,
|
||||||
try:
|
try:
|
||||||
evs, meta, qa = load_history(path)
|
evs, meta, qa = load_history(path)
|
||||||
except (ValueError, KeyError, OSError) as exc:
|
except (ValueError, KeyError, OSError) as exc:
|
||||||
message = str(exc)
|
skipped.append(f"{path.name}: {exc}")
|
||||||
skipped.append(message if path.name in message
|
|
||||||
else f"{path.name}: {message}")
|
|
||||||
continue
|
continue
|
||||||
events.extend(evs)
|
events.extend(evs)
|
||||||
metas.append(meta)
|
metas.append(meta)
|
||||||
qas.append(qa)
|
qas.append(qa)
|
||||||
|
|
||||||
if not events:
|
if not events:
|
||||||
if skipped:
|
detail = " ".join(skipped) or "no readable rows"
|
||||||
# The per-file reason is the useful part; do not bury it behind a
|
raise ValueError(f"None of the files could be read as a change-history export. {detail}")
|
||||||
# generic "could not be read".
|
|
||||||
raise ValueError(" ".join(skipped) if len(skipped) == 1
|
|
||||||
else "No file could be used. " + " ".join(skipped))
|
|
||||||
raise ValueError("The file has no readable change rows.")
|
|
||||||
|
|
||||||
events, overlap_rows = dedupe_events(events)
|
events, overlap_rows = dedupe_events(events)
|
||||||
days = score_all(events, merge_gap_min=int(settings_in.get("merge_gap", 5)))
|
days = score_all(events, merge_gap_min=int(settings_in.get("merge_gap", 5)))
|
||||||
|
|
|
||||||
|
|
@ -556,20 +556,6 @@ def _sheet_quality(wb: Workbook, qas: list[QaReport], days: list[CampaignDay],
|
||||||
if m.status and m.status != "completed":
|
if m.status and m.status != "completed":
|
||||||
check(f"Extraction status: {qa.path.name[:28]}", False, m.status, "completed",
|
check(f"Extraction status: {qa.path.name[:28]}", False, m.status, "completed",
|
||||||
"A partial extraction can be missing whole campaigns, not just rows.")
|
"A partial extraction can be missing whole campaigns, not just rows.")
|
||||||
if qa.warning_categories:
|
|
||||||
check(f"Exporter warnings: {qa.path.name[:28]}", False,
|
|
||||||
", ".join(f"{n}x {c}" for c, n in qa.warning_categories.items()), "none",
|
|
||||||
"The export recorded its own failures. PARTIAL_PAGE_HARVEST means it "
|
|
||||||
"collected fewer rows than the page reported, so changes are missing and "
|
|
||||||
"every duration here is a lower bound. Re-run the extraction.")
|
|
||||||
if qa.mixed_sources:
|
|
||||||
check(f"Mixed sources: {qa.path.name[:28]}", False,
|
|
||||||
f"{len(qa.source_urls)} consoles / {len(qa.row_marketplaces)} marketplaces",
|
|
||||||
"1 / 1",
|
|
||||||
"This file holds rows from more than one console or marketplace ("
|
|
||||||
+ ", ".join(sorted(qa.row_marketplaces)[:4])
|
|
||||||
+ "). Account spend and ROAS come from one metadata block, so money would be "
|
|
||||||
"attributed to the wrong marketplace. Export each marketplace separately.")
|
|
||||||
|
|
||||||
if len(qas) > 1:
|
if len(qas) > 1:
|
||||||
check("Overlapping exports", True,
|
check("Overlapping exports", True,
|
||||||
|
|
|
||||||
|
|
@ -57,8 +57,6 @@ class WorkbookMeta:
|
||||||
rows_exported: int | None = None
|
rows_exported: int | None = None
|
||||||
duplicates_skipped: int | None = None
|
duplicates_skipped: int | None = None
|
||||||
pages_processed: int | None = None
|
pages_processed: int | None = None
|
||||||
started_at: str = ""
|
|
||||||
completed_at: str = ""
|
|
||||||
spend: float | None = None
|
spend: float | None = None
|
||||||
sales: float | None = None
|
sales: float | None = None
|
||||||
roas: float | None = None
|
roas: float | None = None
|
||||||
|
|
@ -75,22 +73,10 @@ class QaReport:
|
||||||
rows_unparsable_time: int = 0
|
rows_unparsable_time: int = 0
|
||||||
rows_no_campaign: int = 0 # account- or portfolio-level rows
|
rows_no_campaign: int = 0 # account- or portfolio-level rows
|
||||||
rows_blank: int = 0
|
rows_blank: int = 0
|
||||||
# Warnings the exporter itself recorded, and provenance of the rows. A file
|
|
||||||
# holding two marketplaces or two extraction runs was stitched together.
|
|
||||||
extraction_warnings: list[str] = field(default_factory=list)
|
|
||||||
warning_categories: dict = field(default_factory=dict)
|
|
||||||
source_urls: list[str] = field(default_factory=list)
|
|
||||||
row_marketplaces: list[str] = field(default_factory=list)
|
|
||||||
row_run_ids: list[str] = field(default_factory=list)
|
|
||||||
distinct_campaigns: int = 0
|
distinct_campaigns: int = 0
|
||||||
campaigns_with_budget_events: int = 0
|
campaigns_with_budget_events: int = 0
|
||||||
date_keys: list[str] = field(default_factory=list)
|
date_keys: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
@property
|
|
||||||
def mixed_sources(self) -> bool:
|
|
||||||
"""True when one file holds rows from more than one console or run."""
|
|
||||||
return len(self.source_urls) > 1 or len(self.row_marketplaces) > 1
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def rows_seen(self) -> int:
|
def rows_seen(self) -> int:
|
||||||
"""Every data row in the sheet, including ones we deliberately drop."""
|
"""Every data row in the sheet, including ones we deliberately drop."""
|
||||||
|
|
@ -132,39 +118,6 @@ class QaReport:
|
||||||
return " = ".join([parts[0], " + ".join(parts[1:])])
|
return " = ".join([parts[0], " + ".join(parts[1:])])
|
||||||
|
|
||||||
|
|
||||||
_WHEN_FORMATS = (
|
|
||||||
"%d %b %Y %H:%M:%S.%f", "%d %b %Y %H:%M:%S", "%d %b %Y %H:%M",
|
|
||||||
"%m/%d/%Y %H:%M:%S.%f", "%m/%d/%Y %H:%M:%S", "%m/%d/%Y %H:%M",
|
|
||||||
"%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def parse_when(*candidates) -> datetime | None:
|
|
||||||
"""First readable timestamp among the given cells.
|
|
||||||
|
|
||||||
The ISO column is preferred, but some exports fill only the human-readable
|
|
||||||
one, so both are tried before a row is written off.
|
|
||||||
"""
|
|
||||||
for raw in candidates:
|
|
||||||
if raw is None or raw == "":
|
|
||||||
continue
|
|
||||||
if isinstance(raw, datetime):
|
|
||||||
return raw
|
|
||||||
text = str(raw).strip()
|
|
||||||
if not text or text == "None":
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
return datetime.fromisoformat(text.replace("Z", "+00:00"))
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
for fmt in _WHEN_FORMATS:
|
|
||||||
try:
|
|
||||||
return datetime.strptime(text, fmt)
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _num(value) -> float | None:
|
def _num(value) -> float | None:
|
||||||
if value is None or value == "":
|
if value is None or value == "":
|
||||||
return None
|
return None
|
||||||
|
|
@ -215,8 +168,6 @@ def _read_meta(wb, path: Path) -> WorkbookMeta:
|
||||||
meta.date_range = str(pairs.get("Date range", "") or "")
|
meta.date_range = str(pairs.get("Date range", "") or "")
|
||||||
meta.run_id = str(pairs.get("Extraction run ID", "") or "")
|
meta.run_id = str(pairs.get("Extraction run ID", "") or "")
|
||||||
meta.status = str(pairs.get("Status", "") or "")
|
meta.status = str(pairs.get("Status", "") or "")
|
||||||
meta.started_at = str(pairs.get("Started at", "") or "")
|
|
||||||
meta.completed_at = str(pairs.get("Completed at", "") or "")
|
|
||||||
for attr, key in (
|
for attr, key in (
|
||||||
("rows_expected", "Rows expected"),
|
("rows_expected", "Rows expected"),
|
||||||
("rows_exported", "Rows exported"),
|
("rows_exported", "Rows exported"),
|
||||||
|
|
@ -238,37 +189,6 @@ def _read_meta(wb, path: Path) -> WorkbookMeta:
|
||||||
return meta
|
return meta
|
||||||
|
|
||||||
|
|
||||||
def _read_warnings(wb, qa: QaReport) -> None:
|
|
||||||
"""The exporter records its own failures on an 'Errors and Warnings' sheet.
|
|
||||||
|
|
||||||
Ignoring it means silently inheriting whatever it could not collect, so the
|
|
||||||
categories and a sample of messages are carried through to Data Quality.
|
|
||||||
"""
|
|
||||||
sheet = next((n for n in wb.sheetnames if "error" in n.lower()
|
|
||||||
or "warning" in n.lower()), None)
|
|
||||||
if sheet is None:
|
|
||||||
return
|
|
||||||
rows = list(wb[sheet].iter_rows(values_only=True))
|
|
||||||
if len(rows) < 2:
|
|
||||||
return
|
|
||||||
header = [str(h).strip().lower() if h else "" for h in rows[0]]
|
|
||||||
|
|
||||||
def idx(name):
|
|
||||||
return header.index(name) if name in header else None
|
|
||||||
|
|
||||||
i_cat, i_msg, i_pg = idx("category"), idx("message"), idx("page number")
|
|
||||||
for row in rows[1:]:
|
|
||||||
if not row or all(v is None for v in row):
|
|
||||||
continue
|
|
||||||
cat = str(row[i_cat]).strip() if i_cat is not None and row[i_cat] else "WARNING"
|
|
||||||
msg = str(row[i_msg]).strip() if i_msg is not None and row[i_msg] else ""
|
|
||||||
page = row[i_pg] if i_pg is not None else None
|
|
||||||
qa.warning_categories[cat] = qa.warning_categories.get(cat, 0) + 1
|
|
||||||
if len(qa.extraction_warnings) < 8:
|
|
||||||
qa.extraction_warnings.append(
|
|
||||||
f"{cat}" + (f" (page {page})" if page else "") + (f": {msg}" if msg else ""))
|
|
||||||
|
|
||||||
|
|
||||||
def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]:
|
def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]:
|
||||||
"""Parse one change-history workbook."""
|
"""Parse one change-history workbook."""
|
||||||
path = Path(path)
|
path = Path(path)
|
||||||
|
|
@ -278,7 +198,6 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
|
||||||
|
|
||||||
meta = _read_meta(wb, path)
|
meta = _read_meta(wb, path)
|
||||||
qa = QaReport(path=path, meta=meta)
|
qa = QaReport(path=path, meta=meta)
|
||||||
_read_warnings(wb, qa)
|
|
||||||
|
|
||||||
rows = wb["History"].iter_rows(values_only=True)
|
rows = wb["History"].iter_rows(values_only=True)
|
||||||
header = next(rows, None)
|
header = next(rows, None)
|
||||||
|
|
@ -312,8 +231,10 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
|
||||||
campaign = str(campaign).strip()
|
campaign = str(campaign).strip()
|
||||||
campaigns.add(campaign)
|
campaigns.add(campaign)
|
||||||
|
|
||||||
when = parse_when(cell(row, "Date and time (ISO)"), cell(row, "Date and time"))
|
iso = cell(row, "Date and time (ISO)")
|
||||||
if when is None:
|
try:
|
||||||
|
when = datetime.fromisoformat(str(iso))
|
||||||
|
except (TypeError, ValueError):
|
||||||
qa.rows_unparsable_time += 1
|
qa.rows_unparsable_time += 1
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
|
@ -327,13 +248,6 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
|
||||||
if in_b[0] != in_b[1]:
|
if in_b[0] != in_b[1]:
|
||||||
qa.crossover_violations += 1
|
qa.crossover_violations += 1
|
||||||
|
|
||||||
for name, bucket in (("Source URL", qa.source_urls),
|
|
||||||
("Marketplace", qa.row_marketplaces),
|
|
||||||
("Run ID", qa.row_run_ids)):
|
|
||||||
v = cell(row, name)
|
|
||||||
if v and str(v).strip() not in bucket:
|
|
||||||
bucket.append(str(v).strip())
|
|
||||||
|
|
||||||
date_key = when.date().isoformat()
|
date_key = when.date().isoformat()
|
||||||
dates.add(date_key)
|
dates.add(date_key)
|
||||||
events.append(
|
events.append(
|
||||||
|
|
@ -356,29 +270,6 @@ def load_history(path: str | Path) -> tuple[list[Event], WorkbookMeta, QaReport]
|
||||||
|
|
||||||
wb.close()
|
wb.close()
|
||||||
|
|
||||||
# No timestamps means no timeline, and no timeline means out-of-budget hours
|
|
||||||
# cannot be measured at all. Say exactly that, plus whatever the exporter
|
|
||||||
# already admitted, instead of a generic "could not be read".
|
|
||||||
if not events and qa.rows_unparsable_time:
|
|
||||||
detail = [
|
|
||||||
f"{path.name}: none of the {qa.rows_unparsable_time:,} rows have a usable "
|
|
||||||
"timestamp — 'Date and time' and 'Date and time (ISO)' are both empty. "
|
|
||||||
"Without a time on each change there is no way to reconstruct when a "
|
|
||||||
"campaign was in or out of budget, so hours cannot be measured."
|
|
||||||
]
|
|
||||||
if qa.warning_categories:
|
|
||||||
worst = ", ".join(f"{n}x {c}" for c, n in qa.warning_categories.items())
|
|
||||||
detail.append(f"The export also recorded its own problems ({worst}), "
|
|
||||||
"so rows are missing as well.")
|
|
||||||
if not qa.row_accounting_ok and meta.rows_expected:
|
|
||||||
detail.append(f"Its row counts do not reconcile either: "
|
|
||||||
f"{meta.rows_expected:,} expected - "
|
|
||||||
f"{meta.duplicates_skipped:,} duplicates != "
|
|
||||||
f"{meta.rows_exported:,} exported.")
|
|
||||||
detail.append("Re-run the extraction in the Amazon Ads console and let it "
|
|
||||||
"finish before downloading.")
|
|
||||||
raise ValueError(" ".join(detail))
|
|
||||||
|
|
||||||
qa.rows_parsed = len(events) + qa.rows_unparsable_time
|
qa.rows_parsed = len(events) + qa.rows_unparsable_time
|
||||||
qa.distinct_campaigns = len(campaigns)
|
qa.distinct_campaigns = len(campaigns)
|
||||||
qa.campaigns_with_budget_events = len(
|
qa.campaigns_with_budget_events = len(
|
||||||
|
|
|
||||||
|
|
@ -106,24 +106,6 @@ def _quality(qas: list[QaReport], days: list[CampaignDay], totals: Totals,
|
||||||
if m.status and m.status != "completed":
|
if m.status and m.status != "completed":
|
||||||
add(f"Extraction status - {qa.path.name}", False, m.status,
|
add(f"Extraction status - {qa.path.name}", False, m.status,
|
||||||
"A partial extraction can be missing whole campaigns, not just rows.")
|
"A partial extraction can be missing whole campaigns, not just rows.")
|
||||||
if qa.warning_categories:
|
|
||||||
worst = ", ".join(f"{n}x {c}" for c, n in qa.warning_categories.items())
|
|
||||||
add(f"Exporter warnings - {qa.path.name}", False, worst,
|
|
||||||
"The export recorded its own failures on an 'Errors and Warnings' sheet. "
|
|
||||||
"PARTIAL_PAGE_HARVEST means it collected fewer rows than the page reported, so "
|
|
||||||
"changes are missing and every duration here is a lower bound. Re-run the "
|
|
||||||
"extraction and let it finish. Sample: "
|
|
||||||
+ " | ".join(qa.extraction_warnings[:3]))
|
|
||||||
|
|
||||||
if qa.mixed_sources:
|
|
||||||
add(f"Mixed sources - {qa.path.name}", False,
|
|
||||||
f"{len(qa.source_urls)} consoles, {len(qa.row_marketplaces)} marketplaces",
|
|
||||||
"This one file holds rows from more than one Amazon console or marketplace: "
|
|
||||||
+ ", ".join(sorted(qa.row_marketplaces)[:4])
|
|
||||||
+ ". Account-level spend and ROAS come from a single metadata block, so the "
|
|
||||||
"money figures would be attributed to the wrong marketplace. Export each "
|
|
||||||
"marketplace separately.")
|
|
||||||
|
|
||||||
if qa.crossover_violations:
|
if qa.crossover_violations:
|
||||||
add("State machines crossed", False, qa.crossover_violations,
|
add("State machines crossed", False, qa.crossover_violations,
|
||||||
"A 'Campaign status' row mixed budget and delivery vocabularies, so splitting "
|
"A 'Campaign status' row mixed budget and delivery vocabularies, so splitting "
|
||||||
|
|
|
||||||
91
serve.py
91
serve.py
|
|
@ -16,7 +16,6 @@ from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
import json
|
||||||
import re
|
|
||||||
import shutil
|
import shutil
|
||||||
import tempfile
|
import tempfile
|
||||||
import threading
|
import threading
|
||||||
|
|
@ -36,75 +35,7 @@ MAX_UPLOAD = 200 * 1024 * 1024
|
||||||
|
|
||||||
MIME = {".html": "text/html; charset=utf-8", ".css": "text/css; charset=utf-8",
|
MIME = {".html": "text/html; charset=utf-8", ".css": "text/css; charset=utf-8",
|
||||||
".js": "text/javascript; charset=utf-8", ".svg": "image/svg+xml",
|
".js": "text/javascript; charset=utf-8", ".svg": "image/svg+xml",
|
||||||
".ico": "image/x-icon", ".woff2": "font/woff2"}
|
".ico": "image/x-icon"}
|
||||||
|
|
||||||
|
|
||||||
def _web_file(url_path: str) -> Path | None:
|
|
||||||
"""Map a URL path to a file under web/, mirroring the hosted /static mount."""
|
|
||||||
rel = url_path.removeprefix("/static/").lstrip("/")
|
|
||||||
if url_path in ("/", ""):
|
|
||||||
rel = "index.html"
|
|
||||||
target = (WEB / rel).resolve()
|
|
||||||
if not str(target).startswith(str(WEB.resolve())) or not target.is_file():
|
|
||||||
return None
|
|
||||||
return target
|
|
||||||
|
|
||||||
|
|
||||||
_DISPOSITION = re.compile(
|
|
||||||
r'content-disposition:\s*form-data;\s*(.*)',
|
|
||||||
re.IGNORECASE,
|
|
||||||
)
|
|
||||||
_FIELD = re.compile(r'name="([^"]+)"')
|
|
||||||
_FILENAME = re.compile(r'filename="([^"]*)"')
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_multipart(body: bytes, content_type: str) -> tuple[str, bytes, str]:
|
|
||||||
"""Extract (filename, file_bytes, kind) from a multipart upload."""
|
|
||||||
if "multipart/form-data" not in content_type:
|
|
||||||
raise ValueError("Expected a multipart upload.")
|
|
||||||
boundary = None
|
|
||||||
for part in content_type.split(";"):
|
|
||||||
part = part.strip()
|
|
||||||
if part.startswith("boundary="):
|
|
||||||
boundary = part[9:].strip().strip('"')
|
|
||||||
break
|
|
||||||
if not boundary:
|
|
||||||
raise ValueError("Upload is missing a boundary.")
|
|
||||||
|
|
||||||
filename = "upload.xlsx"
|
|
||||||
kind = "history"
|
|
||||||
file_data = b""
|
|
||||||
for section in body.split(f"--{boundary}".encode()):
|
|
||||||
if not section or section in (b"--", b"--\r\n"):
|
|
||||||
continue
|
|
||||||
chunk = section.lstrip(b"\r\n")
|
|
||||||
if not chunk:
|
|
||||||
continue
|
|
||||||
header_end = chunk.find(b"\r\n\r\n")
|
|
||||||
if header_end < 0:
|
|
||||||
continue
|
|
||||||
headers = chunk[:header_end].decode("latin-1", errors="replace")
|
|
||||||
payload = chunk[header_end + 4:]
|
|
||||||
if payload.endswith(b"\r\n"):
|
|
||||||
payload = payload[:-2]
|
|
||||||
|
|
||||||
disp = _DISPOSITION.search(headers)
|
|
||||||
if not disp:
|
|
||||||
continue
|
|
||||||
name_match = _FIELD.search(disp.group(1))
|
|
||||||
if not name_match:
|
|
||||||
continue
|
|
||||||
name = name_match.group(1)
|
|
||||||
if name == "kind":
|
|
||||||
kind = payload.decode("utf-8", errors="replace").strip() or "history"
|
|
||||||
elif name == "file":
|
|
||||||
file_data = payload
|
|
||||||
fn = _FILENAME.search(disp.group(1))
|
|
||||||
if fn and fn.group(1):
|
|
||||||
filename = fn.group(1)
|
|
||||||
if not file_data:
|
|
||||||
raise ValueError("That file was empty.")
|
|
||||||
return filename, file_data, kind
|
|
||||||
|
|
||||||
|
|
||||||
class Session:
|
class Session:
|
||||||
|
|
@ -118,12 +49,8 @@ class Session:
|
||||||
self.lock = threading.Lock()
|
self.lock = threading.Lock()
|
||||||
|
|
||||||
def add(self, name: str, data: bytes, kind: str) -> Path:
|
def add(self, name: str, data: bytes, kind: str) -> Path:
|
||||||
# One subfolder per upload keeps path.name the user's real filename, so
|
|
||||||
# error messages and the Data Quality sheet never show a "0_" prefix.
|
|
||||||
safe = Path(name).name.replace("/", "_") or "upload.xlsx"
|
safe = Path(name).name.replace("/", "_") or "upload.xlsx"
|
||||||
slot = self.dir / f"{len(self.history)}{'p' if kind == 'perf' else ''}"
|
target = self.dir / f"{len(self.history)}_{safe}"
|
||||||
slot.mkdir(parents=True, exist_ok=True)
|
|
||||||
target = slot / safe
|
|
||||||
target.write_bytes(data)
|
target.write_bytes(data)
|
||||||
if kind == "perf":
|
if kind == "perf":
|
||||||
self.perf = target
|
self.perf = target
|
||||||
|
|
@ -192,9 +119,8 @@ class Handler(BaseHTTPRequestHandler):
|
||||||
|
|
||||||
if path == "/api/state":
|
if path == "/api/state":
|
||||||
self._json({
|
self._json({
|
||||||
"mode": "local",
|
"history": [p.name.split("_", 1)[-1] for p in SESSION.history],
|
||||||
"history": [p.name for p in SESSION.history],
|
"perf": SESSION.perf.name.split("_", 1)[-1] if SESSION.perf else None,
|
||||||
"perf": SESSION.perf.name if SESSION.perf else None,
|
|
||||||
})
|
})
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
@ -202,8 +128,9 @@ class Handler(BaseHTTPRequestHandler):
|
||||||
self._export(parse_qs(route.query).get("format", ["xlsx"])[0])
|
self._export(parse_qs(route.query).get("format", ["xlsx"])[0])
|
||||||
return
|
return
|
||||||
|
|
||||||
target = _web_file(path)
|
rel = "index.html" if path in ("/", "") else path.lstrip("/")
|
||||||
if target is None:
|
target = (WEB / rel).resolve()
|
||||||
|
if not str(target).startswith(str(WEB.resolve())) or not target.is_file():
|
||||||
self._send(HTTPStatus.NOT_FOUND, b"Not found", "text/plain; charset=utf-8")
|
self._send(HTTPStatus.NOT_FOUND, b"Not found", "text/plain; charset=utf-8")
|
||||||
return
|
return
|
||||||
self._send(HTTPStatus.OK, target.read_bytes(),
|
self._send(HTTPStatus.OK, target.read_bytes(),
|
||||||
|
|
@ -213,10 +140,6 @@ class Handler(BaseHTTPRequestHandler):
|
||||||
path = urlparse(self.path).path
|
path = urlparse(self.path).path
|
||||||
try:
|
try:
|
||||||
if path == "/api/upload":
|
if path == "/api/upload":
|
||||||
ctype = self.headers.get("Content-Type", "")
|
|
||||||
if "multipart/form-data" in ctype:
|
|
||||||
name, data, kind = _parse_multipart(self._body(), ctype)
|
|
||||||
else:
|
|
||||||
name = self.headers.get("X-Filename", "upload.xlsx")
|
name = self.headers.get("X-Filename", "upload.xlsx")
|
||||||
kind = self.headers.get("X-Kind", "history")
|
kind = self.headers.get("X-Kind", "history")
|
||||||
data = self._body()
|
data = self._body()
|
||||||
|
|
|
||||||
|
|
@ -137,39 +137,6 @@ def test_row_accounting_explains_unscoreable_rows():
|
||||||
assert "UNACCOUNTED" in broken.accounting_detail
|
assert "UNACCOUNTED" in broken.accounting_detail
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------ broken exports
|
|
||||||
|
|
||||||
def test_human_readable_timestamp_is_a_valid_fallback():
|
|
||||||
"""Some extractions fill only 'Date and time', not the ISO column."""
|
|
||||||
from ppcbudget.ingest import parse_when
|
|
||||||
from datetime import datetime
|
|
||||||
assert parse_when(None, "5 Aug 2026 23:59") == datetime(2026, 8, 5, 23, 59)
|
|
||||||
assert parse_when("", "5 Aug 2026 14:17:18.440").hour == 14
|
|
||||||
assert parse_when("2026-08-05T14:17:18", "ignored").minute == 17
|
|
||||||
assert parse_when("2026-08-05T14:17:18Z") is not None
|
|
||||||
# The ISO column wins when both are present and disagree.
|
|
||||||
assert parse_when("2026-08-05T01:00:00", "5 Aug 2026 23:59").hour == 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_no_timestamp_anywhere_is_unusable_and_says_why():
|
|
||||||
"""A timeline needs a time on every row. Failing quietly would be worse."""
|
|
||||||
from ppcbudget.ingest import parse_when
|
|
||||||
assert parse_when(None, None) is None
|
|
||||||
assert parse_when("", "") is None
|
|
||||||
assert parse_when("None", "not a date") is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_exporter_warnings_are_never_ignored():
|
|
||||||
"""The reference file is clean, but the reader must expose the fields the
|
|
||||||
Data Quality panel relies on."""
|
|
||||||
qa = load()["qa"]
|
|
||||||
assert qa.warning_categories == {}
|
|
||||||
assert qa.extraction_warnings == []
|
|
||||||
assert not qa.mixed_sources
|
|
||||||
assert len(qa.source_urls) == 1, "one export should come from one console"
|
|
||||||
assert len(qa.row_marketplaces) == 1
|
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------- last meaningful action
|
# ------------------------------------------------------- last meaningful action
|
||||||
|
|
||||||
def test_amazon_pacing_rows_are_not_actions():
|
def test_amazon_pacing_rows_are_not_actions():
|
||||||
|
|
|
||||||
18
web/app.js
18
web/app.js
|
|
@ -133,16 +133,6 @@ function render() {
|
||||||
row.unt = row.act.unt;
|
row.unt = row.act.unt;
|
||||||
}
|
}
|
||||||
|
|
||||||
// A file that could not be read must never disappear quietly.
|
|
||||||
const skipped = d.skipped || [];
|
|
||||||
$('skipped').hidden = skipped.length === 0;
|
|
||||||
$('skipped-list').innerHTML = skipped.map((line) => {
|
|
||||||
const cut = line.indexOf(': ');
|
|
||||||
const name = cut > 0 ? line.slice(0, cut) : 'A file';
|
|
||||||
const why = cut > 0 ? line.slice(cut + 2) : line;
|
|
||||||
return `<div><b>${escapeHtml(name)}</b>${escapeHtml(why)}</div>`;
|
|
||||||
}).join('');
|
|
||||||
|
|
||||||
$('grain').hidden = t.days < 2;
|
$('grain').hidden = t.days < 2;
|
||||||
renderAnswer(t, m);
|
renderAnswer(t, m);
|
||||||
renderKpis(t, m);
|
renderKpis(t, m);
|
||||||
|
|
@ -823,13 +813,7 @@ $('settings').addEventListener('close', (ev) => {
|
||||||
// expired session redirects to the sign-in page rather than silently rendering
|
// expired session redirects to the sign-in page rather than silently rendering
|
||||||
// an empty dashboard.
|
// an empty dashboard.
|
||||||
A.api('/api/state').then((r) => r.json()).then((s) => {
|
A.api('/api/state').then((r) => r.json()).then((s) => {
|
||||||
if (s.mode === 'local') {
|
if (s.user) {
|
||||||
const note = $('upload-note');
|
|
||||||
if (note) {
|
|
||||||
note.textContent = 'Everything stays on this machine. Nothing is uploaded anywhere.';
|
|
||||||
}
|
|
||||||
} else if (s.user) {
|
|
||||||
$('btn-signout').hidden = false;
|
|
||||||
$('whoami').textContent = s.user.name || s.user.email;
|
$('whoami').textContent = s.user.name || s.user.email;
|
||||||
$('whoami').title = s.user.email;
|
$('whoami').title = s.user.email;
|
||||||
$('link-admin').hidden = !s.user.is_admin;
|
$('link-admin').hidden = !s.user.is_admin;
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@
|
||||||
<!-- Outside the set that stage() toggles, so these stay visible throughout. -->
|
<!-- Outside the set that stage() toggles, so these stay visible throughout. -->
|
||||||
<span id="whoami" class="muted" style="align-self:center;font-size:12px"></span>
|
<span id="whoami" class="muted" style="align-self:center;font-size:12px"></span>
|
||||||
<a id="link-admin" href="/admin" hidden><button type="button" class="ghost">Accounts</button></a>
|
<a id="link-admin" href="/admin" hidden><button type="button" class="ghost">Accounts</button></a>
|
||||||
<button id="btn-signout" class="ghost" hidden>Sign out</button>
|
<button id="btn-signout" class="ghost">Sign out</button>
|
||||||
</div>
|
</div>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
|
|
@ -44,7 +44,7 @@
|
||||||
<h2>Drop your change-history exports here</h2>
|
<h2>Drop your change-history exports here</h2>
|
||||||
<p>One file or a whole week of them. <button type="button" class="linklike" id="pick">Choose files</button>
|
<p>One file or a whole week of them. <button type="button" class="linklike" id="pick">Choose files</button>
|
||||||
— or drop a folder.</p>
|
— or drop a folder.</p>
|
||||||
<p class="fineprint" id="upload-note">Your files are uploaded to this server, analysed, and deleted
|
<p class="fineprint">Your files are uploaded to this server, analysed, and deleted
|
||||||
when you sign out or go idle. Nobody else using this dashboard can see them.</p>
|
when you sign out or go idle. Nobody else using this dashboard can see them.</p>
|
||||||
<input type="file" id="file-input" multiple accept=".xlsx,.xlsm" hidden>
|
<input type="file" id="file-input" multiple accept=".xlsx,.xlsm" hidden>
|
||||||
</div>
|
</div>
|
||||||
|
|
@ -84,11 +84,6 @@
|
||||||
|
|
||||||
<!-- --------------------------------------------------------- dashboard -->
|
<!-- --------------------------------------------------------- dashboard -->
|
||||||
<section id="stage-dash" hidden>
|
<section id="stage-dash" hidden>
|
||||||
<div class="panel skipped-panel" id="skipped" hidden>
|
|
||||||
<h2>Some files were not used</h2>
|
|
||||||
<div id="skipped-list"></div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="answer panel" id="answer">
|
<div class="answer panel" id="answer">
|
||||||
<h2>A typical campaign's day</h2>
|
<h2>A typical campaign's day</h2>
|
||||||
<p class="lede" id="answer-lede"></p>
|
<p class="lede" id="answer-lede"></p>
|
||||||
|
|
|
||||||
|
|
@ -469,17 +469,6 @@ table.grid th small { display: block; font-size: 9px; font-weight: 500; opacity:
|
||||||
.axis { display: flex; justify-content: space-between; font-size: 10px; color: var(--ink-3); margin-top: 4px; }
|
.axis { display: flex; justify-content: space-between; font-size: 10px; color: var(--ink-3); margin-top: 4px; }
|
||||||
.drawer h3 { margin: 18px 0 8px; }
|
.drawer h3 { margin: 18px 0 8px; }
|
||||||
|
|
||||||
/* ------------------------------------------------------------ skipped files */
|
|
||||||
|
|
||||||
.skipped-panel { border-left: 4px solid var(--red); }
|
|
||||||
.skipped-panel h2 { color: var(--red); }
|
|
||||||
#skipped-list { margin-top: 8px; display: grid; gap: 8px; }
|
|
||||||
#skipped-list div {
|
|
||||||
background: var(--red-soft); border-radius: 8px; padding: 10px 13px;
|
|
||||||
font-size: 12.5px; color: var(--ink); word-break: break-word;
|
|
||||||
}
|
|
||||||
#skipped-list b { display: block; margin-bottom: 3px; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------- action log (drawer) */
|
/* ------------------------------------------------------- action log (drawer) */
|
||||||
|
|
||||||
.act-none, .act-yes {
|
.act-none, .act-yes {
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue