Skip to main content

Catalog onboarding

What it does​

Per ADR-0010 D5, D7, D8, D12, Ratiba lets tenants self-onboard their existing service catalog without re-typing it row by row. The pipeline accepts five input modalities — image, multi_image, pdf, text, csv — and produces the same ExtractedRow envelope regardless of source, so the review-screen UI handles a single shape.

Vision OCR and modality dispatch​

Image, multi-image, and PDF modalities call Anthropic Vision (see Glossary — Anthropic Vision OCR) through the M11 T3 LLMRouter vision role. Multi-image uploads run one Vision call per image then merge near-duplicate rows by name similarity (Levenshtein distance < 3 OR difflib SequenceMatcher ratio >= 0.9), with per-field confidence merged via max so the more confident extraction wins each cell. PDFs are rendered page-by-page via pypdf + Pillow and treated as multi-image. The text modality skips vision and uses the text-LLM answer_shaper role; csv bypasses the LLM entirely via a stdlib parser.

After extraction, every row passes through a gap-fill pass that asks the LLM to suggest typical values for missing duration_min / name_sw / description with confidence < 1.0.

Price safety floor​

Safety floor (D7): price is NEVER auto-filled. Rows where extraction left price=None stay None; the gap-fill step explicitly discards any LLM-supplied price; the review screen marks those rows red and the admin must type the price manually before commit. The same rule is enforced again at the persistence boundary — a draft with a None price and no admin override raises CatalogImporterError("price_floor_violation") before any DB write.

This is a hard, two-layer guard: client enforcement (Submit disabled) + server enforcement (HTTP 422). Neither layer alone is sufficient because an authenticated API client could bypass the dashboard. Both must be in place.

Why no auto-fill? East African service pricing varies sharply by location, salon tier, and promotions the LLM cannot know. An incorrect auto-filled price could be published to customers before the owner notices. The review screen friction is intentional.

LLM-inferred relation graph​

The relation inferrer (M11 T5) is invoked once per import and emits complementary / alternative / sequential rows for the tenant's service_relations graph at temperature=0 for re-run idempotency. The inferrer reasons over service names, not IDs — this avoids hallucinating numeric foreign keys. Relations are stored in <tenant>.service_relations and consumed directly by the cross-sell engine; the cross-sell page owns the full runtime flow: see Cross-sell.

Inference is declared non-load-bearing by ADR-0010 D5: if the inferrer call fails or returns empty, the import still commits. The operator sees zero cross-sell candidates until the next re-import succeeds.

The three relation types and their semantics:

TypeMeaningCross-sell action
complementaryServices that pair well (e.g. manicure + pedicure)First-candidate for a slot-aware cross-sell offer
alternativeSubstitutable services (e.g. deep-tissue vs Swedish massage)Offer when primary service is unavailable
sequentialServices with a natural ordering (e.g. consultation → treatment)Reserved for future use; stored now, unused at runtime

Atomic commit with advisory lock​

CatalogImporter.import_catalog() returns an ImportDraft and writes nothing — extraction + relation inference + a snapshot read of the current services state happen first, the dashboard renders the review screen, and only commit_import() opens the transaction. Commit holds a per-tenant pg_advisory_xact_lock for the duration, atomically: upserts each services row by LOWER(name) match, records one catalog_audits row per mutation (insert / update), inserts the service_relations rows, and lands one catalog_imports row carrying snapshot + extracted + committed JSONB payloads.

Why a Postgres advisory lock? Without it, two concurrent browser tabs on the same admin session could submit two overlapping drafts. The lock is xact-scoped (auto-released on commit/rollback) so no manual cleanup is needed even on error.

Idempotent re-import​

Idempotency (D8): a re-import matches existing rows by LOWER(name) and updates rather than duplicates. The catalog_imports.type column distinguishes 'initial' (snapshot was empty) from 're-import'. All LOWER(name) matches are case-insensitive — "Swedish Massage" and "swedish massage" are the same service. New rows in the re-import that have no LOWER(name) match in the existing catalog are inserted fresh.

Single-snapshot rollback​

Rollback (D12): the snapshot persisted into catalog_imports.snapshot_jsonb powers the 7-day single-snapshot rollback UI. The snapshot captures the full services state at the moment import_catalog() reads it — before commit_import() touches any row. If the operator discovers a bad import (wrong prices, stale services), they open the Imports tab, locate the most recent import row, and click Rollback. The backend re-applies the snapshot, replacing the live services rows with the saved state. Only the most recent snapshot is actionable; older imports can be viewed for audit but cannot be rolled back to.

For programmatic seeding (e.g. development or bulk pilot setup), see Seed test data.

How it fits in the system​

How it flows​

Initial import (both audiences)​

On a fresh tenant the snapshot is empty, so type='initial'. Every extracted row that passes review is inserted, not updated. The admin types prices for any red rows before Submit is enabled.

Re-import and merge logic​

On re-upload, the snapshot contains the live catalog. The merge walk:

  1. For each extracted row, compute LOWER(name).
  2. Scan snapshot for a match.
  3. If match found → UPDATE the existing row; audit row gets action='update' with before and after JSONB.
  4. If no match → INSERT the new row; audit row gets action='insert'.
  5. Services in the snapshot that have no corresponding extracted row are left untouched (the import is additive by design; deletions require a manual /toggle-service command via the admin rail — see Admin orchestrator — slash commands).

Multi-image deduplication​

When a menu spans multiple images (e.g. a spa menu split across two photos), the extractor runs one Vision call per image and then merges rows that represent the same service. The deduplication pass:

  1. Normalise names: lowercase + strip punctuation.
  2. Pair rows across images using Levenshtein distance < 3 OR SequenceMatcher ratio >= 0.9.
  3. For each matched pair, keep the row whose extraction confidence is higher. On field-by-field conflict, take max confidence per field independently.

This means a service visible in image 1 with a clear price but a blurry name, and visible in image 2 with a clear name but no price, will produce a merged row with the best confidence for each field.

Where it lives in code​

ConcernFileKey entry point
Pipeline orchestratorbackend/app/services/catalog_importer.pyCatalogImporter.import_catalog() / commit_import()
Vision LLM extractionbackend/app/services/vision_extractor.pyAnthropicVisionExtractor.extract()
Test stubbackend/app/services/vision_extractor_fakes.pyFakeVisionExtractor
Relation inferencebackend/app/services/relation_inferrer.pyLLMRelationInferrer.infer()
Import DAO — snapshot + upsertbackend/app/persistence/catalog_imports.pyfetch_services_snapshot() / upsert_service()
Import DAO — audit + import rowbackend/app/persistence/catalog_imports.pyinsert_catalog_import() / insert_catalog_audit()

Decisions​

  • ADR-0010 — Tenant self-service customisation:
    • D5 locks the LLM-inferred relation graph as the relation source (vs hand-authored or per-booking inference) and pins relation inference as non-load-bearing (failures degrade to "no relations"). Downstream cross-sell runtime is owned by Cross-sell.
    • D7 locks Anthropic Vision as the extraction LLM and the price safety floor: the LLM is never permitted to populate price; missing prices stay None and the review screen forces admin manual entry.
    • D8 locks idempotent re-imports keyed off LOWER(name) and the catalog_imports.type audit signal ('initial' vs 're-import').
    • D12 locks the single-snapshot 7-day rollback semantics — every catalog_imports row carries the pre-commit services state in snapshot_jsonb.

Try this on local dev​

  1. Drop a sample menu. Open /admin/catalog (the M11 T11 onboarding wizard surface) and drag in tests/fixtures/sample_menu.png. Watch the server logs for vision_extractor.extracted followed by relation_inferrer.short_circuit_small_catalog (or the inferrer call if your fixture has >=2 services). The review screen should render the extracted rows with per-cell confidence colour-coding.

  2. Force the safety floor. Confirm any row with confidence(price) < 0.5 is rendered red and the Submit button is disabled until you type a price. Try clicking Submit anyway via the API directly — the server returns CatalogImporterError(reason="price_floor_violation") with HTTP 422 (per ADR-0010 D7, the rule is enforced both client- and server-side).

  3. Verify the round-trip. Click Submit; expect a green toast with the CommitResult summary. In psql, run SELECT name, price FROM <tenant>.services to see the inserted rows, SELECT relation_type, count(*) FROM <tenant>.service_relations GROUP BY 1 to see the inferred graph, and SELECT type, modality, committed_at FROM <tenant>.catalog_imports ORDER BY committed_at DESC LIMIT 1 to confirm the audit row landed with type='initial'. Re-upload the same image to verify the second commit lands type='re-import' and every catalog_audits row records action 'update'.

  4. Test multi-image deduplication. Drag in two overlapping crop images of the same menu (create them from sample_menu.png with an image editor). Confirm the review screen shows the expected merged row count — the extractor log line vision_extractor.merge_dedup shows how many duplicate pairs were collapsed.

  5. Trigger the advisory lock. Open two browser tabs to /admin/catalog, submit the same draft from both tabs simultaneously. The second request should block briefly then complete (not corrupt) — both commits cannot overlap. Check catalog_imports has exactly one row per submit, not two partial rows.