Restaurant Menu Document Import Implementation Plan

For Codex: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Import restaurant menus from PDF and raster images into a reviewable proposal, preserving page-level provenance and preventing unreviewed OCR output from changing a live menu.

Architecture: The API issues tenant-scoped upload URLs, validates and scans the uploaded object, and runs an idempotent BullMQ pipeline for OCR and menu structuring. Provider output is normalized into MenuImportProposalV1; an owner or admin reviews field-level confidence and accepts selected changes in one transaction. Source files and intermediate output use explicit retention and never enter the public snapshot.

Tech Stack: NestJS, TypeORM/PostgreSQL with FORCE RLS, BullMQ/Redis, S3-compatible object storage, LiteLLM-compatible vision adapter, Next.js, Jest, Playwright.

---

Global Constraints

• Run python3 .agent/tools/recall.py "restaurant menu document import OCR migration failing test" before implementation and surface the result.
• Obtain approval before installing file-type, pdfjs-dist, or sharp, and before applying the migration.
• Accept only PDF, PNG, JPEG, and WebP; verify magic bytes rather than trusting extensions or request headers.
• Default limits: 25 MiB per upload, 40 PDF pages, 30 megapixels per image, and 120 source files per tenant per day.
• Never infer allergens, ingredients, dietary claims, or prices that are absent from the source.
• OCR and structuring output stays in needs_review; only an explicit accepted-field request may update a draft.
• Store originals for 30 days after terminal state, normalized OCR for 90 days, and audit metadata according to the application audit policy.
• Never commit source menu documents or real customer OCR output as fixtures.

Produced and Consumed Interfaces

export type MenuImportJobState =
  | 'uploaded' | 'validating' | 'ocr' | 'structuring'
  | 'needs_review' | 'accepted' | 'rejected' | 'failed' | 'expired';

export interface ProposedField<T> {
  value: T | null;
  confidence: number; // inclusive 0..1
  source: { page: number; bounds?: [number, number, number, number] };
}

export interface MenuImportProposalV1 {
  schemaVersion: 1;
  sections: Array<{
    name: ProposedField<string>;
    items: Array<{
      clientKey: string;
      name: ProposedField<string>;
      description: ProposedField<string>;
      priceMinor: ProposedField<number>;
      currency: ProposedField<string>;
    }>;
  }>;
  warnings: Array<{ code: string; message: string; page?: number }>;
}

export interface OcrProvider {
  recognize(input: { objectKey: string; mimeType: string; pageCount: number }): Promise<OcrDocumentV1>;
}

export interface DocumentScanner {
  scan(input: { objectKey: string; sha256: string }): Promise<{ clean: boolean; engine: string }>;
}

The import service consumes ObjectStore from the core publishing plan and produces accepted edits for MenuDraftService. All tenant-facing endpoints consume the existing authenticated tenant context.

Task 1: Persist the Import State Machine

Files:

• Create: apps/api/src/menu/import/entities/menu-import-job.entity.ts

• Create: apps/api/src/menu/import/entities/menu-import-artifact.entity.ts

• Create: apps/api/src/menu/import/menu-import.types.ts

• Create: apps/api/src/db/migrations/1723500000000-MenuImports.ts

• Modify: apps/api/src/db/data-source.ts

• Modify: apps/api/src/menu/menu.module.ts

• Test: apps/api/src/menu/import/menu-import.entity.spec.ts

• [ ] Step 1: Write failing entity and transition tests

Cover valid state transitions, terminal immutability, (tenant_id, idempotency_key) uniqueness, proposal schema version, and rejection of confidence outside 0..1.

• [ ] Step 2: Run the focused test and confirm failure

bun test apps/api/src/menu/import/menu-import.entity.spec.ts

Expected: FAIL because import entities and transition guards do not exist.

• [ ] Step 3: Implement the entities and migration

Create tenant-scoped menu_import_jobs and menu_import_artifacts; include state, source_sha256, idempotency_key, proposal jsonb, failure_code, timestamps, and object keys. Enable and force RLS using the repository tenant-setting expression and add tenant indexes plus terminal-state checks.

• [ ] Step 4: Register entities in runtime and CLI data sources

• [ ] Step 5: After migration approval, run tests and a migration round trip

bun test apps/api/src/menu/import/menu-import.entity.spec.ts
bun run test:db:up
DATABASE_URL=postgres://app:app@localhost:54332/checklist_test bun run --cwd apps/api migration:run
DATABASE_URL=postgres://app:app@localhost:54332/checklist_test bun run --cwd apps/api migration:revert
DATABASE_URL=postgres://app:app@localhost:54332/checklist_test bun run --cwd apps/api migration:run
bun run test:db:down

• [ ] Step 6: Commit

git add apps/api/src/menu/import apps/api/src/db/data-source.ts apps/api/src/db/migrations/1723500000000-MenuImports.ts apps/api/src/menu/menu.module.ts
git commit -m "feat(menu): persist document import jobs"

Task 2: Secure Source Upload and Validation

Files:

• Create: apps/api/src/menu/import/menu-import.controller.ts

• Create: apps/api/src/menu/import/menu-import.service.ts

• Create: apps/api/src/menu/import/document-validator.service.ts

• Create: apps/api/src/menu/import/document-scanner.adapter.ts

• Modify: apps/api/src/config/env.schema.ts

• Test: apps/api/src/menu/import/menu-import-upload.spec.ts

• [ ] Step 1: Write failing upload contract tests

Test POST /menus/:menuId/imports, presigned upload completion, magic-byte mismatch, encrypted/malformed PDF, page/pixel/size limits, dirty scan, cross-tenant access, quota exhaustion, and idempotent retry.

• [ ] Step 2: Run the test and confirm failure

bun test apps/api/src/menu/import/menu-import-upload.spec.ts

• [ ] Step 3: Implement create and complete-upload endpoints

Use the routes below and return RFC 7807 problem details with stable codes.

POST /menus/:menuId/imports
POST /menus/:menuId/imports/:importId/upload-complete
GET  /menus/:menuId/imports/:importId
DELETE /menus/:menuId/imports/:importId

Validate object size and SHA-256 after upload, inspect magic bytes, extract PDF page count or image dimensions in a bounded worker, scan through DocumentScanner, then enqueue OCR. Move rejected objects to the quarantine prefix and deny reads outside service credentials.

• [ ] Step 4: Add scanner configuration

Require MENU_SCAN_ENDPOINT, MENU_SCAN_TOKEN, and a five-second timeout outside test. Tests use a deterministic in-process fake; production fails closed when the scanner is unavailable.

• [ ] Step 5: Run focused API tests

bun test apps/api/src/menu/import/menu-import-upload.spec.ts

• [ ] Step 6: Commit

git add apps/api/src/menu/import apps/api/src/config/env.schema.ts
git commit -m "feat(menu): validate document imports"

Task 3: Normalize OCR Behind a Provider Adapter

Files:

• Create: apps/api/src/menu/import/ocr/ocr-provider.ts

• Create: apps/api/src/menu/import/ocr/litellm-vision-ocr.adapter.ts

• Create: apps/api/src/menu/import/ocr/fake-ocr.adapter.ts

• Create: apps/api/src/menu/import/workers/menu-ocr.processor.ts

• Test: apps/api/src/menu/import/ocr/ocr-provider.contract.spec.ts

• Test: apps/api/src/menu/import/workers/menu-ocr.processor.spec.ts

• Test: apps/api/src/menu/import/fixtures/two-page-menu.expected.json

• [ ] Step 1: Write a provider contract suite

Use generated fixture pages containing RU, EN, SR Latin, decimal commas, multiple currencies, and rotated text. Assert page order, text blocks, bounds normalization, deterministic retry behavior, and provider timeout mapping.

• [ ] Step 2: Run and confirm failure

bun test apps/api/src/menu/import/ocr/ocr-provider.contract.spec.ts

• [ ] Step 3: Implement fake and LiteLLM-compatible adapters

Send bounded page images, require JSON-schema output, strip provider reasoning, and normalize coordinates to 0..1. Persist provider/model/request ID and usage metadata without logging source content.

• [ ] Step 4: Implement the idempotent OCR processor

Lock by import ID, resume completed pages, cap retries at three with exponential backoff, and map timeout, invalid output, policy rejection, and quota errors to distinct failure codes.

• [ ] Step 5: Run contract and processor tests

bun test apps/api/src/menu/import/ocr/ocr-provider.contract.spec.ts apps/api/src/menu/import/workers/menu-ocr.processor.spec.ts

• [ ] Step 6: Commit

git add apps/api/src/menu/import/ocr apps/api/src/menu/import/workers apps/api/src/menu/import/fixtures
git commit -m "feat(menu): add OCR import pipeline"

Task 4: Structure OCR Into a Review Proposal

Files:

• Create: apps/api/src/menu/import/menu-proposal.schema.ts

• Create: apps/api/src/menu/import/menu-structurer.service.ts

• Create: apps/api/src/menu/import/workers/menu-structure.processor.ts

• Test: apps/api/src/menu/import/menu-structurer.spec.ts

• Test: apps/api/src/menu/import/fixtures/ambiguous-prices.expected.json

• [ ] Step 1: Write failing golden tests

Assert section/item order, minor-unit conversion, provenance on every proposed field, warnings for ambiguous price/currency, no invented dietary data, and identical output for repeated input.

• [ ] Step 2: Run and confirm failure

bun test apps/api/src/menu/import/menu-structurer.spec.ts

• [ ] Step 3: Implement schema validation and deterministic normalization

Reject unknown keys, require integer priceMinor, clamp no values, and emit a warning instead of guessing. Set needs_review only after the entire proposal validates.

• [ ] Step 4: Run golden tests

bun test apps/api/src/menu/import/menu-structurer.spec.ts

• [ ] Step 5: Commit

git add apps/api/src/menu/import/menu-proposal.schema.ts apps/api/src/menu/import/menu-structurer.service.ts apps/api/src/menu/import/workers/menu-structure.processor.ts apps/api/src/menu/import/fixtures
git commit -m "feat(menu): structure OCR proposals"

Task 5: Review and Transactionally Accept Selected Fields

Files:

• Create: apps/api/src/menu/import/menu-import-accept.dto.ts

• Create: apps/api/src/menu/import/menu-import-merge.service.ts

• Modify: apps/api/src/menu/import/menu-import.controller.ts

• Test: apps/api/src/menu/import/menu-import-merge.spec.ts

• [ ] Step 1: Write failing merge tests

Cover selected-field acceptance, rejected fields, stale draft revision, duplicate submit, partial validation failure, deleted target menu, role denial, and rollback of the entire transaction.

• [ ] Step 2: Run and confirm failure

bun test apps/api/src/menu/import/menu-import-merge.spec.ts

• [ ] Step 3: Implement acceptance

POST /menus/:menuId/imports/:importId/accept accepts { draftRevision, acceptedFieldPaths, overrides }. Validate each JSON pointer against the stored proposal, apply changes through MenuDraftService, append an audit record, and transition to accepted in the same tenant transaction.

• [ ] Step 4: Run merge and authorization tests

bun test apps/api/src/menu/import/menu-import-merge.spec.ts

• [ ] Step 5: Commit

git add apps/api/src/menu/import
git commit -m "feat(menu): review and accept imports"

Task 6: Build the Import Review UI

Files:

• Create: apps/web/app/[locale]/app/menus/[menuId]/imports/new/page.tsx

• Create: apps/web/app/[locale]/app/menus/[menuId]/imports/[importId]/page.tsx

• Create: apps/web/ui/app/menus/imports/ImportUploader.tsx

• Create: apps/web/ui/app/menus/imports/ProposalReview.tsx

• Create: apps/web/ui/app/menus/imports/SourcePreview.tsx

• Create: apps/web/ui/app/menus/imports/menuImportsClient.ts

• Test: apps/web/ui/app/menus/imports/ProposalReview.test.tsx

• Test: apps/web/e2e/menu-import.spec.ts

• Modify: apps/web/messages/ru.json

• Modify: apps/web/messages/en.json

• Modify: apps/web/messages/sr.json

• [ ] Step 1: Write failing component and journey tests

Cover upload progress, validation failure, processing refresh, low-confidence emphasis, page-source navigation, field override, selective acceptance, keyboard operation, and RU/EN/SR strings.

• [ ] Step 2: Run and confirm failure

bun test --cwd apps/web ui/app/menus/imports/ProposalReview.test.tsx

• [ ] Step 3: Implement upload and review pages

Poll only non-terminal jobs with capped backoff, show exact warnings and provenance, select no low-confidence fields by default, and require confirmation before acceptance.

• [ ] Step 4: Run UI tests

bun test --cwd apps/web ui/app/menus/imports/ProposalReview.test.tsx
bun apps/web/node_modules/playwright/cli.js test apps/web/e2e/menu-import.spec.ts

• [ ] Step 5: Commit

git add apps/web/app apps/web/ui/app/menus/imports apps/web/e2e/menu-import.spec.ts apps/web/messages
git commit -m "feat(menu): add import review UI"

Task 7: Prove Retention, Resilience, and Operations

Files:

• Create: apps/api/src/menu/import/workers/menu-import-retention.processor.ts

• Create: test/menu-import.e2e.test.ts

• Create: test/menu-import-resilience.e2e.test.ts

• Create: docs-internal/menu-document-import-operations.md

• [ ] Step 1: Write failing retention and resilience tests

Prove object expiry, metadata preservation, provider timeout/retry, worker restart/resume, duplicate queue delivery, poisoned output, tenant isolation, and source-content redaction.

• [ ] Step 2: Implement the daily retention worker and metrics

Delete eligible objects through ObjectStore, record completion without deleting audit facts, and emit counts for queue age, stage latency, failures by code, and retained bytes.

• [ ] Step 3: Write the runbook

Document limits, scanner/OCR configuration, replay rules, quarantine access, safe job cancellation, retention verification, provider incident response, and tenant support workflow.

• [ ] Step 4: Run the acceptance suite

bun test apps/api/src/menu/import
bun apps/web/node_modules/playwright/cli.js test apps/web/e2e/menu-import.spec.ts
bun test test/menu-import.e2e.test.ts test/menu-import-resilience.e2e.test.ts

• [ ] Step 5: Commit

git add apps/api/src/menu/import/workers/menu-import-retention.processor.ts test/menu-import.e2e.test.ts test/menu-import-resilience.e2e.test.ts docs-internal/menu-document-import-operations.md
git commit -m "test(menu): prove document import lifecycle"

Workstream Acceptance Checklist

• [ ] Only validated, scanned PDF/PNG/JPEG/WebP objects enter OCR.
• [ ] Size, page, pixel, tenant quota, and timeout bounds pass at boundaries.
• [ ] Every proposed field has confidence and page provenance.
• [ ] Ambiguous or absent facts produce warnings, never invented values.
• [ ] OCR output cannot modify a draft without explicit selected-field acceptance.
• [ ] Acceptance is tenant-scoped, revision-checked, transactional, idempotent, and audited.
• [ ] Originals and intermediate artifacts expire on the documented schedule.
• [ ] RU/EN/SR review UI, accessibility, resilience, and runbook checks pass.