Skip to content

Ingestion Control Plane

Architectural Source of Truth

The ingestion subsystem is a backend-owned control plane with persisted state for:

  • coverage snapshots
  • ingestion plans
  • ingestion jobs
  • job steps
  • checkpoints
  • reconciliation runs
  • structured logs

This means:

  • dashboard and CLI call the same service/runtime layer (softcopy.ingestion.service)
  • orchestration state is persisted in control-plane storage, not in frontend memory or terminal output
  • resume/cancel semantics are checkpoint-driven, not ad hoc reruns

CLI Control-Plane Commands

Use the first-class ingestion CLI group:

softcopy ingestion coverage
softcopy ingestion plan --mode catch_up_recent
softcopy ingestion run --mode backfill_range --range-start-day 2026-01-01 --range-end-day 2026-01-31
softcopy ingestion resume <job_id>
softcopy ingestion cancel <job_id>
softcopy ingestion verify --job-id <job_id>
softcopy ingestion reconcile

Each command is a thin wrapper over the same backend service methods used by dashboard ingestion APIs.

Backfill and Migration

Backfill is planned and executed through canonical plans/jobs, not separate scripts:

softcopy ingestion plan --mode backfill_range --range-start-day 2026-01-01 --range-end-day 2026-01-31
softcopy ingestion run --mode backfill_range --range-start-day 2026-01-01 --range-end-day 2026-01-31

Existing stage-level commands remain available for compatibility:

  • softcopy memory export
  • softcopy memory embed
  • softcopy memory publish
  • legacy softcopy ingest/ledger/cache flows

These are maintained as primitives, while product-facing ingestion orchestration should use the control plane (softcopy ingestion ... or /api/ingestion/...).

Runtime-Boundary Rules

  • Frontend does not own dependency or repair orchestration.
  • CLI does not implement a second runtime; it delegates to backend service/runtime.
  • Logs supplement state but do not replace persisted job/step/checkpoint contracts.
  • Verify and reconcile remain explicit control-plane operations.