Ferryman

Ferryman copies tables between databases on a schedule. You describe a pipeline in a web UI: one table from a source connection to a destination connection, how to read it, how to write it, and when to run. Ferryman runs it, keeps the destination in step, and shows you every run, its counts and its errors.

• Three ways to read a table: a full copy, an incremental read past a cursor column, or its changes: change data capture (CDC) from the Postgres write-ahead log or a Delta table's change data feed, or SQL Server's change tracking.
• Safe to retry. Delivery is at-least-once, and every write is idempotent on the table's primary key, so a retry never produces duplicate rows.
• No broker. The job queue lives in the control-plane Postgres database. There is no Celery, Redis or message broker to run.
• Watched. Runs update live in the UI, failures and missed SLAs go to Slack or any webhook, and every process exports Prometheus metrics.
• One organization, with access control. Permissions through built-in and custom roles, an audit log, and API keys for automation. Connection passwords and webhook URLs are stored encrypted (Fernet) and never returned by the API.

How it works

 browser ──► web (React SPA) ──► api (FastAPI) ──┐
 scripts, CI ── API key ────────►                ├──► control-plane Postgres
              scheduler ── queues due runs ──────┤     (tenants, pipelines, runs,
              worker(s) ── claim and execute ────┘      encrypted secrets, the queue)
                  │
                  ├── reads from a source database
                  └── writes to a destination database

Component	What it does	Entry point
web/	The UI: connections, the pipeline wizard, pipeline and run monitoring, alerts, users, roles, API keys and the audit log. React 19, TypeScript, Vite.	npm run dev
engine/ API	Everything the UI does, over HTTP. FastAPI.	ferryman-api
engine/ scheduler	Turns schedules and retries into queued runs, checks SLAs, sends alerts, purges old history.	ferryman-scheduler
engine/ worker	Claims queued runs and executes them: extract, stage, then merge into the destination.	ferryman-worker
Control-plane Postgres	Ferryman's own state and its job queue.

A run goes QUEUED → CLAIMED → RUNNING → STAGED → MERGING → SUCCEEDED. Rows are first written to a staging table; the destination table's data changes only in the final merge (its structure is reconciled at the start of the run). A failed run is retried, by default up to 3 times, after 1, 2 and then 4 minutes (each pipeline can set its own); a misconfiguration ends as CONFIG_ERROR and is never retried.

Schedules are FIXED_INTERVAL (every N), SLA_DRIVEN (keep the data no staler than a target) or MANUAL. There are no cron expressions.