Data Ingestion Framework

Land every source. Lose nothing.

Databases, Salesforce and any REST API, landed in your data lake or warehouse as typed Parquet, loaded as snapshots or increments, and recoverable by simply running again. New tables are configuration, not code.

Book a demo
0
lines of code to add a table
3
source families: databases, Salesforce, REST
2
load modes: snapshot and incremental
Any
cloud: Google Cloud, AWS, Azure or Kubernetes
Why teams choose it

Ingestion you stop worrying about

Most data platforms lose their first months to ingestion scripts that work until they don’t. This framework replaces them with one tested, configurable engine that has already run in an enterprise.

New sources in days, not sprints

Point it at a database schema or a Salesforce org and importers generate the configuration for every table. You review a diff, merge it, and the data starts landing.

Every source, one framework

Relational databases, Salesforce and REST APIs share one pipeline, one landing format and one way of operating it, so the hundredth table costs what the first did.

Snapshots and increments, CDC included

Incremental loads follow a cursor with a safety overlap and keep CDC operations and commit order intact. Full snapshots catch the deletes that leave no trace.

Nothing skipped, nothing duplicated

A watermark moves only after its data has landed, one run per source at a time. Rerun any range at any time; the downstream history simply absorbs it.

A landing zone your warehouse trusts

Typed Parquet, partitioned by date and hour, with declared schemas instead of guesses and time zones resolved on the way in. It is ready for external tables the moment it lands.

You hear about drift before users do

Scheduled snapshots of every source’s data dictionary feed your catalogue, and an alert names each table or column that appeared or disappeared, alongside failed-job alerts.

The difference

Hand-built pipelines vs the framework

What you get on day one instead of rebuilding it source by source.

Capability Hand-built pipelines(typical) Data Ingestion Framework
Adding a table New code, new tests One reviewed YAML entry
Configuration generated from live schemas
Snapshot and incremental loads, with CDC Per pipeline
Readable, resettable watermarks
Safe reruns after a failure Often manual
One run per source, guaranteed
Typed, partitioned landing format Varies
Source schema-change alerts
Dev, test and prod from one codebase Sometimes
Runs in your own cloud: GCP, AWS or Azure
Sources

Databases, Salesforce and any REST API

Three families of source share one landing format, one way of tracking progress and one set of operations.

01

Databases

Operational systems read directly, including the change logs they already keep.

  • Oracle and MySQL connectors ship today, and PostgreSQL, SQL Server and other SQLAlchemy engines use the same path
  • CDC tables read with their operation (insert, update, delete) and commit sequence preserved
  • Commit-ordered cursors, so long transactions and late commits are never skipped
  • Zero dates, unconstrained numerics and source time zones normalised on landing

Built on: dlt’s SQL source with typed column contracts.

02

Salesforce

Every object and field you need, deletes included, with no field lists kept by hand.

  • Bulk API 2.0 with queryAll, so recycled records arrive flagged as deleted
  • JWT bearer authentication for unattended runs
  • Field lists generated from your object model, with audit fields added and compound addresses expanded
  • A preflight that tells fields hidden by field-level security from fields missing in the org

Built on: A Bulk API 2.0 client with an incremental SystemModstamp cursor and overlap.

03

REST APIs

Any HTTP API, from SaaS platforms to internal services, landed the same way.

  • Declarative endpoint configuration: authentication, pagination and incremental parameters
  • Nested JSON flattened into typed tables, with child tables where the data is nested
  • The same watermarks, retries, leases and landing layout as every other source
  • Corporate HTTPS proxies supported end to end

Built on: dlt’s declarative REST API source.

Under the hood

From source to landing zone, in four stages

The watermark is the most important number in an ingestion system, because a wrong one silently skips data. Every stage is designed to keep it right.

  1. 01

    Extract

    cursor > watermark − overlap

    Each table reads only what changed since its watermark, or the whole object when it is configured as a snapshot.

    • Parallel workers per source, in threads or processes
    • Separate backfill jobs with more CPU, memory and runtime for the first load
    • A lease keeps two runs of one source from ever overlapping
  2. 02

    Land

    <bucket>/<table>/dt=/hr=/*.parquet

    Typed Parquet in a hive-partitioned layout on Cloud Storage, Amazon S3 or Azure Blob / ADLS, one bucket per source system, with bookkeeping kept out of the table folders.

    • Buckets created on demand, private by default with public access blocked
    • Column types use the warehouse’s own names, so external tables read them unchanged
    • Snapshots stamped with the instant they were taken
  3. 03

    Commit

    watermark.json

    The new watermark is written only after the data has landed, and only if nobody else changed it in the meantime.

    • Plain JSON on object storage that a person can read and fix under pressure
    • Optimistic concurrency on the object version, so two writers can never both win
    • A reset command per table that keeps the last twenty values
  4. 04

    Orchestrate & alert

    scheduled, then the transforms

    A workflow runs the ingestion jobs in parallel and then the transformation jobs in order, every few hours.

    • An overlap guard so the scheduled and initial pipelines never run at once
    • A dry run that shows exactly what an execution would start
    • Email alerts for failed jobs, failed workflows, a stale catalogue and changed source schemas
Tech stack

Open source at the core, at home on any cloud

Proven open-source Python, packaged as containers and run on the managed services of the cloud you already use.

Sources

Oracle, MySQL and Salesforce connectors ship today; the rest use the same SQL and REST paths.

  • Oracle
  • MySQL
  • MariaDB
  • PostgreSQL
  • SQL Server
  • Salesforce
  • REST APIs

Built on

  • Python
  • dlt
  • SQLAlchemy
  • Apache Parquet
  • Pydantic
  • FastAPI
  • Docker

Any cloud

Containerised jobs on each cloud’s managed services or on Kubernetes, landing on Cloud Storage, Amazon S3 or Azure Data Lake Storage.

  • Google Cloud
  • AWS
  • Microsoft Azure
  • Kubernetes

Operated with

  • Cloud Workflows
  • AWS Step Functions
  • Azure Data Factory
  • Cloud Monitoring
  • Amazon CloudWatch
  • GitLab CI
  • GitHub Actions
How it works

From source inventory to a running pipeline

A fixed onboarding path into your own cloud, not an open-ended project.

  1. 01

    Map your sources

    We inventory the systems, their change indicators and their delete behaviour, and agree which tables load incrementally and which as snapshots.

  2. 02

    Generate the configuration

    Importers read your live schemas and field lists into versioned YAML. You review the diff and nothing is written by hand.

  3. 03

    Deploy in your cloud

    Jobs, buckets, schedules and alerts go into your Google Cloud, AWS or Azure account for dev, test and prod, with secrets kept in your own secret store.

  4. 04

    Backfill, then run

    An initial pipeline loads history once. From then on the schedule keeps every table current and alerts on anything unusual.

FAQ

Questions data teams ask us

Which sources does the framework support?

Relational databases (Oracle and MySQL connectors today, and other SQLAlchemy-supported engines such as PostgreSQL and SQL Server through the same path), Salesforce through Bulk API 2.0, and REST APIs through a declarative configuration. Every source shares the same landing format, watermarks and operations.

How are deletes handled?

CDC sources carry the delete as an operation. Salesforce returns recycled records flagged as deleted. Objects that are hard-deleted without a trace are loaded as full snapshots, and a key missing from the next snapshot closes its history downstream.

What happens when a run fails halfway?

Nothing is lost. A table that fails keeps its watermark, so the next run reads the same range again. Because the history layer merges on a surrogate key, a row landed twice simply updates itself.

Where does it run, and where does the data go?

In your own cloud account, on Google Cloud, AWS or Azure, or on Kubernetes. The jobs are containers that run on each cloud’s managed services and land on its object storage (Cloud Storage, Amazon S3 or Azure Blob / ADLS), and your data never passes through Duces.

Does it model the data?

No, by design. The ingestion framework moves rows and derives no history. History and the business layers belong to the Data Processing & Transformation Framework, which reads what this one lands and whose models are generated from the same configuration.

Product names and logos shown are trademarks of their respective owners. Salesforce is a trademark of Salesforce, Inc., and Oracle and MySQL are trademarks of Oracle and/or its affiliates. The Data Ingestion Framework is an independent product by Duces and is not affiliated with, sponsored or endorsed by them.

Contact Duces

Book a demo

Tell us which systems you need to land and where your warehouse lives, and we’ll show you the framework running on sources like yours.