dbt at enterprise scale: project structure, CI/CD and testing

A dbt project holds up at enterprise scale when four things are settled early: a layered structure with a named owner for every domain, enforced contracts and versions on the models other teams consume, a test strategy that combines unit tests, data tests and source freshness, and a slim CI pipeline that builds only what changed and defers the rest to production. Most organizations should start with one project and move to a multi-project dbt Mesh only when team boundaries, release cadence or access rules require it. Cost control is then a per-warehouse job: incremental models where they pay off, compute sized per folder, and clustering or partitioning on the few tables large enough to need it.

On names: since dbt v2 went GA on September 16, 2026, the Rust engine once called Fusion is “dbt”, its Apache 2.0 subset is dbt OSS, the Python dbt Core continues as dbt v1, and dbt Cloud is the dbt platform. The practices below apply to v1 and v2; where something needs a paid plan, we say so.

One dbt project or several: when does dbt Mesh make sense?

Splitting costs more than teams expect: duplicated macros, lineage across projects, version drift. A single project with domain folders, groups and access modifiers takes most organizations a long way. dbt Labs’ Mesh guidance lists the signals for splitting: model count slowing development, teams with separate workflows, data products losing reliability, and security requirements that call for isolation. We would add release cadence: if finance needs a frozen month-end and marketing ships daily, one repository keeps one of them waiting.

Situation	Setup we recommend	What to watch
One platform team, one release cadence	Single project, folders per domain, groups with owners	Parse and CI time as the model count grows
Several domain teams sharing one warehouse and cadence	Single project with access modifiers and CODEOWNERS per domain folder	Teams referencing each other’s intermediate models
Domains with their own cadence, access rules or budget	Multiple projects (dbt Mesh), with public, contracted models as the only interface	Duplicated macros, lineage across projects, upstream version deprecations

Which Mesh features are open source and which need the dbt platform?

The governance features have been in open-source dbt since v1.5: groups, access modifiers, model contracts and model versions. The native cross-project ref (a project declared in dependencies.yml, then {{ ref('finance', 'fct_invoices') }}) relies on a dbt metadata service and is available on dbt Enterprise or Enterprise+ plans.

Self-hosted teams can install the upstream project as a package, which pulls its full source code in, so you parse its models and can run them by accident. Or they can use dbt-loom, a community plugin for dbt Core that reads public models from upstream manifests (files, object storage, Snowflake stages, Databricks volumes) and injects them. dbt-loom hooks into dbt Core’s Python plugin system, so confirm v2 support before you plan an engine upgrade around it.

How should an enterprise dbt project be structured?

Staging, intermediate and marts

The dbt Labs structure guide defines three layers that hold up at scale if you enforce the rules between them:

models/
├── staging/
│   ├── erp/            stg_erp__customers.sql, stg_erp__invoices.sql
│   └── crm/            stg_crm__accounts.sql
├── intermediate/
│   └── finance/        int_invoices_joined_to_payments.sql
└── marts/
    ├── finance/        fct_invoices.sql, dim_customers.sql
    └── sales/          fct_opportunities.sql

• Staging: one model per source table. Rename, cast, standardize time zones and units. No joins and no business logic. Materialize as views.
• Intermediate: purpose-built steps that prepare staging models for marts. Never exposed to BI or to other teams, so mark them private or protected.
• Marts: business entities and facts at a stated grain. This is the only layer that BI tools and other projects read, and the only one that gets access: public.

Each layer reads only from the one before it. One mart reading straight from a source “just this once” makes lineage unreliable, so check depends_on in the manifest during CI.

Audit-heavy environments usually need more: an external layer over the landing zone, full history before anything is conformed, and a canonical model that restates every source in one vocabulary. Per-table snapshots work for a handful of sources and get hard to manage at hundreds. Our Data Processing & Transformation Framework uses this layout: five layers, from external tables through SCD2 history and a canonical model to current views and marts.

Naming and ownership

Pick one naming convention and lint it, for example stg_<source>__<entity>, int_<entity>_<verb>, and fct_ / dim_ or business names in marts. Override generate_schema_name so production gets clean schema names per layer or domain and developers get dev_<user> schemas, with no environment logic inside model files.

Make ownership explicit in three places: every model belongs to a group with a named owner, CODEOWNERS mirrors the groups so a pull request touching marts/finance routes to the finance data owner, and meta carries owner and criticality so alerts and the catalog know whom to tell.

When are model contracts and versions worth it?

A model contract (contract: {enforced: true}) declares every column’s name and data type, and the build fails if the SQL produces a different shape. Contracts work on tables, views (without constraints) and incremental models with on_schema_change set to append_new_columns or fail, but not on Python, ephemeral or materialized-view models. Enforced constraints vary by platform, so treat primary_key and foreign_key as documentation unless your warehouse enforces them.

Our rule: contract every public model and anything a BI tool or application reads, and nothing in staging, where contracts slow refactoring without protecting anyone.

Model versions are for breaking changes to a contracted model. You publish v2 next to v1, consumers pin ref('fct_invoices', v=1) until they migrate, and you set a deprecation_date on the old version. The usual failure is never retiring old versions, so several copies of a large fact table get rebuilt nightly. Keep at most one previous version live, with a date.

What does a good dbt testing strategy look like?

Test type	What it catches	Where it runs
Unit tests (v1.8+)	Wrong logic: CASE boundaries, window functions, date arithmetic, incremental filters, checked on static input rows	Development and CI only
Generic data tests (unique, not_null, accepted_values, relationships)	Broken keys, unexpected values, orphaned rows in real data	Every build
Singular tests	Business invariants written as SQL that returns failing rows (one open row per key, debits equal credits)	Every build
Source freshness	Upstream loads that silently stopped	Before the build
Contracts	Shape changes in public models	At build time

Every model gets a primary-key test (unique plus not_null). Choose severity per test: error stops the run before anything is served, warn gets reported. Turn on store_failures where people investigate failures. Scope relationship tests on large facts to recent partitions with a where config. dbt Labs recommends running unit tests only in development and CI; exclude them in production with --exclude-resource-type unit_test.

How does slim CI work in dbt?

Slim CI builds only the models a pull request changed plus their downstream dependents, and resolves every other reference to production. It relies on state selection and deferral, using the manifest.json from the last production run:

# On every pull request: fetch the production manifest, then
dbt deps
dbt build --select state:modified+ --defer --state ./prod-artifacts --target ci

Each pull request builds into its own schema. The dbt platform’s CI jobs do this natively (dbt_cloud_pr_<job_id>_<pr_id>); self-hosted, publish the production manifest to object storage after each run and derive the schema from the PR number. Common pitfalls:

• Incremental models in an empty schema run as full refreshes. Run dbt clone first (zero-copy on Snowflake, Databricks and BigQuery) so the incremental logic runs against a realistic table.
• Jinja that renders differently per target can make unchanged models look modified. Keep environment logic in profiles and generate_schema_name.
• No cheap first stage. dbt build --empty (v1.8+) runs the SQL against zero-row inputs, which catches compile and schema errors before you pay for real reads.
• Orphaned PR schemas. Drop them when the pull request closes.

dbt State is the newer option: a dbt Labs service that decides per node whether to rebuild, clone or skip, and defers to production automatically. It supports dbt v1.7+ and v2 on Snowflake, BigQuery, Databricks and Redshift, needs a dbt account and is metered, with a free monthly allowance for self-hosted users. Decide whether you want build decisions to depend on an external service.

How should dbt environments and promotion work?

Promote code between environments and let each one build its own data: developer schemas, a per-PR CI schema, an optional UAT environment for business sign-off, then production. All run from one codebase; differences live only in profiles and environment variables. Build one container image per release with dbt and package versions pinned, and deploy it by git tag, so what passed UAT is exactly what runs in production.

Run production with dbt build rather than separate run and test steps, so a failing test skips downstream models instead of publishing bad data. For zero-downtime releases, build into a staging schema and promote with zero-copy clones, the blue/green pattern the dbt clone docs describe.

Which orchestrator should run dbt in production?

Option	Fits when	Trade-off
dbt platform jobs	You license the platform and want managed CI, Catalog and cross-project refs	Another control plane and another vendor contract
Airflow with Astronomer Cosmos	Airflow is already your standard	Model-level tasks give visibility and retries, but large projects produce large DAGs
Dagster (dagster-dbt)	You want dbt models as assets next to Python and Spark jobs	One more platform to run if you don’t use Dagster already
Databricks Lakeflow Jobs dbt task	The warehouse is Databricks	Tied to one platform
dbt Projects on Snowflake	The warehouse is Snowflake (GA November 2025; slim CI and defer GA September 2026)	Tied to one platform
Container on a cloud workflow service (Cloud Workflows, Step Functions, Azure Data Factory)	A small number of jobs and minimal extra infrastructure	You build retries, alerting and artifact handling yourself

How do you control dbt costs on Snowflake, Databricks and BigQuery?

On every warehouse: staging models are views, large facts are incremental, and big time-series tables can use the microbatch strategy (v1.9+), which processes batches keyed on event_time so backfills and retries stay bounded. Treat --full-refresh as a planned operation. Tests are queries too, so run them on smaller compute.

Snowflake

• Route compute with snowflake_warehouse per folder or model. The dbt docs show the pattern of data tests on an extra-small warehouse and heavy models on a larger one.
• Snowflake bills warehouses per second with a 60-second minimum, and the default auto-suspend is 600 seconds. Set a short auto-suspend on warehouses that only dbt uses.
• Add cluster_by only to large tables that are filtered on the key. Automatic Clustering is a serverless feature billed separately from your warehouses.
• Set query_tag per model so cost can be attributed to models and domains.

Databricks

• Route models to different SQL warehouses or clusters with databricks_compute, defined in the profile.
• Prefer liquid_clustered_by (dbt-databricks 1.6.2+) or auto_liquid_cluster (1.10+). Databricks describes liquid clustering as replacing partitioning and ZORDER.
• Choose the incremental strategy per model (append, insert_overwrite, replace_where, delete+insert, merge, microbatch). merge is the default and is not always the cheapest for append-only facts.

BigQuery

• Combine partition_by and cluster_by. Set require_partition_filter on large tables so consumers cannot scan them whole, and partition_expiration_days on transient ones.
• Use insert_overwrite with dynamic partitions for large incremental facts, and optionally copy_partitions.
• Set maximum_bytes_billed in the connection so a runaway query fails instead of running up the bill, and use the reservation config to route dbt jobs to the right reservation.

How do dbt docs and lineage feed a data catalog?

Write descriptions in YAML, reuse them with docs blocks, and turn on persist_docs so they land as warehouse comments where analysts and BI tools see them. Then publish the artifacts on every production run. DataHub’s dbt source reads manifest.json and catalog.json, and also run_results.json if you want test outcomes on each dataset’s timeline. Copy run_results.json aside before dbt docs generate overwrites it. The dbt platform’s Catalog adds cross-project lineage, and v2 ships a rewritten docs site. For the catalog side, see running DataHub in production.

Which practice fixes which dbt problem?

Symptom	Likely cause	Practice
CI takes hours and costs as much as production	Full builds on every pull request	state:modified+ with --defer, dbt clone for incrementals, --empty as a first stage
Dashboards break after an upstream refactor	No interface between teams	Contracts on public models, versions with deprecation dates
The same metric shows two numbers	Logic duplicated across marts	Shared intermediate models, access: private, one mart per concept
The warehouse bill grows faster than data volume	Table rebuilds, oversized compute, unpruned scans	Incremental or microbatch, compute per folder, partitioning and clustering, byte caps
Tests pass but the numbers are wrong	Only schema-level tests	Unit tests on logic, singular tests on business invariants, reconciliation against sources

Where Duces fits

Our Data & AI practice designs, builds and runs data warehouses and data lakes, on-prem or in the cloud, on a stack that includes Snowflake, Databricks and dbt. For teams that want this structure without growing it by hand, the Data Processing & Transformation Framework is a layered dbt warehouse with SCD2 history, a canonical model, unit, singular and generic tests, dev, test and prod from one codebase with a sandbox per developer, and documentation persisted to the warehouse and published on every deploy. It runs on any dbt-supported warehouse in Google Cloud, AWS or Azure. If you are deciding between one project and a mesh, or your CI bill is growing faster than your models, talk to the people who build it.