DataHub production deployment: what enterprise teams plan for

Running open-source DataHub in production means operating a small distributed system: a Java metadata service and React frontend on top of a relational database, Elasticsearch or OpenSearch, and Kafka, deployed on Kubernetes with Helm. The software is free under Apache 2.0, but upgrades, backups, SSO, ingestion scheduling and on-call are yours to own. Enterprises usually choose between self-hosting with their own platform team, buying DataHub Cloud as a managed service, or self-hosting with an external support subscription.

This guide covers what each of those responsibilities involves, based on the current DataHub documentation, and ends with a decision table.

What does a self-hosted DataHub deployment consist of?

The Kubernetes guide lists four main components: the metadata service (GMS), the frontend, and two optional consumers (MAE and MCE) that can run inside GMS or as separate deployments. They depend on four external systems: Kafka, a relational database (MySQL, PostgreSQL or MariaDB), a search index (Elasticsearch, or OpenSearch via ELASTICSEARCH_IMPLEMENTATION), and a graph index that is either Elasticsearch/OpenSearch or Neo4j.

Two more pieces matter in practice. The datahub-system-update job applies schema changes and reindexing on every upgrade. The actions container (acryl-datahub-actions) runs scheduled UI ingestion and any event-driven Actions you configure.

             users (browser, SSO)            ingestion (CLI / orchestrator)
                     |                                   |
             datahub-frontend                            |
                     |  GraphQL                          |  REST / Kafka
                     v                                   v
   +------------------------- GMS (metadata service) -------------------------+
   |   writes versioned aspects  ->  MySQL / PostgreSQL (metadata_aspect_v2)  |
   |   emits change log events   ->  Kafka (MetadataChangeLog topics)         |
   +--------------------------------------------------------------------------+
                     |                                   |
            MAE consumer (in GMS or standalone)    datahub-actions
                     |                             (UI ingestion executor,
                     v                              event-driven actions)
       Elasticsearch / OpenSearch: search, graph, timeseries indices

The single most useful fact for operations is that the relational table metadata_aspect_v2 is the source of truth. Search and graph indices can be rebuilt from it by replaying aspects as change log events. Timeseries data (usage statistics, dataset profiles, assertion runs) lives only in the search cluster and needs its own backup.

How do you deploy DataHub on Kubernetes with Helm?

The official charts live in acryldata/datahub-helm: a datahub chart and a prerequisites chart that installs Elasticsearch, Kafka, MySQL and optionally Neo4j. The prerequisites chart is the quickest way to a working cluster. In production we recommend running those dependencies as managed services (a managed PostgreSQL or MySQL, a managed OpenSearch or Elasticsearch domain, managed Kafka) or on infrastructure your platform team already operates and backs up. The AWS guide notes that if you run the storage layer as pods you need at least three nodes, and that is only the start of the sizing conversation.

Points that cause trouble later if you skip them on day one:

• Pin immutable image tags. Since v1.7.0 the project tells Kubernetes users to pin a release tag (v*) or a commit tag (sha-…) and never use quickstart in clusters, because it moves with every green build.
• Respect the Helm hooks. helm upgrade runs the blocking system-update job as a pre-upgrade hook and a non-blocking one afterwards. If your GitOps pipeline renders the chart with helm template and applies YAML through Kustomize, Spinnaker or kubectl, those hooks don’t run. The docs describe a three-phase rendered-manifest procedure for that case.
• Put secrets in Kubernetes secrets or your vault, including the OIDC client secret, the token signing key and salt, and the system client secret used between components.
• Expose only the frontend. GMS is called by ingestion and integrations, so it needs authenticated network access. Keep metadata service authentication on (METADATA_SERVICE_AUTH_ENABLED defaults to true).
• Separate environments. Run a non-production instance that receives a restored copy of production metadata. Upgrades are rehearsed there.

How often does DataHub release, and how should upgrades be planned?

DataHub moves quickly. The official release list shows v1.7.0.1 on 3 September 2026 and v1.8.0 on 12 October 2026, with v1.9.0 release candidates already published. The project also maintains a v1.6.0 line it calls LTS: four patch releases (v1.6.0.1 to v1.6.0.4) shipped between 13 August and 29 October 2026, mostly security and dependency fixes. If you plan to stay on that line, check the release notes for how long it will keep receiving patches before you commit to it.

The upgrade guide is explicit about three things:

1. There is no automated rollback. Some upgrade steps reindex or migrate data and are not reversible. A restorable backup of both the database and the search indices is what makes a failed upgrade recoverable.
2. Read every version you pass through. Going from 1.5.0 to 1.8.0 means reading the 1.6.0, 1.7.0 and 1.8.0 entries in Breaking Changes by Version, and planning the window around the largest Potential Downtime item, not the last one.
3. Some versions are mandatory stops. The breaking-changes page says you must upgrade to v1.6.0 before v1.7.0 (Helm chart 1.0.3, then 1.1.0), because v1.6.0 ships the groundwork for zero-downtime index upgrades.

Breaking changes also reach ingestion, not only the server. A recent entry makes recipes for several sources fail on configuration keys they don’t recognize, where they used to ignore them silently. Each release lists the CLI and Helm chart versions it expects, so pin the ingestion CLI version alongside the server version.

A workable cadence for most enterprises is to pick a target version on a fixed schedule rather than chase every release, rehearse it on the non-production instance, and apply security patch releases in between. After each upgrade, run the checks the docs suggest: GMS healthy, UI loads, a known entity is searchable, its lineage opens, and one ingestion run completes.

How do you set up SSO for DataHub?

SSO is configured on datahub-frontend with OIDC environment variables: client ID, client secret, discovery URI and base URL. It works with Microsoft Entra ID, Okta, Google, Keycloak and other OIDC providers. The callback URL (<base URL>/callback/oidc) must match the redirect URL registered with the identity provider exactly, which is the most common first-day failure.

The decisions that need a governance owner, not just an engineer:

• Provisioning: by default DataHub creates users and groups just-in-time at login. If you want only known users, set AUTH_OIDC_PRE_PROVISIONING_REQUIRED=true and load users and groups with an identity source such as the azure-ad ingestion source.
• Group sync: with AUTH_OIDC_EXTRACT_GROUPS_ENABLED, every login re-syncs group membership from the identity provider and clears memberships assigned in the DataHub UI. Decide which side is the master before you enable it.
• Login restriction: AUTH_OIDC_REQUIRED_GROUPS limits login to members of named groups.
• Machine access: SSO covers people. Ingestion and integrations use access tokens, and ACCESS_TOKEN_ALLOWED_DURATIONS controls which lifetimes users can request.

Authorization in Core is policy-based, with default roles such as Admin and Editor. Fine-grained access control is listed as a DataHub Cloud feature, so check the comparison against your access requirements early.

How should ingestion recipes be run and scheduled?

There are two models, and most enterprises end up using both.

UI ingestion. Sources are created in the DataHub UI with a cron schedule and timezone, and run by the datahub-actions executor inside your cluster. You can also manage these sources as code: datahub ingest deploy --name … --schedule "5 * * * *" -c recipe.yaml creates or updates a scheduled source from a YAML recipe.

CLI ingestion from your orchestrator. datahub ingest -c recipe.yaml runs from Airflow or whichever scheduler you already operate. The docs recommend this when sources aren’t reachable from DataHub’s network, when you need custom logic, or when you want to spread ingestion across environments.

Whichever you pick, a few rules hold up:

• Treat ingestion privileges as code execution. The docs warn that Manage Metadata Ingestion lets a user run Python on the executor, so grant it only to trusted operators.
• Keep recipes in Git, reviewed like any other config, with credentials pulled from a secret store rather than pasted into recipes.
• Enable stateful ingestion with a unique, stable pipeline_name per source, so entities deleted at the source are removed from the catalog instead of lingering.
• Give each source a least-privilege identity. On EKS, the AWS guide shows attaching an IAM role to the actions pod through a service account. Profiling needs more permissions than metadata extraction, because it reads data.

If dbt is your transformation layer, its manifest and run results are among the most valuable inputs to the catalog. We cover the dbt side of that in dbt at enterprise scale.

How do you scale and back up DataHub?

GMS and the frontend scale horizontally. As ingestion volume grows, run the MAE and MCE consumers as standalone deployments so that indexing load doesn’t compete with API traffic. Watch Kafka consumer lag on the change log topics, which is where an overloaded indexing path shows up first. For large reindexing jobs, the restore guide recommends at least 10 to 20 partitions on the change log topic in production, plus enough search shards and temporarily more consumer pods.

Backups follow the storage split:

Store	What it holds	Can it be rebuilt?	Backup approach
MySQL / PostgreSQL (metadata_aspect_v2)	All versioned aspects: schemas, ownership, docs, glossary, lineage edges	No. This is the source of truth	Managed snapshots or a dump of the table (docs suggest 14 to 30 days retention for RDS snapshots)
Search and graph indices	Search documents and relationships	Yes, with the RestoreIndices job	Snapshots still help, because rebuilding a large catalog takes time
Timeseries indices	Usage statistics, dataset profiles, assertion runs	No	Elasticsearch/OpenSearch snapshot repository with a schedule
Kafka	Events in flight	Not needed long term	Retention long enough to cover consumer outages

The backup guide suggests testing restores quarterly or after significant infrastructure changes. We would make one restore test part of every major upgrade rehearsal, since that is when you are most likely to need it.

Who owns DataHub once it is live?

The software is the easier half. A catalog without named owners fills up with stale descriptions and failed ingestion runs that nobody notices. In our delivery approach we split ownership three ways:

• Platform team: Kubernetes, dependencies, upgrades, backups, SSO configuration, monitoring of the DataHub stack itself.
• Data governance lead: domains, glossary, ownership types, access policies, who may create ingestion sources, and the rules for certification or deprecation.
• Source and domain owners: their recipes, schedules and credentials, plus the documentation and ownership of their assets.

Failure modes to look for in the first six months: ingestion sources with no owner that silently fail, access policies widened to all users for convenience and never narrowed again, glossary terms created by several teams for the same concept, and an upgrade backlog that grows until nobody wants to touch the cluster.

Open-source DataHub vs DataHub Cloud: what is different?

DataHub Cloud is the managed commercial edition from the company behind the project, which rebranded from Acryl Data to DataHub in May 2025. Core remains Apache 2.0.

The official comparison shows both editions sharing the catalog fundamentals: 140+ source connectors, column-level lineage, glossary, ownership, data contracts and incident management. Features it lists for Cloud only include a 99.5% uptime SLA, fine-grained access control, AWS PrivateLink and IP restrictions, freshness, volume, schema and custom SQL monitors, AI documentation generation, change proposals, access request workflows, metadata tests, Slack/Teams/email notifications, the Ask DataHub agent, a hosted MCP server, and a private support channel.

The architectural difference is where the metadata lives. DataHub Cloud is a SaaS service; its optional Remote Executor runs in your network so credentials and source connectivity stay with you, and it reports results back to DataHub Cloud. For many companies that is acceptable. For others, where metadata includes sensitive column names, query history or sample values, keeping the catalog inside their own tenancy is a requirement, and that points toward self-hosting.

Self-host, managed or supported self-host: how do you decide?

Criterion	Self-host Core, own team	DataHub Cloud (managed)	Supported self-host
Who runs the infrastructure	You	The vendor	You, with an external support provider
Where metadata is stored	Your tenancy	Vendor SaaS (Remote Executor optional)	Your tenancy, including air-gapped
Upgrades and patching	Your platform team tracks releases	Vendor	Provider certifies and tests versions, you apply them
Support	Community Slack and docs	Vendor support, 99.5% uptime SLA	Contracted with the provider
Features beyond Core	Build them yourself	Vendor’s Cloud-only features	Depends on the provider’s add-ons
Skills you need in-house	Kubernetes, Kafka, search, Java service ops	Catalog administration	Kubernetes basics, catalog administration
Good fit when	Strong platform team, catalog is one of many self-run services	SaaS is acceptable and you want the vendor’s full feature set	Metadata must stay in your tenancy and you don’t want to staff DataHub expertise alone

Two questions settle most decisions. Can your metadata leave your tenancy? If not, DataHub Cloud is out. Will you staff two to three people who understand Kafka, Elasticsearch or OpenSearch and the DataHub upgrade path well enough to handle an incident? That figure is illustrative, not a benchmark, but if the honest answer is no, plain self-hosting is a risk.

Where Duces fits

Duces offers DataHub+, a certified, supported distribution of open-source DataHub that runs in your own tenancy: on-prem, private cloud or fully air-gapped. It runs an unmodified DataHub Core. We pin, test and publish a certified release on a fixed cadence and support the current and previous certified release, with signed images, SBOMs, CVE scanning, and Helm or operator install with tested upgrade, backup and restore tooling. Support includes a private channel, a named technical account manager and response targets agreed in your contract. Three optional plugins (AI documentation with your own LLM, monitoring and assertions, alerting and propagation) talk to DataHub through its public APIs, the Actions framework and the Metadata Change Log.

DataHub+ sits within our Data & AI practice. Duces is independent: DataHub+ is not affiliated with, sponsored or endorsed by the DataHub project or its commercial sponsor, and “DataHub” is a trademark of its respective owner.

If you are weighing these options for your own environment, talk to the people who build it.