Connection to Databricks (native)

To connect a Databricks workspace to the DQC Platform, you provide the workspace host, an HTTP path pointing at a SQL Warehouse or interactive cluster, the Unity Catalog name and credentials. DQC uses the native Databricks SQL protocol — no Spark cluster startup is required when targeting a SQL Warehouse.

If you previously used the legacy Databricks (Spark) connector, this is the modern replacement. The legacy connector is being phased out — see the migration note at the bottom of this page.

Add a Databricks connector

To begin:

1. Go to the Connectors > Data sources screen

2. Click the Databricks tile

3. Enter the connection parameters described below and choose an authentication method

Required connection information

Choose an authentication method

Option A — Personal Access Token

In Databricks open User Settings → Developer → Access tokens and generate a new token. Recommended for quick setup and individual workspaces.

Option B — OAuth service principal

Recommended for production. Provide:

• Client ID — the application ID of the service principal

• Client Secret — the OAuth secret generated for the service principal

Grant the service principal USE CATALOG, USE SCHEMA and SELECT privileges on the target catalog/schema in Unity Catalog.

Unity Catalog permissions

Grant the technical user or service principal the following privileges on the target catalog and schema:

GRANT USE CATALOG ON CATALOG <catalog> TO `<service_principal>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service_principal>`;
GRANT SELECT ON SCHEMA <catalog>.<schema> TO `<service_principal>`;
GRANT CREATE VOLUME ON SCHEMA <catalog>.<schema> TO `<service_principal>`;

Why CREATE VOLUME?

The native connector follows Unity Catalog best practice for handling temporary data: instead of writing intermediate results to ungoverned storage, it stages them in a governed Unity Catalog Volume inside the catalog and schema you configure. The volume serves purely as a temporary staging area during query execution. In most cases it only holds technical metadata and short-lived helper files that Databricks compute requires. Staged files are removed after use, and no permanent copies of your data are created.

This permission does not:

• grant access to any data you have not explicitly shared

• allow permanent modifications to your data

• bypass existing Unity Catalog access controls

SQL Warehouse vs. Compute Cluster

• SQL Warehouse — recommended. Lower latency, no Spark session start-up, HTTP Path looks like /sql/1.0/warehouses/<id>.

• Compute Cluster — use only if you need an existing interactive cluster. HTTP Path looks like /sql/protocolv1/o/<workspace_id>/<cluster_id>.

What gets imported

• All tables and views in the configured catalog/schema that the user (or service principal) can read

• Delta, Parquet and external tables are all supported

• Catalog and schema are bound to the connector — to query a different catalog or schema, create a new connector

Migrating from the legacy Databricks (Spark) connector

The legacy Databricks (Spark) connector — which used PySpark and required a Cluster ID — is deprecated and will be removed in a future release. Recreate your connection using this connector and the SQL Warehouse compute type whenever possible. Migration is straightforward: host, credentials (token or service principal), catalog and schema stay the same; only the HTTP Path is added. Remember to also grant the CREATE VOLUME privilege described above.

Connect and assign permissions

After saving, DQC validates the credentials, lists the available tables and lets you pick which ones to expose. To control who can use the connector inside DQC, see Data source access permissions.