Databricks

This guide walks you through connecting Databricks to Conversion and setting up your first sync.

Before You Begin

Make sure you have the following before starting:

Workspace admin access in Databricks: You’ll need to create a service principal, generate an OAuth secret, and grant it access to a SQL warehouse and your data.
A SQL warehouse: Conversion runs queries on a SQL warehouse, not an all-purpose cluster. Serverless warehouses start in seconds; classic and pro warehouses can take a few minutes.
The catalog(s) you want to sync identified: Know which Unity Catalog catalogs, and optionally which schemas and tables, Conversion should be able to read from.

Step 1: Set Up Databricks Access

Conversion connects to Databricks using OAuth machine-to-machine authentication with a service principal. In this step you’ll create a service principal, generate an OAuth secret, grant it read access, and (if needed) configure network access.

Create a Service Principal

We recommend creating a dedicated service principal for Conversion so its activity is easy to monitor and revoke.

In your Databricks workspace, click your username at the top right and select Settings.
Go to Identity and access → Service principals and click Manage.
Click Add service principal, create a new Databricks-managed service principal (e.g. conversion_sync), and click Add.
Copy the service principal’s Application ID. This is the Client ID you’ll enter in Step 2.

On Azure Databricks, the service principal must be Databricks-managed. Microsoft Entra ID-managed service principals are not supported.

Generate an OAuth Secret

Open the service principal and go to the Secrets tab.
Click Generate secret. The default scope is sufficient.
Copy the Secret; it is only shown once. You’ll paste it into Conversion in Step 2.

OAuth secrets expire after at most 730 days. To rotate, generate a new secret, then open your connection in Conversion, click Edit connection, and enter the new Client secret. Conversion tests the new secret before saving it.

Grant Read Access to Your Data

Run the following in the Databricks SQL editor as a catalog owner or admin. Replace my_catalog and my_schema with your names, and use the service principal’s Application ID in backticks.

-- Allow the service principal to read everything (current + future) in the schema
GRANT USE CATALOG ON CATALOG my_catalog           TO `<application-id>`;
GRANT USE SCHEMA  ON SCHEMA  my_catalog.my_schema TO `<application-id>`;
GRANT SELECT      ON SCHEMA  my_catalog.my_schema TO `<application-id>`;

Schema-level grants also cover tables you add later. To grant a whole catalog, run GRANT USE SCHEMA, SELECT ON CATALOG my_catalog instead.

The schema browser in Conversion shows only the tables the service principal has been granted.

Granting Access table-by-table

If your schema contains data Conversion shouldn’t see, grant access to specific tables instead:

GRANT USE CATALOG ON CATALOG my_catalog                  TO `<application-id>`;
GRANT USE SCHEMA  ON SCHEMA  my_catalog.my_schema        TO `<application-id>`;
GRANT SELECT      ON TABLE   my_catalog.my_schema.users  TO `<application-id>`;
GRANT SELECT      ON TABLE   my_catalog.my_schema.orders TO `<application-id>`;
-- Add more tables as needed

With table-by-table grants, Conversion only sees the tables you explicitly include. Grant access to each new table you want to sync.

Grant Access to the SQL Warehouse

In the Databricks sidebar, click SQL Warehouses and open your warehouse.
Click Permissions, add the service principal, and give it Can use.
Open the Connection details tab and copy the Server hostname and HTTP path. You’ll need both in Step 2.

Allow Conversion’s IP Addresses

If your workspace uses IP access lists, allow connections from Conversion’s IP addresses:

Region	IP Addresses
US	35.239.90.161, 35.188.167.166, 34.56.101.43, 34.122.97.230, 34.29.176.66, 35.226.154.44

Large query results are downloaded directly from your workspace’s cloud storage. If that storage has a firewall or bucket policy, allow the same IP addresses there too.

Workspaces that only accept front-end PrivateLink connections (public access disabled) are not supported.

Step 2: Connect Databricks to Conversion

In Conversion, go to Settings → CRM & Syncing → Connections.
Click Add Databricks connection.

Fill in the connection details using the values from Step 1:

Field	What to enter
Name	A label such as `Production Databricks`
Server hostname	Your workspace hostname, e.g. `dbc-a1b2c3d4-e5f6.cloud.databricks.com`
HTTP path	Your SQL warehouse path, e.g. `/sql/1.0/warehouses/abc123def456`
Client ID	The service principal’s Application ID
Client secret	The OAuth secret you generated. Conversion stores it encrypted.

Click Create connection.

Conversion will verify the connection. If successful, you’re ready to create syncs.

A stopped classic or pro warehouse starts automatically when a sync runs, which can take a few minutes. Frequent syncs can keep the warehouse running, so use serverless or a longer schedule to manage cost.

Step 3: Create a Sync

After connecting, open your new Databricks connection and go to the Syncs tab to create a sync. Read more about Setting Up a Sync.

Databricks SQL Reference

Table names

Refer to tables by their full name, catalog.schema.table. The connection has no default catalog or schema. The schema browser inserts the full name when you click a table. The schema browser lists Unity Catalog tables only. Tables in hive_metastore can still be queried by full name.

Converting Timestamps

Conversion expects Unix timestamps for date/time fields. Use unix_timestamp() to convert TIMESTAMP columns:

SELECT
  email,
  unix_timestamp(created_at) AS created_at,
  unix_timestamp(last_login) AS last_login
FROM my_catalog.my_schema.users
WHERE updated_at >= from_unixtime({{last_sync_time}})

Using last_sync_time

For incremental syncing, compare your TIMESTAMP column with from_unixtime({{last_sync_time}}). The variable is an integer in Unix seconds, so don’t wrap it in quotes:

WHERE updated_at >= from_unixtime({{last_sync_time}})

Or convert your column to Unix seconds and compare integers directly:

WHERE unix_timestamp(updated_at) >= {{last_sync_time}}

To add a buffer that catches rows whose updated_at might be slightly stale, subtract seconds before converting:

WHERE updated_at >= from_unixtime({{last_sync_time}} - 300)  -- 5 minute buffer

Building Nested Objects with named_struct

Use named_struct to create nested objects like relationshipFields:

SELECT
  email,
  named_struct(
    'role', role,
    'quantity', quantity,
    'started_at', unix_timestamp(started_at)
  ) AS relationshipFields
FROM my_catalog.my_schema.memberships

Handling NULLs

Use COALESCE or NVL to provide default values:

SELECT
  email,
  COALESCE(first_name, '') AS first_name,
  NVL(phone, '') AS phone
FROM my_catalog.my_schema.users

Casting Types

Use CAST to convert between types:

SELECT
  CAST(user_id AS STRING) AS id,
  CAST(score AS INT) AS lead_score
FROM my_catalog.my_schema.users

Troubleshooting

”Permission denied” errors

Ensure your service principal has the required grants:

Can use on the SQL warehouse
USE CATALOG on the catalog
USE SCHEMA on the schema
SELECT on each table you want to query

”Could not connect” errors

Verify that:

The server hostname is the workspace hostname only, without https:// or a path
The HTTP path points to a SQL warehouse (/sql/1.0/warehouses/...), not a cluster
The Client ID is the service principal’s Application ID and the secret has not expired
Conversion’s IP addresses are allowed if your workspace uses IP access lists

Previews work but syncs fail

Syncs download large results from your workspace’s cloud storage. If that storage has a firewall or bucket policy, allow Conversion’s IP addresses there as well.

Sync taking too long

Ensure you’re filtering by last_sync_time to reduce rows
Select only the columns you need
Consider using a larger or serverless SQL warehouse
Reduce sync frequency for large datasets

Frequently Asked Questions

What Databricks permissions does Conversion need?

The service principal needs Can use on the SQL warehouse, plus USE CATALOG, USE SCHEMA, and SELECT on the data you want to sync. Conversion only reads data; it never writes to your Databricks workspace.

Can I use a personal access token?

No. Conversion connects with OAuth using a Databricks-managed service principal.

How do I sync from multiple catalogs or schemas?

Grant the service principal access to each one and use full table names in your query:

SELECT u.email AS email, o.order_id AS order_id
FROM crm.core.users u
JOIN sales.core.orders o ON u.id = o.user_id

Records

Emails

Workflows

Audiences

Forms

Content

Campaigns

Scoring

Data Syncing

Unified Query Engine

Folders

Liquid

Tracking

Workspace Settings

Data and Privacy

Guides

Before You Begin

Step 1: Set Up Databricks Access

Create a Service Principal

Generate an OAuth Secret

Grant Read Access to Your Data

Granting Access table-by-table

Grant Access to the SQL Warehouse

Allow Conversion’s IP Addresses

Step 2: Connect Databricks to Conversion

Step 3: Create a Sync

Databricks SQL Reference

Table names

Converting Timestamps

Using last_sync_time

Building Nested Objects with named_struct

Handling NULLs

Casting Types

Troubleshooting

”Permission denied” errors

”Could not connect” errors

Previews work but syncs fail

Sync taking too long

Frequently Asked Questions

​Before You Begin

​Step 1: Set Up Databricks Access

​Create a Service Principal

​Generate an OAuth Secret

​Grant Read Access to Your Data

​Granting Access table-by-table

​Grant Access to the SQL Warehouse

​Allow Conversion’s IP Addresses

​Step 2: Connect Databricks to Conversion

​Step 3: Create a Sync

​Databricks SQL Reference

​Table names

​Converting Timestamps

​Using last_sync_time

​Building Nested Objects with named_struct

​Handling NULLs

​Casting Types

​Troubleshooting

​”Permission denied” errors

​”Could not connect” errors

​Previews work but syncs fail

​Sync taking too long

​Frequently Asked Questions

Before You Begin

Step 1: Set Up Databricks Access

Create a Service Principal

Generate an OAuth Secret

Grant Read Access to Your Data

Granting Access table-by-table

Grant Access to the SQL Warehouse

Allow Conversion’s IP Addresses

Step 2: Connect Databricks to Conversion

Step 3: Create a Sync

Databricks SQL Reference

Table names

Converting Timestamps

Using last_sync_time

Building Nested Objects with named_struct

Handling NULLs

Casting Types

Troubleshooting

”Permission denied” errors

”Could not connect” errors

Previews work but syncs fail

Sync taking too long

Frequently Asked Questions