Databricks

Documentation Index

Fetch the complete documentation index at: https://docs.conversion.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Before You Begin

• Create a dedicated service principal: Set up a Databricks service principal specifically for Conversion with read-only access to the tables you need. This limits exposure and makes it easy to revoke access if needed.
• Use a dedicated SQL warehouse: Consider creating a dedicated SQL warehouse for Conversion queries to prevent sync operations from impacting other workloads.
• Sync only the data you need: Select only the columns you’ll use, and always filter with last_sync_time to limit results to changed rows.

​

Step 1: Set Up Databricks Access

​

Create a Service Principal

1. In your Databricks workspace, click your user dropdown at the top right and select Settings
2. Navigate to Identity and access → Service Principals and click Manage
3. Click Add new, give your service principal a name (e.g., conversion_sync), and click Add
4. Note the Application ID (a UUID like 63fc7e90-a8a2-4639-afd8-36ef6bb67cfa); you’ll need this later

​

Generate OAuth Credentials

1. On the Service Principal page, go to the Secrets tab
2. Click Generate secret
3. Save the Client ID and Secret securely; the secret is only displayed once

​

Grant Catalog Permissions

1. In the Databricks sidebar, click Catalog and locate your schema
2. Select the schema and open the Permissions page
3. Click Grant and enter your service principal name
4. Grant the following permissions:
   • USE SCHEMA
   • EXECUTE
   • READ VOLUME
   • SELECT
5. Enable “Also grant USE CATALOG” at the bottom

-- Alternatively, run these commands in a SQL worksheet
GRANT USE CATALOG ON CATALOG my_catalog TO `conversion_sync`;
GRANT USE SCHEMA ON SCHEMA my_catalog.my_schema TO `conversion_sync`;
GRANT SELECT ON SCHEMA my_catalog.my_schema TO `conversion_sync`;

​

Grant Warehouse Permissions

1. In the Databricks sidebar, click SQL Warehouses
2. Find your warehouse, click the ⋮ menu, then Permissions
3. Add your service principal and set permissions to Can use

​

Step 2: Allow Conversion’s IP Addresses

​

Step 3: Gather Connection Details

1. Go to SQL Warehouses and select your warehouse
2. Click the Connection details tab
3. Note the following:
   • Server hostname (e.g., dbc-a1b2c3d4-e5f6.cloud.databricks.com)
   • Port (typically 443)
   • HTTP path (e.g., /sql/1.0/warehouses/abc123def456)

​

Step 4: Connect Databricks to Conversion

1. In Conversion, go to Settings → Integrations → Data Warehouse
2. Click Add Integration and select Databricks
3. Enter your connection details:
   • Name: A friendly name for this connection (e.g., “Production Databricks”)
   • Server hostname: Your Databricks server hostname
   • Port: The port number (default: 443)
   • HTTP path: The HTTP path to your SQL warehouse
   • Catalog: The Unity Catalog name (optional, uses default catalog if empty)
   • Schema: The default schema
   • Client ID: The OAuth Client ID from your service principal
   • Client Secret: The OAuth secret you generated
4. Click Connect

​

Step 5: Create a Sync

​

Databricks SQL Reference

​

Converting Timestamps

SELECT 
  email,
  unix_timestamp(created_at) AS created_at,
  unix_timestamp(last_login) AS last_login
FROM users
WHERE updated_at >= from_unixtime({{last_sync_time}})

​

Using last_sync_time

WHERE updated_at >= from_unixtime({{last_sync_time}})

WHERE updated_at_epoch >= {{last_sync_time}}

​

Building Nested Objects with named_struct

named_struct(
  'role', role,
  'quantity', quantity,
  'started_at', unix_timestamp(started_at)
) AS relationshipFields

to_json(named_struct(
  'role', role,
  'quantity', quantity
)) AS relationshipFields

​

Converting Booleans

SELECT 
  email,
  CASE WHEN is_active THEN 'true' ELSE 'false' END AS is_active,
  CASE WHEN is_admin THEN 'true' ELSE 'false' END AS is_admin
FROM users
WHERE updated_at >= from_unixtime({{last_sync_time}})

​

Handling NULLs

SELECT 
  email,
  COALESCE(first_name, '') AS first_name,
  NVL(phone, '') AS phone
FROM users

​

Casting Types

SELECT 
  CAST(user_id AS STRING) AS id,
  CAST(score AS INT) AS lead_score
FROM users

​

Querying Across Catalogs

SELECT u.email, o.order_id
FROM my_catalog.users_schema.users u
JOIN my_catalog.orders_schema.orders o ON u.id = o.user_id

​

Troubleshooting

​

”Permission denied” errors

• USE CATALOG on the catalog
• USE SCHEMA on the schema
• SELECT on each table you want to query
• Can use permission on the SQL warehouse

​

”Could not connect” errors

• Your server hostname is correct
• The HTTP path points to a valid SQL warehouse
• The OAuth Client ID and Secret are correct
• Conversion’s IP addresses are allowed if you use IP access lists

​

”Unauthorized/Forbidden: 403” errors

​

Sync taking too long

• Ensure you’re filtering by last_sync_time to reduce rows
• Select only the columns you need
• Consider using a larger SQL warehouse size
• Reduce sync frequency for large datasets

​

Subquery errors

-- Instead of subqueries
WITH filtered_users AS (
  SELECT * FROM users
  WHERE status = 'active'
)
SELECT * FROM filtered_users
WHERE updated_at >= from_unixtime({{last_sync_time}})

​

Frequently Asked Questions

SELECT u.email, o.order_id
FROM my_catalog.users_schema.users u
JOIN my_catalog.orders_schema.orders o ON u.id = o.user_id