Skip to main content

Connecting Fivetran and Databricks on skyDataP

Overview​

Skypoint provides built-in connectors and the flexibility to integrate with external tools like Fivetran, dbt, Power BI, and Tableau to seamlessly import data from user databases. With Fivetran, users can securely transfer data from multiple sources to Databricks on skyDataP within a robust cloud network. This guide will walk you through the steps to set up your data pipeline.

Find the architectural outline of the data pipeline as follows:

Alt text

Prerequisite​

  • You have an active account with Fivetran and skyDataP & credentials to access them.

Setup Guide to Connect Fivetran and Databricks on skyDataP​

The setup involves two key steps:

  • Integrating your Data Source with Fivetran

  • Connect Fivetran to Databricks on skyDataP

Step 1: Integrate your Data Source with Fivetran​

Fivetran offers a comprehensive range of connectors to integrate with your databases.

  1. Follow the Fivetran setup guide to configure the connectors, required to set up connection with your data source.

  2. Prepare the following prerequisites for your source and destination to streamline the setup:

    • Valid database credentials

    • An active Fivetran account

    • skyData account with access to the Studio.

Step 2: Connect Fivetran to Databricks on skyDataP​

Actions on skyData Studio​

  • Log in to your skyData Studio account.

  • Go to Lakehouse > Workspaces.

  • Click the Show Fields button to display connection strings.

Alt text

  • Copy the following credentials:

    • Server Hostname
    • HTTP Path
    • Personal Access Token

Alt text

Actions on Fivetran​

  • Log in to your Fivetran account.

  • From the left panel, navigate to Destinations.

Alt text

  • Click Add Destination.

Alt text

  • Select Databricks as your destination.
note

This selection is based on the fact that Skypoint Lakehouse is built on Databricks, ensuring seamless integration, enhanced compatibility, and optimal performance.

Alt text

  • Name the destination by following the naming convention for the destination as follows:

skypoint_aip_<Tenant_name>__<Instance_name>_lakehouse_bronze

Components:

  • skypoint_aip: Identifies the skyDataP platform.
  • <Tenant_name>: Placeholder for the tenant's name.
  • <Instance_name>: Placeholder for the instance name.
  • lakehouse_bronze: Specifies the Bronze layer in the Lakehouse architecture.

Alt text

  • Key in the required credentials by copying from skyData Studio.

Alt text

  • Under Authentication Type, select Personal Access Token from the dropdown menu.

  • Key in the personal access token in the field below.

  • Click Save & Test to verify the connection.

Alt text

Congratulations! You’ve successfully connected Fivetran with Databricks on skyDataP. Your data pipeline is ready to sync securely!​

Post-Setup Verification​

  • The new connection will appear on the Destinations page.

  • Click on the connection to view its details.

  • Use the Actions menu to:

    • Test the connection

    • Rename the destination

    • Edit connection details

Alt text

Next step​

After completing the data import, start the Master Data Management (MDM) - Stitch process to develop a unified view of your customers.