Skip to main content

​
Imports

​
Overview

Dataflow allows you to import data from the relevant sources, transform, and load it into the Lakehouse. skyData imports raw customer data from across systems and applications with over 200 connectors spanning e-commerce platforms, marketing automation tools, email service providers, cloud platforms, and many more. You can ingest data to skyData from any data source including pre-built connectors with popular applications such as Shopify, Salesforce, etc. skyData Studio ingests raw data and transforms it into a standardized, modular, and extensible collection of data schemas for further processing in downstream processes such as Master Data Management (MDM) - Resolve process. Alt text

​
To add new Dataflow

  1. In the left pane, go to Dataflow > Imports.
The Dataflow window appears. Alt text
  1. Click New Dataflow to create dataflow.
Alt text
  1. Enter a Name to identify your Dataflow.
The name must start with a letter and can only contain letters and numbers, with no spaces between the characters.
  1. Click Next.

​
Add connectors

  1. Choose one of the existing Connectors to access the data.
Alt text

​
Connectors categorization

You can filter the list of connectors concerning the following categories:
  1. You can use the search function to find the available connectors. For more information about different type of connectors and their configuration, refer to the Connector section.
Alt text
  1. Enter the Display Name and the Description of the connector.
  2. Click Next.
Alt text
  1. Enter the account credentials to configure with the selected connector.
For each connector, credentials may differ. You need to enter the relevant information based on the connector to configure and import the data. In this example, we have selected the Azure Data Lake Storage Gen2 connector to import data. For the Azure Data Lake Storage Gen2 connector, you need to enter the Storage account name, Account key, and Storage path to import the data.
  1. If you want to keep the same date-time format for all the tables, select the check box.
  2. Click Save
  3. Select the Table Details to process the data.
After configuration, all tables that are part of the connector are listed in the Table Details. By default, it displays a single table upon successful connection, showing information specific to the client connector and the tables it supports. For multiple tables, you can select only those tables you want to import and process. For example, to import customer data, choose the tables that contain customer information such as name, email, address, and contact details.
Alt text
You can select the Datetime in MM/DD/YYYY format, for example, 11/25/1979. It helps the system to understand the data format and apply further formatting if required during downstream processes. The same Datetime format as set in the instance settings can be seen for the Datetime fields in the profile card. If you wish to see the profile card with Datetime as imported from the source, you can unselect the option in Stitch > Map > advanced settings.
  1. If necessary, apply the Advance settings to modify the default settings.
Advanced settings allow you to modify the default settings. It gives more flexibility to apply advanced use cases. However, the default settings are adequate to perform the task.
Alt text
  1. Click Save to apply the changes.

​
Run, edit, and delete the imported data

  1. After saving the connection, imported data appears on the Dataflow page. Also, you can see a list of the created tables in the Databases section.
Alt text
  1. Select the horizontal ellipsis in the Actions column and do the following:
The Run history for SQL connector import in Dataflow shows all the states for ingestion and the completion time reflects the complete time for the pipeline completion. Also, you can see the error message corresponding to failure while importing data from a data source under Description.
Alt text

​
To configure Email Notification for pipeline status

Now you can set up email notifications to stay informed about pipeline success or failure events in real time. Follow these steps to configure notifications.
  1. Click on the Pipeline name for which you want to configure notifications.
Alt text Configuration page appears Alt text
  1. Click Notification Settings on the right corner of the screen.
  2. Notification settings form appears, check the box Enable Notifications to proceed.
Alt text
  1. Enter the mail id to Add user.
  2. Click on the Add button to add multiple users.
  3. Access the Manage User option to configure notifications for pipeline completion of statuses of success, failure, or both.
Alt text
  1. Click Save to apply for changes.
  2. User will get the mail notification based on the configuration, such as Sucess/Failure or Both.
Notifications set for a specific connector are limited to that connector only. You won’t receive updates for other connectors unless configured separately.
  1. Follow the same steps to configure notifications for other pipelines.

​
Get assistance for your connector

In case, you are unable to find your desired import connector, you can perform one of the action in the Looking for more section. You can raise a support ticket for the Skypoint team to look into your request.

​
List of all import connectors

​
Next step

After completing the data import, start the Master Data Management (MDM) - Stitch process to develop a unified view of your customers.

​
See also