> ## Documentation Index
> Fetch the complete documentation index at: https://docs.skypoint.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Google Cloud Storage

> Connect Google Cloud Storage to import data into Skypoint AI

<Tabs>
  <Tab title="Import data">
    ## Overview

    Google cloud Storage enables you to store, access, and maintain data so that you do not need to own and operate data centers, moving expenses from a capital expenditure model to operational. You can use SkyPoint AI's built-in connector for importing data from Google Cloud Storage.

    This document will guide you through the process of connecting Google Cloud Storage to SkyPoint AI.

    ## Prerequisite

    You will need the following details to configure and import data using [Google Cloud Storage](https://cloud.google.com/storage):

    * Access key ID and Secret access key
    * Project ID
    * Service URL
    * Service account JSON file
    * Google cloud bucket.

    ### Permissions required for Service account

    You can control the access to the resources in your Google Cloud project with the help of [IAM](https://cloud.google.com/storage/docs/access-control/iam) (Identity and Access Management). The resources comprise of Cloud Storage buckets and objects that are stored within buckets, along with other Google Cloud entities such as [Compute Engine instances](https://cloud.google.com/compute).

    [Permissions](https://cloud.google.com/transfer-appliance/docs/4.0/prepare-permissions) allow principals to perform specific actions on buckets or objects in Cloud storage. There are two options to prepare the Google cloud permissions and storage:

    * Option 1: Using the Transfer Appliance Cloud Setup Application
    * Option 2: Configuring Google Cloud permissions and Cloud Storage step-by-step.

    <Note>
      You need to create the JSON file and Permissions by logging into Google Cloud Console. You can refer to the Setting up Google Cloud Storage document to learn more.
    </Note>

    ## Import data using Google Cloud Storage connector

    Follow the below steps to create a new dataflow for the Google Cloud Storage import connector:

    1. Go to **Dataflow > Imports**.
    2. Click **New dataflow**.

    The **Set dataflow name** page appears.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Setdataflowname.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=ed4b5f39befc5c1825ced7d995b0654a" alt="Set dataflow name" style={{border: "1px solid black"}} width="727" height="365" data-path="doc_snippets/Googlecloud_Setdataflowname.png" />

    3. In the **Set dataflow name** page, type dataflow name in the **Name** text area.
    4. Click **Next**.

    The **Choose connector** page appears.

    <img src="https://mintcdn.com/skypoint/GxPM0bRCdkUs_uDj/doc_snippets/Googlecloud_Chooseconnector.png?fit=max&auto=format&n=GxPM0bRCdkUs_uDj&q=85&s=d15a3e602309c7ddb50e7a229d89a829" alt="Choose connector" style={{border: "1px solid black"}} width="1625" height="686" data-path="doc_snippets/Googlecloud_Chooseconnector.png" />

    ## To add Google Cloud Storage connector

    1. In the **Choose connector** page, select **Google Cloud Storage** connector.

    <Note>
      You can use the **Search** feature too to find the connector. Also, the **Google Cloud Storage** connector can be found under **Cloud** category.
    </Note>

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Setdataflowconfigure1.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=ff31047b7707b3c434a0c0e9752dbd97" alt="Set dataflow configure" style={{border: "1px solid black"}} width="435" height="551" data-path="doc_snippets/Googlecloud_Setdataflowconfigure1.png" />

    2. Enter **Display Name** for your dataflow in the text area.
    3. Enter **Description** for your dataflow in the text area.
    4. Click **Next**.

    The **Connect to Google Cloud Storage** page appears.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Configurationpage.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=86341f367494b91af2a7059d28b01745" alt="Configuration page" style={{border: "1px solid black"}} width="1731" height="745" data-path="doc_snippets/Googlecloud_Configurationpage.png" />

    ## To configure Google Cloud Storage

    Follow the below steps to configure the connection to Google Cloud Storage:

    1. Enter your credentials such as **Access key ID**, **Secret access key**, **Project ID**, and **Service URL** to configure with Google Cloud Storage.
    2. Click **Choose file** to upload the JSON file.
    3. Click the **Folder icon** in the **Google cloud bucket** text area.

    Once you select the cloud bucket, the **Table Details** columns appear.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Tabledetails.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=efcd088c949402d47e0b15368b62e44c" alt="Table details" style={{border: "1px solid black"}} width="1658" height="761" data-path="doc_snippets/Googlecloud_Tabledetails.png" />

    4. Enter the **Table Details** to process the data.

    | Item                | Description                                                                                      |
    | :------------------ | :----------------------------------------------------------------------------------------------- |
    | Purpose             | Option to assign a purpose (Data or Metadata) for each table.                                    |
    | Data                | Loads customer data.                                                                             |
    | Metadata            | Loads Metadata.                                                                                  |
    | File Name           | Displays the name of the file that you imported.                                                 |
    | Table Name          | Displays the imported table name.                                                                |
    | Datetime format     | Displays a number of **Datetime Formats** and Skypoint AI is set to automatically detect them.   |
    | Delimiter           | Displays available separators for the variables in the imported data.                            |
    | First Row as Header | Check the box for the system to automatically collect the data according to the Header Contents. |
    | Advanced Settings   | Select the options to fine tune the Import process with minute details.                          |

    5. Click the **Advanced settings** for your desired file name.

    The **Advanced settings** pop-up appears.

    <img src="https://mintcdn.com/skypoint/GxPM0bRCdkUs_uDj/doc_snippets/Googlecloud_Advancedsettings.png?fit=max&auto=format&n=GxPM0bRCdkUs_uDj&q=85&s=dee9b50c17a38477fbf311b5da68a0a1" alt="Advanced settings" style={{border: "1px solid black"}} width="402" height="494" data-path="doc_snippets/Googlecloud_Advancedsettings.png" />

    | Item             | Description                                                                                                                                                                                 |
    | :--------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
    | Compression type | Method that is used for compressing the details from source, Azure Data Lake Storage Gen2.                                                                                                  |
    | Row delimiter    | A separator that identifies the boundaries of the flow of a data stream. In case, a different separator is used in it, the information requires change for more accuracy in data ingestion. |
    | Encoding         | As the data comes in data stream, there is a type of encoding used for deciphering it. The default encoding is **UTF-8**.                                                                   |
    | Escape character | It is a particular case of metacharacters that is given an identification of start or end. You can manually select it from the drop-down list.                                              |
    | Quote character  | You can select one of the advanced Quote characters mentioned in the drop-down list.                                                                                                        |

    6. Click **Save** on the **Advanced settings** pop-up to save the advanced settings.
    7. Click **Save**.

    ## Run, edit, and delete the imported data

    Once you save the connector, the Google Cloud Storage connector gets displayed in the list of tables created in the Dataflow page.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Importoutput.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=d0e59f1f9936e5443f7dfd75f13136b8" alt="Import output" style={{border: "1px solid black"}} width="1645" height="502" data-path="doc_snippets/Googlecloud_Importoutput.png" />

    | Item              | Description                                                                                                   |
    | :---------------- | :------------------------------------------------------------------------------------------------------------ |
    | Name              | Displays the name of the imported Dataflow.                                                                   |
    | Type              | Displays connector type symbol.                                                                               |
    | Status            | Indicates whether the data is imported successfully.                                                          |
    | Tables Count      | Displays the number of tables.                                                                                |
    | Created Date      | Displays date of creation.                                                                                    |
    | Last refresh type | Displays the refresh value. You can see whether the value is Full or Incremental after the last data refresh. |
    | Updated Date      | Displays last modified date.                                                                                  |
    | Last Refresh      | Displays the latest refresh date, which updates each time the data is refreshed.                              |
    | Group by          | Option to view the items in a specific Group (For example, name, type, status).                               |

    * Select the horizontal ellipsis in the **Actions** column and do the following:

    | If you want to                       | Then                                                                                               |
    | :----------------------------------- | :------------------------------------------------------------------------------------------------- |
    | Modify the Dataflow                  | Select **Edit** and modify the Dataflow. Click **Save** to apply your changes.                     |
    | Execute the Dataflow                 | Select **Run**.                                                                                    |
    | Bring the data to its previous state | Select **Rollback**.                                                                               |
    | Delete the Dataflow                  | Select **Remove** and then click the **Delete** button. All tables in the data source get deleted. |
    | See the run history of the Dataflow  | Select **Run** history.                                                                            |
  </Tab>

  <Tab title="Setup guide">
    ## Overview

    Specific actions can be performed on buckets or objects in Cloud Storage by principals. This is enabled by Permissions. The process of giving permissions to the principals includes granting of roles, comprising of one or more permissions bundled with them.

    ## Prerequisite

    You need the following resources to set up permissions for service account:

    * Google Cloud Storage bucket
    * Google Cloud account.

    ## Download JSON to Set up Google Cloud Storage

    Follow the below steps to download the JSON file from Google Cloud Storage console:

    1. Log in to your [Google Cloud Storage console](https://developers.google.com/maps/premium/devconsole-access).
    2. Go to **IAM & Admin > Service Accounts**.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_ConsoleServiceacc.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=d404c9e6b6e88bee277905f5103936a4" alt="Console Service account" style={{border: "1px solid black"}} width="639" height="584" data-path="doc_snippets/Googlecloud_ConsoleServiceacc.png" />

    <Note>
      You can find your buckets under Cloud Storage > Buckets. To create new buckets, click on **Create**.
    </Note>

    3. In the Service accounts page, click the **Vertical ellipsis** under **Actions** column.

    The **Actions** menu appears.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_ConsoleServiceacccreated.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=20100ddf9270e260b6f8af9e01e04090" alt="Console Service account created" style={{border: "1px solid black"}} width="1659" height="525" data-path="doc_snippets/Googlecloud_ConsoleServiceacccreated.png" />

    4. Click **Manage keys**.
    5. Click **ADD KEY > Create new key**.

    The **Create private key** pop-up appears.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_ConsoleKeyJSON.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=e3a839efebca989c2eed044c4f1dc838" alt="Console Key JSON" style={{border: "1px solid black"}} width="541" height="345" data-path="doc_snippets/Googlecloud_ConsoleKeyJSON.png" />

    6. Select **JSON**.
    7. Click **CREATE**.

    The **Service Account JSON** file gets created.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Privatekeyoutput.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=15f882eb8b24075fc7d581a4cbb2272b" alt="Private key output" style={{border: "1px solid black"}} width="843" height="214" data-path="doc_snippets/Googlecloud_Privatekeyoutput.png" />

    <Note>
      You will have to upload this JSON file by clicking on Choose File while configuring the Google Cloud Storage import connector on Skypoint AI.
    </Note>

    ## To add Permissions for Service Accounts

    Follow the below steps to add permissions for your service accounts:

    1. Go to **IAM & Admin > Service Accounts**.
    2. Click **Permissions** tab.
    3. Click **GRANT ACCESS**.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Grantaccess.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=6a8b9f44699d339a946b9cfddbf976a1" alt="Grant access" style={{border: "1px solid black"}} width="966" height="645" data-path="doc_snippets/Googlecloud_Grantaccess.png" />

    4. In the **Grant access** page, click **ADD ANOTHER ROLE**.

    <img src="https://mintcdn.com/skypoint/GxPM0bRCdkUs_uDj/doc_snippets/Googlecloud_Addrole.png?fit=max&auto=format&n=GxPM0bRCdkUs_uDj&q=85&s=94e4557ae39cc8b3e0bd8c4391c051d6" alt="Add role" style={{border: "1px solid black"}} width="610" height="673" data-path="doc_snippets/Googlecloud_Addrole.png" />

    5. Select a desired role to assign under **Cloud Storage** in the **Role** drop-down list.
    6. Click **Save**.

    ### Create a custom Role

    In case, you need to create custom roles, follow the below steps:

    1. Go to **IAM & Admin > Roles**.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Createrole.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=4b0dd8cebd06f7a7add399dfe6d8cb64" alt="Create role" style={{border: "1px solid black"}} width="1056" height="432" data-path="doc_snippets/Googlecloud_Createrole.png" />

    2. Click **CREATE ROLE**.

    <Info>
      To edit an existing role, click the vertical ellipses under the Status column. This new role can be assigned in the Service Accounts.
    </Info>

    3. Select **General Availability** in the **Role launch** stage drop-down list.
    4. Click **ADD PERMISSIONS** to create permission for the assigned role.
    5. Click **CREATE**.

    <img src="https://mintcdn.com/skypoint/K3_LcToM0Avux2w8/doc_snippets/Googlecloud_Roleedit.png?fit=max&auto=format&n=K3_LcToM0Avux2w8&q=85&s=9beacc71228d0a07243f27eb5360ef4c" alt="Role edit" style={{border: "1px solid black"}} width="874" height="824" data-path="doc_snippets/Googlecloud_Roleedit.png" />
  </Tab>
</Tabs>

## Next step

After completing the data import, start the [Master Data Management (MDM) - Stitch](stitch.md) process to develop a unified view of your customers.