Google Cloud Storage

In this doc, we will cover how to setup a Connection to Google Cloud Storage.

Prerequisites

• Access credentials
• Data location on Google Cloud Storage
• Data Schema (column names and column type)

Create a New Google Cloud Storage Connection

Fields denoted with a red dot are required

• ACCESS TYPE: Select whether this connection can be used for both Read and Write Connectors.
• CONNECTION NAME: The name to identify the connection.
• PROJECT: The name to identify the project.
• BUCKET: Google Cloud Storage bucket where the data is located, such as bucket-io-gcs.
• REQUIRES CREDENTIALS: If de-selected, only the publicly accessible buckets can be accessed.
• CHOOSE CREDENTIALS: Choose from an existing credential or create new credential for connecting to Google Cloud Storage if the Required Credentials checkbox is selected. When creating a new connection that requires credentials, access the drop down to select an existing credential or a new credential. The new credential will require Storage Legacy Object Reader and Storage Legacy Bucket Reader level access or higher in GCS. Although higher permissions will result in a successful connection, keep best practices in mind and adhere to the principle of least privilege.

• CREDENTIAL NAME: The name to identify the credential.
• CREDENTIAL TYPE: Google Cloud Storage.
• GOOGLE CLOUD CREDENTIALS: Private key used to identify the Google Cloud Storage account. When a Service Account is created a private key is produced to provide authentication between Google Cloud and third party platforms. In the GOOGLE CLOUD CREDENTIALS field enter the private key for the selected service account. The private key will be a block of JSON, provide the entire key which includes the type, project_id, private_key_id, private_key, client_email, client_id, auth_uri, token_uri, auth_provider_x509_cert_url and client_x509_cert_url. Keep in mind that Google Service Accounts are bound to a project. Each Google project will require a new credential object.

Google Cloud Storage Credentials for Write Connection

Enter the JSON key for the service account which has Storage Admin and Storage Object Admin for the GCS path. Refer to Google documentation for more details on GCS authentication.

🚧

Storage API

We relied on the Google Storage API to write into GCS locations. As a result, we require the Storage API to be enabled. This should be enabled by default in the GCP but in case it's not, user can enable it by going to https://console.developers.google.com/apis/api/storage-api.googleapis.com/overview?project={your gcp id} to enable this API.

Testing Connection

Use TEST CONNECTION button to check whether credentials are correctly configured and If SUCCESS, click CREATE AND USE CONNECTION.