Google Cloud Storage Read Connector (Legacy)

Creating a Google Cloud Storage (GCS) Connector

Prerequisites:

• Access credentials
• Data location on Google Cloud Storage
• Data Schema (column names and column type)

Specify the Google Cloud Storage (GCS) path

Location

GCS Read Connectors have location settings comprised of:

• Bucket: The bucket name, such as ascend-io-sample-read-data.
• Pattern: The pattern used to identify eligible files:
  • Match: Matches the pattern precisely character-for-character.
  • Glob: Glob applies a pattern matching algorithm.
  • Regex: Regex applies a pattern matching algorithm.

Google Cloud Storage Credentials

Enter the JSON key for the service account which has write access to everything matching the GCS path. Refer to Google documentation for more details on GCS authentication.

Testing Connection

Use Test Connection to check whether all GCS permissions are correctly configured.

Parsers & Schema

Data formats currently available are: Avro, Grok, JSON, Parquet and XSV. However, you can create your own parser functions or define a UDP (User Defined Parser) to process a file format.

Schema information will automatically be fetched for JSON, Parquet and XSV files with a header row.