Read Connector

After you have an Ascend Read Connection set up, then you continue to set up the Read Connector in your dataflow.

Create New Read Connector

Figure 1Figure 1

Figure 1

  • NAME: Read connector name to identify the specific data being collected.
  • DESCRIPTION: Additional information about the connector.
Figure 2Figure 2

Figure 2

  • BROWSE CONNECTION: Click this button to explore resource and locate assets to ingest. This will give you access to the Google dataset where you can navigate to different tables within it. Select the assets you want to ingest and press CONFIRM.
Figure 3Figure 3

Figure 3

  • DATASET: Dataset within a Google BigQuery project.

Let's look at the following table to learn about the configurations that are partition specific.

Partition Style

Description

Required Fields

None

No partition is used and the data resides in one large table.

TABLE NAME: Name used to identify the BigQuery table.

Partitioned Table

A table that is separated into sections such as Ingestion time or Date. Google identifies partitioning as a best practice over sharding.

TABLE NAME: Name used to identify the BigQuery table.

PARTITION PATTERN MATCHING: Select either Regex, Glob, or Match.

PARTITION PATTERN: Pattern used to identify the partition.

Sharded Table

Another way to sub section data. A subset of data can be pulled from multiple database tables based on a parameter such as Date.

TABLE NAME PATTERN MATCHING: Select either Regex, Glob, or Match.

TABLE NAME PATTERN: Pattern used to identify the table.

Additional information on Google BigQuery Partitioning

Generate Schema

Figure 4Figure 4

Figure 4

Once you click on the GENERATE SCHEMA button, the parser will create a schema and a data preview will be populated as in the image below.

  • Add schema column: Add a custom column to the generated schema.

Refresh Schedule

The refresh schedule specifies how often Ascend checks the data location to see if there's new data. Ascend will automatically kick off the corresponding big data jobs once new or updated data is discovered.

Processing Priority (optional)

  • Assigned Priority: Is used to determine which components to schedule first. Higher priority numbers are scheduled before lower ones. Increasing the priority on a component also causes all its upstream components to be prioritized higher. Negative priorities can be used to postpone work until excess capacity becomes available.

Did this page help you?