Delta Lake

Prerequisites

  • Object Storage and Cloud provider credentials (e.g: S3 and AWS)
  • Table name (Stored in S3 or Azure)

Create new Delta Lake connection

Figure 1Figure 1

Figure 1

  • Access Type : The type of connection: Read-Only, Write-Only, Read-Write.
  • Connection Name: The name to identify this connection with, such as connection_delta_lake.
  • Storage System: The cloud object storage where Delta Lakes stores its tables. e.g Amazon S3.

If 'Amazon S3' is selected as Storage System:

  • Bucket: the name of the bucket used to store the data e.g: test-bucket-delta-lake
  • Connection Type: the connection type of bucket e.g: with Region, default: Standard (use the default configuration that is selected by AWS when a bucket is created)
    • S3 region: if 'with Region' is selected as Connection Type, this is the name of the region to be selected.
  • Register in AWS Glue Catalog: Selects whether to register the table into AWS Glue Catalog or not.
    • Glue Database: if 'Register in Catalog' is selected, this is the name of the Glue Database to be selected.
    • Glue Region: if 'Register in Catalog' is selected, this is the name of the Glue Region where you Glue database is located.
  • Requires Credentials: If checked, it allows passing credentials to access the object storage
  • Choose Credentials: Choose from existing credentials or create new credential for connecting to Delta Lake object storage if 'Requires Credentials' checkbox is selected. You can also choose to edit a credential with the EDIT button next to the existing credential.

If 'Azure Data Lake Storage' is selected as Storage System:

  • Storage Account Name: the name of the Storage Account used to store the data.
  • Container Name: the name of the Container in the previous selected Storage Account where to store the data.
  • Requires Credentials: If checked, it allows passing credentials to access the object storage
  • Choose Credentials: Choose from existing credentials or create new credential for connecting to Delta Lake object storage if 'Requires Credentials' checkbox is selected. You can also choose to edit a credential with the EDIT button next to the existing credential.

Use TEST CONNECTION button and check that the test shows ALL PASS.
Lastly, click CREATE.


Did this page help you?