In this doc, we will cover how to setup a Connection to Kafka.

Prerequisites

  • Access credentials
  • Kafka topic, consumer configs

Create new Kafka Read Connector

2094

Figure 1

1284

Figure 2

  • Connection Name (required): The name to identify this connection with, such as Kafka Sample Read Connection.

  • Kafka Cluster Identifier (required): Must be unique across the full Ascend environment. It lets Ascend store the data read from a specific Kafka cluster (e.g., dev cluster) into the correct location in Ascend identified by this identifier.

  • Bootstrap Servers (required): This is a comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself. In other words, it provides the initial hosts that act as the starting point for a Kafka client to discover the full set of alive servers in the cluster. A host and port pair uses : as the separator as following examples:
    localhost:9092
    localhost:9092,another.host:9092

  • Consumer Group Id (optional): Consumers can join a consumer group by using group id. Ascend will assign a default one if this is not passed in.

  • Consumer Configs (optional): Open set of configs in Kaka property style to pass down to the consumer, such as security.protocol=SASL_SSL Learn More

Create/Edit Kafka Credentials

Create new credentials if Required Credentials checkbox was selected and you do not want to select from the already created ones. (Figure 3)

1328

Figure 3

The source SASL.JAAS.CONFIG config file in Kafka looks like the below from SASL Client Config

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="alice" \
    password="alice-secret";

🚧

Important: Format of the SASL.JAAS.CONFIG Credential

The sample given above shows how the sasl.jaas.config key/value pair looks in an actual Kafka config file.

When entering the credential into the Ascend SASL.JAAS.CONFIG credential entry box (Figure 3), you must enter it without the preceding sasl.jaas.config= text. Only enter what comes after that text as the actual value in the credential (seen as obfuscated dots in Figure 3).

If you fail to format the credential correctly, you will likely see error messages about a missing LoginModule or other similar errors when testing the connection.

See below for the correct format to paste into Ascend.

The credential should be pasted in this format into the SASL.JAAS.CONFIG input textbox (note the sasl.jaas.config= is omitted):

org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="alice" \
    password="alice-secret";

You can also choose to Edit an existing credentials by clicking the EDIT button next to each credentials. (Figure 4)

1120

Figure 4

Finally, use the TEST CONNECTION button to check if the connection is a SUCCESS* and then click CREATE AND USE CONNECTION**.

1238

Figure 5

πŸ“˜

If your Kafka is in a private subnet or VPC and not accessible from the public internet. Please contact [email protected] to set up networking.