Reverse SSH

📘

This documentation only applies to Customer Hosted Ascend environments.

Description

Reverse SSH is a highly configurable, production-quality, SSH-based solution that customers can choose to implement if they have private/internal data sources (e.g. on-premise) that need to be accessed by their Ascend environment. This feature is available to all Ascend customers.

Reverse SSH relies on outgoing SSH connections from your network. This enables the Ascend cloud environment to make limited connections "in reverse" to your private/internal data sources over the established outgoing SSH tunnel.

📘

If you need to connect to more than two (2) data sources within your private/internal network, please contact Ascend for support.

Information Required

❗️

Warning: Do not share your private key

Share only your public SSH key with Ascend. Always maintain the security of your private SSH key and do not share your private key with anyone!

From Customer:

  • Your Public SSH Key File/String (required): share your public SSH key file/string with your Ascend support representative
  • Path to SSH Private Key file (required): the path to your locally-stored SSH _private _key file to use when establishing the SSH tunnel connection (e.g. ~/.ssh/id_rsa) - ensure the permissions on this file are set to "read/write only by owner" (for example, chmod 600 on Linux/Unix)
  • Private/Internal Hostname/IP (required): the hostname - FQDN recommended - or IP address of your private/internal data source, such as a database (e.g. mysql-database.my-internal-network.local)
  • Private/Internal Port Number (required): the port number of your private/internal data source, such as the database port number (e.g. MySQL default port is 3306)

From Ascend:

  • Ascend Customer Environment Name (required): the name of your Ascend environment (e.g. if the hostname of your environment is purple-squirrel.ascend.io, then the environment name is "purple-squirrel")
  • Ascend Remote SSH Port (required): 2222 (default, this can be modified by request)
  • Ascend Remote SSH username (required): ascend-io-ssh
  • Ascend SSH Server Host (Reverse Proxy) (required): ssh-reverse-proxy.default.svc.cluster.local
  • Ascend SSH Server Host Bind Address (required): 0.0.0.0
  • Ascend SSH Server Host Port (required): 15000 or 15001 (default, customizable on request, if necessary)

Example Configuration

From Customer:

  • Your Public SSH Key File/String: sample, not for production use
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINH67S9UninAHROCfUYkcgx+S76obkZJqiWRgpMuC8EX
  • Path to SSH Private Key file: ~/.ssh/id_rsa
  • Private/Internal Hostname/IP: mysql-database.my-internal-network.local
  • Private/Internal Port Number: 3306

From Ascend:

  • Ascend Customer Environment Name: purple-squirrel
  • Ascend Remote SSH Port: 2222
  • Ascend Remote SSH username: ascend-io-ssh
  • Ascend SSH Server Host (Reverse Proxy): ssh-reverse-proxy.default.svc.cluster.local
  • Ascend SSH Server Host Bind Address: 0.0.0.0
  • Ascend SSH Server Host Port: 15000

Setup Instructions

Step 1: Obtain your SSH key

The customer should first obtain (or create) their public SSH key.

There are multiple ways to create a public SSH key. Below is a sample method to generate an SSH key interactively using docker with a standard Linux container.

$ docker run --rm -it --entrypoint /keygen.sh linuxserver/openssh-server

Regarding choice of key algorithm and length - Ascend will generally support the recommendations of standards organizations such as NIST (for example, consider NIST SP 800-57 Part 3 Rev. 1). These recommendations and best practices evolve over time, so if you have a specific requirement, don't hesitate to discuss with your Ascend representative.

You might also already have standard tools available in your operating system, such as ssh-keygen, to generate SSH key pairs.

If you need help with this step, please consult with your network/security administrator.

Step 2: Share Public SSH key with Ascend

Provide your public SSH key (only) to your Ascend support representative. Ascend will configure the customer's Ascend environment to allow SSH connections to the platform using only the customer's public SSH key.

Step 3: Test SSH Connectivity

Run a connectivity test to establish the reverse SSH tunnel by running a command similar to the below, which starts an SSH connection in the foreground.

It is recommended to test the SSH connection from a machine on the same internal network that you will be using to establish on-going SSH connections to Ascend (when possible). This is to ensure that your internal firewalls and network routing are set up to enable these outgoing SSH connections.

The command below is formatted using arguments that are helpful for debugging/testing connectivity - in production, you may wish to modify some of the arguments to simplify the verbosity of logging, etc.

$ ssh  -o ServerAliveInterval=15 \
       -o ServerAliveCountMax=3 \
       -o TCPKeepAlive=true \
       -o UserKnownHostsFile=/dev/null \
       -o StrictHostKeyChecking=no \
       -N \
       -i </path/to/private/ssh_key_file> \
       -p 2222 \
       -R 0.0.0.0:<15000 or 15001>:<Internal Host/IP>:<Internal Port Number> \
       -vvv \
       ascend-io-ssh@<environment-name>.ssh.ascend.io

This will establish a connection to the Ascend SSH Server Host, which serves as a remote proxy in the Ascend cloud environment, to allow Ascend infrastructure to connect through the outgoing SSH connection (from your network) to your private/internal data sources.

📘

TIP: Multiple private/internal data sources can be accessed through a single SSH client connection (if desired). Just specify the "-R" flag and related configuration settings multiple times in the command, once for each pair of Ascend bind address/port and Internal Host/IP and Internal Port Number. For example, you can set up reverse SSH for two different private/internal databases via a single reverse SSH connection command.

When the test connection to Ascend's SSH Server Host (using the above command) is successful, you will typically see a set of debug messages similar to the below:

Step 4: Automate SSH Tunnel

Customers should implement measures to keep this reverse SSH tunnel open/connected automatically. SSH tunnels can be disconnected due to network interruptions and may not always recover automatically. This, in turn, can cause interruptions to your active Ascend data flows that use your internal/private data sources.

Consider whether you could use tools such as custom scripts, autossh, systemd, or other similar tools to ensure that the reverse SSH tunnel is kept alive and connected to the Ascend cloud environment.

It is considered a best practice (when possible) to create a service account, with its own public/private SSH key pair, that runs the SSH client which establishes the reverse SSH tunnel connections.

Set Up Ascend Connections

When you want to connect from Ascend to your private/internal data sources, you need to use the dedicated "Ascend SSH Server Host" documented below. Every customer environment has its own private Ascend SSH Server which acts as a reverse proxy. You may use this hostname in either Read Connections or Write Connectors, for example.

  • Ascend SSH Server Host (Reverse Proxy): ssh-reverse-proxy.default.svc.cluster.local
  • Ascend SSH Server Host Port: 15000 or 15001 (defaults) as described above in "Information Required" section.

Instead of using your local data source hostname/port for connections using the Reverse SSH tunnel, use the Ascend SSH Server Host and Ascend SSH Server Host Port for the connection's "Host" and "Port" fields (similar to below).

Configure Ascend Connection

Test Ascend Connection

We recommend that once you have entered the necessary connection information to use the Ascend SSH Server Host as a reverse SSH proxy, that you also then click the "Test Connection" box to confirm the setup is completed successfully. This runs a series of connection tests, and if all the connection tests pass as expected (similar to the screenshot below), you have successfully connected from Ascend to your private/internal data source using reverse SSH!

Common Errors/Issues

While there are many issues that can happen that impact connectivity over public networks, see below for a few common errors/issues and possible resolutions. Contact us if you need additional support!

IssueSample errorScenarioResolution
SSH connection refused trying to connect to port 2222 during connectivity testssh: connect to host .ssh.ascend.io port 2222: Connection refusedYou are trying to connect to Ascend's SSH reverse proxy and the connection fails or is refused.Check local network/firewall and/or contact Ascend for support
SSH tunnel is down/connection lostIn Ascend UI, in a component's "Errors and Warnings" you might see a Partition Error similar to: psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "ssh-reverse-proxy.default.svc.cluster.local" and accepting
TCP/IP connections on port 15000?

Stack trace ends with:
Spark job: spark-worker
(Operation was attempted 2 times)
A Data Flow fails where the Connector contains a similar error in the "Errors and Warnings" section.Check the status of your outgoing SSH tunnel - ensure it is running and connection is established to Ascend, and/or contact Ascend for support
Private/internal data source is down/unreachableIn Ascend UI, in a component's "Errors and Warnings" you might see a Partition Error containing the text: psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Stack trace ends with:
Spark job: spark-worker
A Data Flow fails where the Connector contains a similar error in the "Errors and Warnings" section.Check the system status of your private/internal data source, and/or network/firewall