Fork me on GitHub

Teleport

Using Teleport with OpenSSH

Improve
Using Teleport with OpenSSH

Using Teleport with OpenSSH

Length: 16:04

In this guide, we will show you how to configure the OpenSSH server sshd to join a Teleport cluster. Existing fleets of OpenSSH servers can be configured to accept SSH certificates dynamically issued by a Teleport CA.

Using Teleport and OpenSSH has the advantage of getting you up and running, but in the long run, we would recommend replacing sshd with teleport. We've outlined these reasons in OpenSSH vs Teleport SSH for Servers?

Prerequisites

  • OpenSSH version 6.9 or above on your local machine. View your OpenSSH version with the command:

    ssh -V
  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 10.2.2.

    tctl version

    Teleport v10.2.2 go1.18

    tsh version

    Teleport v10.2.2 go1.18

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 10.2.2, which you can download by visiting the customer portal.

    tctl version

    Teleport v10.2.2 go1.18

    tsh version

    Teleport v10.2.2 go1.18

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.1.9. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.1.9 go1.18

    tsh version

    Teleport v10.1.9 go1.18

  • A Linux host with the OpenSSH server sshd installed, but not Teleport. The SSH port on this host must be open to traffic from the Teleport Proxy Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 10.2.2

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 10.1.9

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Step 1/4. Configure sshd to trust the Teleport CA

Later in this guide, we will generate an SSH client configuration that will use a certificate signed by the Teleport Auth Service to authenticate to your SSH server. For this to work, sshd must be told to allow users to log in with certificates generated by the Teleport Auth Service.

Start by exporting the Teleport CA public key.

On your local machine, print the Teleport certificate authority certificate to stdout:

tctl auth export --type=user | sed "s/cert-authority\ //"

Copy the output.

On the host where you are running sshd, run the following commands.

Assign the output of the tctl auth export command to an environment variable:

export KEY="<pasted output>"

Make the public key accessible to sshd:

sudo bash -c "echo \"$KEY\" > /etc/ssh/teleport_user_ca.pub"
sudo bash -c "echo 'TrustedUserCAKeys /etc/ssh/teleport_user_ca.pub' >> /etc/ssh/sshd_config"

Restart sshd. For systemd-enabled hosts, run the following command:

sudo systemctl restart sshd

Now, sshd will trust users who present a Teleport-issued certificate.

Step 2/4. Configure host authentication

Next, ask Teleport to issue a valid host certificate for your sshd host. Later in this guide, we will configure your SSH client to trust the certificate, authenticating your sshd host for your SSH client. Like the user certificate we created earlier, the host certificate will be signed by the Teleport Auth Service.

Ensure that your user has the correct privileges

Your user must be authorized to read and write host certificates.

On your local machine, create a file called host-certifier.yaml with the following content:

kind: role
version: v5
metadata:
  name: host-certifier
spec:
  allow:
    rules:
      - resources:
          - host_cert
        verbs:
          - list
          - create
          - read
          - update
          - delete

Create the role resource:

tctl create host-certifier.yaml

role 'host-certifier' has been created

Fetch the configuration for your user. Replace USERNAME with the name of the Teleport username that you used to log in to your cluster.

tctl get user/USERNAME > myuser.yaml

Make the following change to myuser.yaml:

  roles:
   - access
   - auditor
   - editor
+  - host-certifier

Apply your change:

tctl create -f myuser.yaml

Log out of your Teleport cluster and log in again.

You will now have the required permissions to export a host key for your sshd host.

Issue a host certificate

On your local machine, assign the IP address or fully qualified domain name of your Node to an environment variable.

ADDR=203.0.113.0

Run the following tctl command to generate a host certificate:

tctl auth sign \ --host=${ADDR?} \ --format=openssh \ --out=myhost

The credentials have been written to myhost, myhost-cert.pub

The above command will result in a private key and certificate.

To generate certificates for multiple hosts, assign the host flag to a comma-separated list of addresses. Certificates for wildcard domains are not supported by OpenSSH, so each domain must be fully qualified.

Use ssh-keygen to verify the contents of the certificate:

ssh-keygen -L -f myhost-cert.pub

The Principals section should contain the address you assigned to ADDR earlier:

myhost-cert.pub:
        Type: [email protected] host certificate
        Public key: RSA-CERT SHA256:nHkp6SnrAW4AV00VUaqPgR6SgdyvV9MmjUrYnwZ779A
        Signing CA: RSA SHA256:euqx2Y8Pq+r0c94GKVNXAklBVTmAJtaQUn3/ehrfEJE (using rsa-sha2-512)
        Key ID: ""
        Serial: 0
        Valid: after 2022-04-22T11:14:16
        Principals: 
                203.0.113.0
        Critical Options: (none)
        Extensions: 
                x-teleport-authority UNKNOWN OPTION (len 33)
                x-teleport-role UNKNOWN OPTION (len 8)

Copy the host key and certificate to your sshd host, placing them in the directory /etc/ssh.

Make sure these files have the correct permissions:

sudo chmod 0600 /etc/ssh/myhost
sudo chmod 0600 /etc/ssh/myhost-cert.pub

Then add the following lines to /etc/ssh/sshd_config on your sshd host:

HostKey /etc/ssh/myhost
HostCertificate /etc/ssh/myhost-cert.pub

Restart sshd.

Step 3/4. Generate an SSH client configuration

The next step is to configure your OpenSSH client to connect to your sshd host using credentials managed by Teleport. This configuration will use the SSH agent and your user's Teleport-issued certificate to authenticate to the sshd host. It will also authenticate the sshd host using the host certificate you generated earlier.

First, make sure you are running OpenSSH's ssh-agent and have logged in to your Teleport cluster:

tsh status

> Profile URL: https://teleport.example.com:443

Logged in as: myuser

Cluster: teleport.example.com

Roles: access, auditor, editor, host-certifier

Logins: ubuntu, root

Kubernetes: enabled

Valid until: 2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s]

Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty

eval `ssh-agent`

Agent pid 5931

The ssh-agent command prints additional commands to export the SSH_AUTH_SOCK and SSH_AGENT_PID environment variables. These variables allow OpenSSH clients to find the SSH agent. Running ssh-agent with eval executes these commands.

On your local machine, run the following tsh command. This will print a configuration block that tells your SSH client to use credentials managed by Teleport to connect to hosts in your cluster.

tsh config > ssh_config_teleport

This command creates an SSH configuration file at a nonstandard location in order to make it easier to clean up, but you can append the output of tsh config to the default SSH config file (~/.ssh/config) if you wish.

Teleport implements an SSH server that includes several subsystems, or predefined commands that are run when the server handles a connection. The Proxy Service implements a proxy subsystem that forwards SSH traffic to remote hosts and Trusted Clusters.

Here is a brief explanation of the configuration that tsh config generates:

# Common flags for all {{ .ClusterName }} hosts
Host *.{{ .ClusterName }} {{ .ProxyHost }}
    UserKnownHostsFile "{{ .KnownHostsPath }}"
    IdentityFile "{{ .IdentityFilePath }}"
    CertificateFile "{{ .CertificateFilePath }}"

If the host you are sshing into belongs to your Teleport cluster (i.e., its address is a subdomain of your cluster's domain), use a Teleport-managed known hosts file, private key, and certificate that are stored in the .tsh directory.

# Flags for all {{ .ClusterName }} hosts except the proxy
Host *.{{ .ClusterName }} !{{ .ProxyHost }}
    Port 3022
    ProxyCommand "{{ .TSHPath }}" proxy ssh --cluster={{ .ClusterName }} --proxy={{ .ProxyHost }} %[email protected]%h:%p

If the host that you are sshing into belongs to your Teleport cluster, the OpenSSH client will first execute a command, the ProxyCommand, that establishes an SSH connection to the Proxy Service. This command, tsh proxy ssh, requests the proxy subsystem in order to forward SSH traffic through the Proxy Service to your chosen host (including a host in a Trusted Cluster).

The tsh proxy ssh command requests the proxy subsystem through a command similar to the following, which assumes you are logging in to a Node called mynode as root with a cluster called teleport.example.com:

/usr/bin/ssh -l root -A -o UserKnownHostsFile=/root/.tsh/known_hosts -p 11105 teleport.example.com -s proxy:mynode:[email protected]

Notice that the known_hosts file used by the command is managed by tsh. Since the sshd host's information is listed in this file, your SSH client can authenticate the host via the certificate we generated earlier.

If using PowerShell on Windows, note that normal shell redirection may write the file with the incorrect encoding. To ensure it's written properly, try the following:

tsh.exe config | out-file .ssh\config -encoding utf8 -append

Proxy Jump

In the generated OpenSSH client configuration, the ProxyCommand for each leaf cluster connects through the root cluster's Proxy Service. In scenarios where leaf cluster Proxy Services are reachable by SSH client, you might prefer to connect directly through the leaf proxies for lower latency.

To enable direct connections to a Proxy Service in a leaf cluster, open the SSH configuration file you generated earlier and update the ProxyCommand of the leaf cluster's configuration block to use the leaf Proxy Service as a jumphost, using the -J flag.

Host *.{{ .NodeName }}.leaf1.example.com
   Port 3022
   ProxyCommand tsh proxy ssh -J proxy.leaf1.example.com:443 %[email protected]%h:%p

Proxy Templates

With Proxy Templates, tsh will dynamically determine the address of the Proxy Service to connect to based on the address of the destination host in your ssh command.

To use Proxy Templates, add -J {{proxy}} to the ProxyCommand line in your ~/.ssh/config.

Host *.example.com
   Port 3022
   ProxyCommand tsh proxy ssh -J {{proxy}} %[email protected]%h:%p

Then, add proxy_templates to your tsh configuration file (~/.tsh/config/config.yaml or a global /etc/tsh.yaml).

proxy_templates:
- template: '^(\w+)\.(leaf1\.example\.com):([0-9]+)$'
  proxy: "$2:443"

tsh proxy ssh -J {{proxy}} will attempt to match the host server address %h:%p with the configured templates. If there is a match, then the jump proxy address {{proxy}} will be replaced using the template's proxy field and the host server address %h:%p will be replaced using the template's host field if set.

FieldDescription
template(Required) Regular expression that the host server address %h:%p is matched against.
proxy(Required) Proxy Service address to use for proxy jump. Can reference capturing groups from the regular expression in template (e.g., $1 or $2).
host(Optional) Host Server address to connect to. Can reference capturing groups from the regular expression in template (e.g., $1 or $2). Defaults to full host spec %h:%p.

Example configuration

proxy_templates:
- template: '^(\w+)\.(leaf1\.example\.com):([0-9]+)$'
  proxy: "$2:443"
- template: '^(\w+)\.(leaf2\.example\.com):([0-9]+)$'
  proxy: "$2:3080"
  host: "$1:$3"
- template: '(\w+(\.\w+)*)\.(example\.com):([0-9]+)$'
  proxy: "leaf1.example.com:443"
  host: "$1:22"

Given the configuration above, the following command will connect to the Node
`node-1.leaf1.example.com:3022` through the Proxy Service `leaf1.example.com:443`:

```code
$ ssh [email protected]

The following command will connect to the Node node-1:3022 through the Proxy Service leaf2.example.com:3080:

The last template can be used to connect to openssh hosts with their own FQDN. For example, you can connect to the host openssh.external.com:22 through the Proxy Service leaf1.example.com:443 with the following command:

Multiple Clusters

If you switch between multiple Teleport Proxy Servers, you'll need to re-run tsh config for each to generate the cluster-specific configuration.

Similarly, if Trusted Clusters are added or removed, be sure to re-run tsh config and replace the previous configuration.

Step 4/4. Connect to your sshd host

Once you have appended the new text to your OpenSSH client configuration file, you can log in to your sshd host using the configuration we generated earlier.

First, define environment variables for the address of your Teleport cluster, the username you will use to log in to your sshd host, and the port on your sshd host you are using for SSH traffic:

See the available logins you can use to access your sshd host

tsh status | grep Logins

Logins: ubuntu, root

USER=ubuntu
CLUSTER=teleport.example.com
PORT=22

Next, SSH in to your remote host:

ssh -p ${PORT?} -F ssh_config_teleport "${USER?}@${ADDR?}.${CLUSTER?}"

This will connect to the node node1 on your Teleport cluster. This name does not need to be resolvable via DNS as the connection will be routed through your Teleport Proxy Service.

By default, the OpenSSH client configuration generated by tsh config directs the Teleport Proxy Service to dial port 3022 of a Node in your Teleport cluster. This works if the Node's SSH Service is listening on port 3022, and means that you can connect to the Teleport SSH Service via your OpenSSH client.

When you join a Teleport Node to a cluster, the Node creates a reverse tunnel to the cluster's Proxy Service. When you run an ssh command to access a host in your Teleport cluster using the configuration we generated, the Teleport Proxy Service will attempt to connect to the host via this reverse tunnel and, if that fails, try directly dialing the address.

In our case, the sshd host is not running Teleport, so no reverse tunnel will exist. Instead, the Proxy Service will establish a direct connection on the host's SSH port.

You can log in to a host in a Trusted Cluster by placing the name of the cluster between the name of the Node and the name of your root Teleport cluster:

ssh -F ssh_config_teleport ${USER?}@node2.leafcluster.${CLUSTER}
Note

Teleport uses OpenSSH certificates instead of keys. When you connect to a remote host, OpenSSH verifies that the address of the host is listed under the Principals section of the OpenSSH certificate. Usually, this is a fully qualified domain name, rather than an IP address.

Revoke an SSH certificate

To revoke the current Teleport CA and generate a new one, run tctl auth rotate. Unless you've highly automated your infrastructure, we would suggest you proceed with caution as this will invalidate the user and host CAs, meaning that the new CAs will need to be exported to every OpenSSH-based machine again using tctl auth export as above.