Simplifying Zero Trust Security for AWS with Teleport
Jan 23
Virtual
Register Now
Support
LOG IN
Teleport logoTry For Free
Contact SalesTry for Free
Support
LOG IN
Fork me on GitHub

Teleport

Using Teleport with OpenSSH in agentless mode (manual installation)

Version 16.x
Using Teleport with OpenSSH

Using Teleport with OpenSSH

Length: 16:04

In this guide, we will show you how to configure the OpenSSH server sshd to join a Teleport cluster. Existing fleets of OpenSSH servers can be configured to accept SSH certificates dynamically issued by a Teleport CA.

Using Teleport and OpenSSH has the advantage of getting you up and running, but in the long run, we would recommend replacing sshd with teleport. teleport SSH servers have support for multiple features that are incompatible with OpenSSH:

Teleport supports OpenSSH by proxying SSH connections through the Proxy Service. When a Teleport user requests to connect to an OpenSSH node, the Proxy Service checks the user's Teleport roles.

If the RBAC checks succeed, the Proxy Service authenticates to the OpenSSH node with a dynamically generated certificate signed by a Teleport CA. This allows the Proxy Service to record and audit connections to OpenSSH nodes.

The Proxy Service prevents Teleport users from bypassing auditing by requiring a certificate signed by a Teleport CA that only the Auth Service possesses.

In this setup, the Teleport SSH Service performs RBAC checks as well as audits and records sessions on its host, which eliminates the need for connection termination when recording SSH sessions.

Note

This guide shows you how to register an OpenSSH node by creating a node resource and configuring OpenSSH to trust the Teleport CA. If you can copy the teleport binary onto your OpenSSH node and execute it however, you can follow the standard registration guide instead, which has fewer steps. Teleport is able to perform many of the steps we show in this guide automatically.

Prerequisites

  • OpenSSH version 6.9 or above on your local machine. View your OpenSSH version with the command:

    ssh -V

  • A running Teleport cluster version 16.4.12 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl admin tool and tsh client tool.

    Visit Installation for instructions on downloading tctl and tsh.

  • A Linux host with the OpenSSH server sshd version 7.4 or above installed, but not Teleport. The SSH port on this host must be open to traffic from the Teleport Proxy Service host.
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example: 
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status
    Cluster  teleport.example.com
    Version  16.4.12
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Upgrading to v14 from legacy OpenSSH nodes

If you have previously configured OpenSSH nodes to trust a Teleport CA without registering them and you upgrade your Teleport cluster to Teleport 14, you won't be able to connect to them anymore by default. This is because open dialing to OpenSSH servers not registered with the cluster is no longer allowed in Teleport 14. To ensure that you will retain access to your OpenSSH nodes you will need to follow this guide to register every OpenSSH node with Teleport that you previously configured. This must be done before your Teleport cluster is upgraded to Teleport 14.

If you are having issues registering OpenSSH nodes or need to upgrade your Teleport cluster to Teleport 14 before registering all of your OpenSSH nodes, you can pass the TELEPORT_UNSTABLE_UNLISTED_AGENT_DIALING environment variable to your Proxy Service and set it to yes. This will allow connections to unregistered OpenSSH nodes but will be removed in Teleport v15.

Step 1/5. Add a node resource to your Teleport cluster

When you request an SSH connection to a OpenSSH node, Teleport needs to be able to find the node's IP address so it can establish a connection to it.

Declare a node resource so Teleport knows how to reach your OpenSSH server. On your local machine, create a file called openssh-node-resource.yaml with the following content:

kind: node
version: v2
sub_kind: openssh
metadata:
  name: a100fdd0-52db-4eca-a7ab-c3afa7a1564a
  labels:
    env: prod
spec:
  addr: 1.2.3.4:22
  hostname: openssh-node

spec.addr and spec.hostname are mandatory. Assign spec.addr to the address and port of your node and spec.hostname to the name of the node as you would like users to see it in Teleport.

The metadata.labels field labels the SSH Service instance so you can apply RBAC rules to it.

The metadata.name field isn't mandatory, but setting it here will save you some work later.

To generate a new universal unique identifier (UUID) suitable for a node name, use the uuidgen on Linux or macOS, or use the New-Guid cmdlet in Powershell on Windows.

Create the node resource:

tctl create openssh-node-resource.yaml
Note

This step can be done with Infrastructure-as-Code (IaC) tools (tctl, Terraform, or Kubernetes Operator). This is described in the OpenSSH server IaC guide.

Step 2/5. Configure sshd to trust the Teleport CA

Later in this guide, we will generate an SSH client configuration that will use a certificate signed by the Teleport Auth Service to authenticate to your SSH server. For this to work, sshd must be told to allow users to log in with certificates generated by the Teleport Auth Service.

Start by exporting the Teleport CA public key.

On the host where you are running sshd, run the following commands, assigning proxy to the address of your Teleport Proxy Service:

export KEY=$(curl 'https://proxy/webapi/auth/export?type=openssh' | sed "s/cert-authority\ //")

Make the public key accessible to sshd:

sudo bash -c "echo \"$KEY\" > /etc/ssh/teleport_openssh_ca.pub"
sudo bash -c "echo 'TrustedUserCAKeys /etc/ssh/teleport_openssh_ca.pub' >> /etc/ssh/sshd_config"

Restart sshd. For systemd-enabled hosts, run the following command:

sudo systemctl restart sshd

Now, sshd will trust users who present a Teleport-issued certificate.

Step 3/5. Configure host authentication

Next, ask Teleport to issue a valid host certificate for your sshd host. Later in this guide, we will configure your SSH client to trust the certificate, authenticating your sshd host for your SSH client. Like the user certificate we created earlier, the host certificate will be signed by the Teleport Auth Service.

Ensure that your user has the correct privileges

Your user must be authorized to read and write host certificates.

On your local machine, create a file called host-certifier.yaml with the following content:

kind: role
version: v5
metadata:
  name: host-certifier
spec:
  allow:
    rules:
      - resources:
          - host_cert
        verbs:
          - list
          - create
          - read
          - update
          - delete

Create the role resource:

tctl create host-certifier.yaml
role 'host-certifier' has been created

Assign the host-certifier role to your Teleport user by running the appropriate commands for your authentication provider:

  1. Retrieve your local user's roles as a comma-separated list:

    ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')

  2. Edit your local user to add the new role:

    tctl users update $(tsh status -f json | jq -r '.active.username') \  --set-roles "${ROLES?},host-certifier"

  3. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Open your github authentication connector in a text editor:

    tctl edit github/github

  2. Edit the github connector, adding host-certifier to the teams_to_roles section.

    The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.

    Here is an example:

      teams_to_roles:
    - organization: octocats
      team: admins
      roles:
        - access
+       - host-certifier

  3. Apply your changes by saving closing the file in your editor.

  4. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your saml configuration resource:

    tctl get --with-secrets saml/mysaml > saml.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the saml.yaml file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource.

  2. Edit saml.yaml, adding host-certifier to the attributes_to_roles section.

    The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

    Here is an example:

      attributes_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - host-certifier

  3. Apply your changes:

    tctl create -f saml.yaml

  4. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your oidc configuration resource:

    tctl get oidc/myoidc --with-secrets > oidc.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the oidc.yaml file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource.

  2. Edit oidc.yaml, adding host-certifier to the claims_to_roles section.

    The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

    Here is an example:

      claims_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - host-certifier

  3. Apply your changes:

    tctl create -f oidc.yaml

  4. Sign out of the Teleport cluster and sign in again to assume the new role.

You will now have the required permissions to export a host key for your sshd host.

Issue a host certificate

When you created a node resource and if you didn't set the metadata.name field earlier, the Teleport Auth Service generated a universal unique identifier (UUID) for that node. Teleport Proxy Services uses the UUID to differentiate nodes with the same hostname, so it must be added to the host certificate. To find your node's UUID, first determine if its hostname is unique:

tctl get node/openssh-node --format text

If only one node is displayed and you have jq installed, you can run the following command to get your node's UUID:

tctl get node/openssh-node --format=json | jq -r ".[0].metadata.name"

Otherwise, find your node's UUID in the metadata.name field of the YAML output of this command:

tctl get node/openssh-node

Create the host certificate

When creating host certificates, it is important to specify all the domain names and addresses that refer to your node. If you try to connect to a node with a name or address that was not specified when creating it's host certificate, Teleport will reject the SSH connection.

On your local machine, assign the IP address, fully qualified domain name of your node, and the node's UUID to an environment variable. If you won't be connecting to your node with its hostname, you can safely omit it.

ADDR=1.2.3.4,openssh-node,a100fdd0-52db-4eca-a7ab-c3afa7a1564a

Run the following tctl command to generate a host certificate:

tctl auth sign \      --host=${ADDR?} \      --format=openssh \      --out=myhost

The credentials have been written to myhost, myhost-cert.pub

The above command will result in a private key and certificate.

To generate certificates for multiple hosts, assign the host flag to a comma-separated list of addresses. Certificates for wildcard domains are not supported by OpenSSH, so each domain must be fully qualified.

Use ssh-keygen to verify the contents of the certificate:

ssh-keygen -L -f myhost-cert.pub

The Principals section should contain the address you assigned to ADDR earlier:

myhost-cert.pub:
        Type: [email protected] host certificate
        Public key: RSA-CERT SHA256:nHkp6SnrAW4AV00VUaqPgR6SgdyvV9MmjUrYnwZ779A
        Signing CA: RSA SHA256:euqx2Y8Pq+r0c94GKVNXAklBVTmAJtaQUn3/ehrfEJE (using rsa-sha2-512)
        Key ID: ""
        Serial: 0
        Valid: after 2022-04-22T11:14:16
        Principals:
                1.2.3.4
                openssh-node
                a100fdd0-52db-4eca-a7ab-c3afa7a1564a
        Critical Options: (none)
        Extensions:
                x-teleport-authority UNKNOWN OPTION (len 33)
                x-teleport-role UNKNOWN OPTION (len 8)

Copy the host key and certificate to your sshd host, placing them in the directory /etc/ssh.

Make sure these files have the correct permissions:

sudo chmod 0600 /etc/ssh/myhost
sudo chmod 0600 /etc/ssh/myhost-cert.pub

Then add the following lines to /etc/ssh/sshd_config on your sshd host:

HostKey /etc/ssh/myhost
HostCertificate /etc/ssh/myhost-cert.pub

Restart sshd.

Step 4/5. Generate an SSH client configuration

The next step is to configure your OpenSSH client to connect to your sshd host using credentials managed by Teleport. This configuration will use your user's Teleport-issued certificate to authenticate to the sshd host. It will also authenticate the sshd host using the host certificate you generated earlier.

First, make sure you have logged in to your Teleport cluster:

tsh status
> Profile URL:        https://teleport.example.com:443  Logged in as:       myuser  Cluster:            teleport.example.com  Roles:              access, auditor, editor, host-certifier  Logins:             ubuntu, root  Kubernetes:         enabled  Valid until:        2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s]  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty
tsh status
> Profile URL:        https://teleport.example.com:443  Logged in as:       myuser  Cluster:            teleport.example.com  Roles:              access, auditor, editor, reviewer, host-certifier  Logins:             ubuntu, root  Kubernetes:         enabled  Valid until:        2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s]  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty
tsh status
> Profile URL:        https://mytenant.teleport.sh:443  Logged in as:       myuser  Cluster:            mytenant.teleport.sh  Roles:              access, auditor, editor, reviewer, host-certifier  Logins:             ubuntu, root  Kubernetes:         enabled  Valid until:        2022-05-06 22:54:01 -0400 EDT [valid for 11h53m0s]  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty

On your local machine, run the following tsh command. This will print a configuration block that tells your SSH client to use credentials managed by Teleport to connect to hosts in your cluster.

tsh config > ssh_config_teleport

This command creates an SSH configuration file at a nonstandard location in order to make it easier to clean up, but you can append the output of tsh config to the default SSH config file (~/.ssh/config) if you wish.

Teleport implements an SSH server that includes several subsystems, or predefined commands that are run when the server handles a connection. The Proxy Service implements a proxy subsystem that forwards SSH traffic to remote hosts and trusted clusters.

Here is a brief explanation of the configuration that tsh config generates:

# Common flags for all {{ .ClusterName }} hosts
Host *.{{ .ClusterName }} {{ .ProxyHost }}
    UserKnownHostsFile "{{ .KnownHostsPath }}"
    IdentityFile "{{ .IdentityFilePath }}"
    CertificateFile "{{ .CertificateFilePath }}"

If the host you are sshing into belongs to your Teleport cluster (i.e., its address is a subdomain of your cluster's domain), use a Teleport-managed known hosts file, private key, and certificate that are stored in the .tsh directory.

# Flags for all {{ .ClusterName }} hosts except the proxy
Host *.{{ .ClusterName }} !{{ .ProxyHost }}
    Port 3022
    ProxyCommand "{{ .TSHPath }}" proxy ssh --cluster={{ .ClusterName }} --proxy={{ .ProxyHost }} %r@%h:%p

If the host that you are sshing into belongs to your Teleport cluster, the OpenSSH client will first execute a command, the ProxyCommand, that establishes an SSH connection to the Proxy Service. This command, tsh proxy ssh, requests the proxy subsystem in order to forward SSH traffic through the Proxy Service to your chosen host (including a host in a Trusted Cluster).

The tsh proxy ssh command requests the proxy subsystem through a command similar to the following, which assumes you are logging in to a node called mynode as root with a cluster called teleport.example.com:

/usr/bin/ssh -l root -A -o UserKnownHostsFile=/root/.tsh/known_hosts -p 11105 teleport.example.com -s proxy:mynode:[email protected]

Notice that the known_hosts file used by the command is managed by tsh. Since the sshd host's information is listed in this file, your SSH client can authenticate the host via the certificate we generated earlier.

If using PowerShell on Windows, note that normal shell redirection may write the file with the incorrect encoding. To ensure it's written properly, try the following:

tsh.exe config | out-file .ssh\config -encoding utf8 -append

Routing in Teleport clusters is case-sensitive by default, but OpenSSH always lowercases hostnames. If you are using an OpenSSH client and have hosts with uppercase letters in their hostnames, you may need to set case_insensitive_routing: true in either the auth_service block of your Teleport config, or in the cluster_networking_config resource.

Multiple Clusters

If you switch between multiple Teleport Proxy Servers, you'll need to re-run tsh config for each to generate the cluster-specific configuration.

Similarly, if trusted clusters are added or removed, be sure to re-run tsh config and replace the previous configuration.

Step 5/5. Connect to your sshd host

Once you have appended the new text to your OpenSSH client configuration file, you can log in to your sshd host using the configuration we generated earlier.

First, define environment variables for the address of your Teleport cluster, the username you will use to log in to your sshd host, and the port on your sshd host you are using for SSH traffic:

See the available logins you can use to access your sshd host
tsh status | grep Logins
Logins:             ubuntu, root
USER=ubuntu
CLUSTER=teleport.example.com
PORT=22
See the available logins you can use to access your sshd host
tsh status | grep Logins
Logins:             ubuntu, root
USER=ubuntu
CLUSTER=mytenant.teleport.sh
PORT=22

Next, SSH in to your remote host:

ADDR_NODE=openssh-node
ssh -p ${PORT?} -F ssh_config_teleport "${USER?}@${ADDR_NODE?}.${CLUSTER?}"

This name does not need to be resolvable via DNS as the connection will be routed through your Teleport Proxy Service.

By default, the OpenSSH client configuration generated by tsh config directs the Teleport Proxy Service to dial port 3022 of a node in your Teleport cluster. This works if the node's SSH Service is listening on port 3022, and means that you can connect to the Teleport SSH Service via your OpenSSH client.

When you join a Teleport node to a cluster, the node creates a reverse tunnel to the cluster's Proxy Service. When you run an ssh command to access a host in your Teleport cluster using the configuration we generated, the Teleport Proxy Service will attempt to connect to the host via this reverse tunnel and, if that fails, try directly dialing the address.

In our case, the sshd host is not running Teleport, so no reverse tunnel will exist. Instead, the Proxy Service will establish a direct connection on the host's SSH port.

You can log in to a host in a trusted leaf cluster by placing the name of the cluster between the name of the node and the name of your root cluster:

ssh -F ssh_config_teleport ${USER?}@node2.leafcluster.${CLUSTER}
Note

Teleport uses OpenSSH certificates instead of keys. When you connect to a remote host, OpenSSH verifies that the address of the host is listed under the Principals section of the OpenSSH certificate. Usually, this is a fully qualified domain name, rather than an IP address.