Fork me on GitHub

Teleport

Generating Short-Lived Host Certificates with Machine ID

Improve

Host certificates are generally created on Teleport for access to OpenSSH servers that can not otherwise join a Teleport cluster. While long-lived certificates may be applied for this purpose, short-lived and regularly rotated host certificates generated through Teleport help to provide a number of security benefits over their long-lived counterparts. Regular certificate rotation reduces risk by ensuring that any potentially stolen certificates are usable for a shorter period of time. Additionally, when coupled with Teleport's RBAC support and host certificate Principals and Predicates, you can apply limitations to both the process of creating host certificates, as well as the host certificates themselves.

In this guide, you will learn how to create host certificates through Machine ID for OpenSSH servers, allowing for the benefits of host certificates with OpenSSH nodes, and reducing risk by ensuring that short-lived certificates adhering to the principle of least privelege can be successfully applied to Teleport clusters.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 11.0.3.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 11.0.3, which you can download by visiting the customer portal.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.3.8. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.3.8 go1.19

    tsh version

    Teleport v10.3.8 go1.19

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 11.0.3

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 10.3.8

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

  • A Linux based host that can support Machine ID. This "Machine ID Host" is the host that will be used to create Host Certificates using tbot. This host should additionally have OpenSSH server sshd version 6.9 or above running. The SSH port on this host must be open to traffic from the Teleport Proxy Service host.

    ssh -V

Step 1/5. Download and install Teleport 11.0.3

Download Teleport's PGP public key

sudo curl https://apt.releases.teleport.dev/gpg \ -o /usr/share/keyrings/teleport-archive-keyring.asc

Source variables about OS version

source /etc/os-release

Add the Teleport APT repository for v11. You'll need to update this

file for each major release of Teleport.

Note: if using a fork of Debian or Ubuntu you may need to use '$ID_LIKE'

and the codename your distro was forked from instead of '$ID' and '$VERSION_CODENAME'.

Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L42-L67

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \ https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v11" \ | sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport

Source variables about OS version

source /etc/os-release

Add the Teleport YUM repository for v11. You'll need to update this

file for each major release of Teleport.

Note: if using a fork of RHEL/CentOS or Amazon Linux you may need to use '$ID_LIKE'

and the codename your distro was forked from instead of '$ID'

Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L133-L153

sudo yum-config-manager --add-repo $(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v11/teleport.repo")
sudo yum install teleport

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

Optional: Using DNF on newer distributions

$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo

$ sudo dnf install teleport

curl https://get.gravitational.com/teleport-v11.0.3-linux-amd64-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v11.0.3-linux-amd64-bin.tar.gz
shasum -a 256 teleport-v11.0.3-linux-amd64-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v11.0.3-linux-amd64-bin.tar.gz
cd teleport
sudo ./install
curl https://get.gravitational.com/teleport-v11.0.3-linux-arm-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v11.0.3-linux-arm-bin.tar.gz
shasum -a 256 teleport-v11.0.3-linux-arm-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v11.0.3-linux-arm-bin.tar.gz
cd teleport
sudo ./install
curl https://get.gravitational.com/teleport-v11.0.3-linux-arm64-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v11.0.3-linux-arm64-bin.tar.gz
shasum -a 256 teleport-v11.0.3-linux-arm64-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v11.0.3-linux-arm64-bin.tar.gz
cd teleport
sudo ./install

Using this APT repo may result in breaking upgrades upon "apt upgrade" as all major versions will be

published under the same component. We recommend following the instructions in the

"Debian/Ubuntu (DEB)" tab instead.

Download Teleport's PGP public key

sudo curl https://deb.releases.teleport.dev/teleport-pubkey.asc \ -o /usr/share/keyrings/teleport-archive-keyring.asc

Add the Teleport APT repository

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] https://deb.releases.teleport.dev/ stable main" \ | sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport
sudo yum-config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo
sudo yum install teleport

Optional: Using DNF on newer distributions

$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo

$ sudo dnf install teleport

Next, log in to your cluster from your local machine using tsh to ensure that your device will be able to interact with the cluster:

tsh login --proxy=myteleportcluster.com --user=teleport-username

Step 2/5. Create a role to issue tbot certificates

In order to create certificates tbot requires a role that provides the necessary permissions. When we apply a role to a tbot user in this case, we restrict a bot user to only create host certificates.

Teleport roles can additionally limit the creation of host certificates to a desired principal. For example, normally when creating an SSH host certificate a principal represents a user or host that the certificate is explicitly applied to.

By checking that a domain name matches the principal on a host certificate in the example configuration we use in this guide, access can be limited only to the desired server or servers. On Teleport, a predicate should be applied to a principal to further define the scope of permissions applied to the principal.

By using the format of where: 'predicate(host_cert.principals, "domain.name")', a Teleport user can create a role that will only be able to able to create host certificates with the designated principal. A list of supported predicates, their behavior, and example configurations are listed below:

PredicateBehaviorExample
all_equalAll requested principals must be equal to the set string. If the set string is foo.example.com, host certificates limit access only to foo.example.com.all_equal(host_cert.principals, "foo.example.com")
all_end_withAll requested principals must end with the set string. If the set string is .example.com, host certificates limit access only to *.example.com.all_end_with(host_cert.principals, ".example.com")
is_subsetAll requested principals must be contained in the listed set.is_subset(host_cert.principals, "foo.example.com", "bar.example.com")

In the case of an example role for use within this guide, use a text editor of your choice from your local administrative device with tsh access to create a role YAML named tbotrole.yaml with the following configuration options, replacing nodename.my.domain.com with the domain name associated with your OpenSSH server:

kind: role
metadata:
  name: hostcert-bot
version: v5
spec:
  allow:
    rules:
      - resources:
          - host_cert
        verbs:
          - create
        where: 'is_subset(host_cert.principals, "nodename.my.domain.com")'

Once complete, create your role to apply to the tbot configuration in the next step:

tctl create tbotrole.yaml

Step 3/5 Create Your tbot Configuration

Before you create a bot user, you need to determine which role(s) you want to assign to it. You can use the tctl command below from your local machine with tsh access to examine what roles exist on your system:

tctl get roles --format=text

You will see something like the output below on a fresh install of Teleport with the preset roles and the addition of the hostcert-bot role created previously—your own cluster may have different roles. In this example, you want to give the bot the hostcert-bot role to allow the bot to generate host certificates.

Role    Allowed to login as                           Node Labels Access to resources
------- --------------------------------------------- ----------- ----------------------------------------
access  {{internal.logins}}                           <all nodes> event:list,read,session:read,list
auditor no-login-6566121f-b602-47f1-a118-c9c618ee5aec             session:list,read,event:list,read
editor
hostcert-bot                                                      host_cert:create                               

Your Machine ID host can now join the Teleport Cluster using a number of supported methods. This guide will highlight both the token and the IAM method which uses identity tokens bound to an AWS account. Note that the following commands may be entered on any device with tctl access to the cluster.

While any role with permissions for creating host certificates will work, this guide will make use of the hostcert-bot role created previously to ensure that the configuration adheres to the principle of least privilege.

tctl bots add robot --roles=hostcert-bot

First, create an IAM method token that specifies the AWS account from which the bot can join. Create the below file as iam-token.yaml:

kind: token
version: v2
metadata:
  # The token name is not a secret because instances must prove that they are
  # running in your AWS account to use this token.
  name: iam-token
  # Set a long expiry time for how long you want to support IAM method for
  # joining. It is safe to set this value to a very long time.
  expires: "3000-01-01T00:00:00Z"
spec:
  # Only allow bots to join using this token.
  roles: [Bot]

  # Set the join method to be IAM.
  join_method: iam

  # Define the name of the bot that will be allowed to use this token.
  bot_name: robot

  allow:
  # Restrict the AWS account and (optionally) ARN that can use this token.
  # This information can be obtained from running the
  # "aws sts get-caller-identity" command from the CLI.
  - aws_account: "111111111111"
    aws_arn: "arn:aws:sts::111111111111:assumed-role/teleport-bot-role/i-*"

With the iam-token-yaml file created, enter the following command:

$ tctl create -f iam-token.yaml

Next, create the bot user.

$ tctl bots add robot --token=iam-token --roles=hostcert-bot

Generating a configuration file

Machine ID may now be configured using the tbot configure command on the Machine ID host.

First, create a basic configuration file using the following parameters:

tbot configure \ --data-dir=/var/lib/teleport/bot \ --token=abcd123-insecure-do-not-use-this \ --join-method=token \ --certificate-ttl=1h0m0s \ --ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \ --auth-server=auth.example.com:3025
tbot configure \ --data-dir=/var/lib/teleport/bot \ --token=iam-token \ --join-method=iam \ --certificate-ttl=1h0m0s \ --ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \ --auth-server=auth.example.com:3025

Replace the following fields with values from your own cluster:

  • token is the token output by the tctl bots add command or the name of your IAM method token.
  • ca-pin is the CA Pin for your Teleport cluster, and is output by the tctl bots add command.
  • data-dir is where Machine ID writes its private data, including its own short-lived renewable certificates. These should not be used by applications and tools.
  • auth-server is the address of your Teleport Cloud Tenant, for example example.teleport.sh:443.
  • certificate-ttl is the option that will determine the time until expiration for any tbot issued certificates, including host certificates. The default TTL is for one hour, meaning that all generated certificates will expire after one hour's time.
  • token is the token output by the tctl bots add command or the name of your IAM method token.
  • ca-pin is the CA Pin for your Teleport cluster, and is output by the tctl bots add command.
  • data-dir is where Machine ID writes its private data, including its own short-lived renewable certificates. These should not be used by applications and tools.
  • certificate-ttl is the option that will determine the time until expiration for any tbot issued certificates, including host certificates. The default TTL is for one hour, meaning that all generated certificates will expire after one hour's time.
  • auth-server is the address of your Teleport Auth Server, for example auth.example.com:3025. If your Machine ID host is in a different network than your Auth Server, use the public web address of your Proxy Service instead (e.g., auth.example.com:443).

Using a text editor of your choice, write the output generated by the tbot configure command to a configuration YAML file. In this example we'll be using tbot.yaml.

Setting the destination directory

Once complete, the Machine ID host configuration will still require a destination directory to be manually referenced within the configuration file. The destination directory is where Machine ID will write certificates which can be used by application and tools to start Machine ID. In the configuration yaml tbot.yaml, this configuration will additionally be used to designate the principal, in this case the domain name that we'll be applying to the tbot configuration to access a Node. The new configuration YAML will require a destination field and ssh_host_cert subfield that resembles the following, replacing nodename.my.domain.com with your own required domain name:

destinations:
  - directory:
      path: /opt/machine-id
    configs:
      - ssh_host_cert:
          principals: [nodename.my.domain.com]

Once completed, your tbot.yaml configuration should resemble the following:

onboarding:
  token: "1234abcd5678efgh9"
  ca_path: ""
  ca_pins:
  - sha256:1234abcd5678efgh910ijklmnop
  join_method: token
storage:
  directory:
    path: /var/lib/teleport/bot
    symlinks: secure
    acls: try
destinations:
  - directory:
      path: /opt/machine-id
    configs:
      - ssh_host_cert:
          principals: [nodename.my.domain.com]
debug: false
auth_server: example.teleport.sh:443
certificate_ttl: 1h0m0s
renewal_interval: 20m0s
oneshot: false

Note the renewal_interval field, which will determine the cadence in which new certificates will be generated. Machine ID uses a default of 20 minutes for the renewal_interval, however this field may be adjusted as needed.

Step 4/5 Start Machine ID and create a host certificate

Using your configuration file, the tbot start command will start running Machine ID on your Machine ID host in a loop, writing renewable certificates to /var/lib/teleport/bot according to your configuration, and the short-lived certificates including the host certificate your bot will use to /opt/machine-id.

In a production environment you will want to run Machine ID in the background using a service manager like systemd. However, in this guide you will run it in the foreground to better understand how it works.

tbot start -c tbot.yaml

Now that Machine ID has successfully started, let's investigate the /opt/machine-id directory to see what was written to disk.

ls -1 /opt/machine-id

identity

key

key-cert.pub

key.pub

known_hosts

ssh_config

ssh_host

ssh_host-cert.pub

ssh_host-user-ca.pub

teleport-database-ca.crt

teleport-host-ca.crt

teleport-user-ca.crt

tlscert

Included in the output is the host certificate, private key, and user CA file Machine ID will have generated. The ssh_host-cert.pub file is the public host certificate, the ssh_host file is a private key file bound to the certificate, and the ssh_host-user-ca.pub file is the public user CA file. To confirm that the outputted files have been created correctly, view the public host certificate's contents by entering the following command:

sudo ssh-keygen -L -f /opt/machine-id/key-cert.pub

The contents of the public certificate will appear, and will contain a principals field which will contain the nodename.my.domain.com domain entered in a previous step. Your output will resemble the following:

        Type: [email protected] host certificate
        Public key: RSA-CERT SHA256:abcde123efghijk567lmnop
        Signing CA: RSA SHA256:abcde123efghijk567lmnop (using rsa-sha2-512)
        Key ID: ""
        Serial: 0
        Valid: from 2022-09-30T13:16:36 to 2022-09-30T17:17:36
        Principals:
                nodename.my.domain.com
        Critical Options: (none)
        Extensions:
                x-teleport-authority UNKNOWN OPTION (len 55)
                x-teleport-role UNKNOWN OPTION (len 8)

Step 5/5 Configuring SSHD and connecting to the Teleport Cluster

In order to ensure that the Machine ID host is able to authorize and authenticate itself to the cluster using sshd, sshd must now be configured to use the generated certificate files.

Using a text editor of your choice, open the sshd configuration file, usually found at /etc/ssh/sshd_config, and add the following lines:

## Path to the private Key File affiliated with the host certificate.
HostKey /opt/machine-id/ssh_host

## Ensures that the OpenSSH server is offering the generated public host certificate when connections are attempted.
HostCertificate /opt/machine-id/ssh_host-cert.pub

## Ensures that the server will trust certificates signed by the remote CA
TrustedUserCAKeys /opt/machine-id/ssh_host-user-ca.pub

Once saved, restart sshd to apply the new configuration:

systemctl restart sshd

The Machine ID host is now fully configured to accept connections from a tsh user. To confirm this, enter the following command from your local machine logged into the cluster using tsh:

The connection should successfully complete without the need to manually enter any additional credentials.

Next steps

Now that host certificates have been created with Teleport, they can be applied to OpenSSH configurations as needed. For more information regarding the next steps to connect to OpenSSH with Teleport, see the following documentation: