Generating Short-Lived Host Certificates with Machine ID

Host certificates are generally created on Teleport for access to OpenSSH servers that can not otherwise join a Teleport cluster. While long-lived certificates may be applied for this purpose, short-lived and regularly rotated host certificates generated through Teleport help to provide a number of security benefits over their long-lived counterparts. Regular certificate rotation reduces risk by ensuring that any potentially stolen certificates are usable for a shorter period of time. Additionally, when coupled with Teleport's RBAC support and host certificate Principals and Predicates, you can apply limitations to both the process of creating host certificates, as well as the host certificates themselves.

In this guide, you will learn how to create host certificates through Machine ID for OpenSSH servers, allowing for the benefits of host certificates with OpenSSH nodes, and reducing risk by ensuring that short-lived certificates adhering to the principle of least privilege can be successfully applied to Teleport clusters.

Prerequisites

To check version information, run the tctl version and tsh version commands. For example:

tctl version
Teleport Enterprise v13.3.9 git:api/14.0.0-gd1e081e go1.21

tsh version
Teleport v13.3.9 go1.21
Proxy version: 13.3.9Proxy: teleport.example.com

To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands on your administrative workstation using your current credentials. For example:

tsh login --proxy=teleport.example.com --user=[email protected]
tctl status
Cluster  teleport.example.com
Version  14.0.0
CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

• A Linux based host that can support Machine ID. This "Machine ID Host" is the host that will be used to create Host Certificates using tbot. This host should additionally have OpenSSH server sshd version 6.9 or above running. The SSH port on this host must be open to traffic from the Teleport Proxy Service host.

  ssh -V

Step 1/5. Download and install Teleport

Select an edition, then follow the instructions for that edition to install Teleport.

curl https://goteleport.com/static/install.sh | bash -s 13.3.9

curl https://goteleport.com/static/install.sh | bash -s 14.0.0

Download Teleport's PGP public key
sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc
Source variables about OS version
source /etc/os-release
Add the Teleport APT repository for v14. You'll need to update this
file for each major release of Teleport.
echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v14" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport-ent

sudo apt-get install teleport-ent-fips

Source variables about OS version
source /etc/os-release
Add the Teleport YUM repository for v14. You'll need to update this
file for each major release of Teleport.
First, get the major version from $VERSION_ID so this fetches the correct
package version.
VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
sudo yum-config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v14/teleport.repo")"
sudo yum install teleport-ent
Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

sudo yum install teleport-ent-fips

Source variables about OS version
source /etc/os-release
Add the Teleport Zypper repository for v14. You'll need to update this
file for each major release of Teleport.
First, get the OS major version from $VERSION_ID so this fetches the correct
package version.
VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
Use zypper to add the teleport RPM repo
sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/cloud/teleport-zypper.repo")
sudo yum install teleport-ent
Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

sudo yum install teleport-ent-fips

Source variables about OS version
source /etc/os-release
Add the Teleport YUM repository for v14. You'll need to update this
file for each major release of Teleport.
First, get the major version from $VERSION_ID so this fetches the correct
package version.
VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
Use the dnf config manager plugin to add the teleport RPM repo
sudo dnf config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v14/teleport.repo")"

Install teleport
sudo dnf install teleport-ent

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

sudo dnf install teleport-ent-fips

Source variables about OS version
source /etc/os-release
Add the Teleport Zypper repository.
First, get the OS major version from $VERSION_ID so this fetches the correct
package version.
VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
Use Zypper to add the teleport RPM repo
sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v14/teleport-zypper.repo")

Install teleport
sudo zypper install teleport-ent

sudo zypper install teleport-ent-fips

curl https://get.gravitational.com/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz
shasum -a 256 teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz
cd teleport-ent
sudo ./install

curl https://get.gravitational.com/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz
shasum -a 256 teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz
cd teleport-ent
sudo ./install

Next, log in to your cluster from your local machine using tsh to ensure that your device will be able to interact with the cluster:

tsh login --proxy=myteleportcluster.com --user=teleport-username

Step 2/5. Create a role to issue tbot certificates

In order to create certificates tbot requires a role that provides the necessary permissions. When we apply a role to a tbot user in this case, we restrict a bot user to only create host certificates.

Teleport roles can additionally limit the creation of host certificates to a desired principal. For example, normally when creating an SSH host certificate a principal represents a user or host that the certificate is explicitly applied to.

By checking that a domain name matches the principal on a host certificate in the example configuration we use in this guide, access can be limited only to the desired server or servers. On Teleport, a predicate should be applied to a principal to further define the scope of permissions applied to the principal.

By using the format of where: 'predicate(host_cert.principals, "domain.name")', a Teleport user can create a role that will only be able to able to create host certificates with the designated principal. A list of supported predicates, their behavior, and example configurations are listed below:

In the case of an example role for use within this guide, use a text editor of your choice from your local administrative device with tsh access to create a role YAML named tbotrole.yaml with the following configuration options, replacing nodename.my.domain.com with the domain name associated with your OpenSSH server:

kind: role
metadata:
  name: hostcert-bot
version: v5
spec:
  allow:
    rules:
      - resources:
          - host_cert
        verbs:
          - create
        where: 'is_subset(host_cert.principals, "nodename.my.domain.com")'

Once complete, create your role to apply to the tbot configuration in the next step:

tctl create tbotrole.yaml

Step 3/5 Create Your tbot Configuration

Before you create a bot user, you need to determine which role(s) you want to assign to it. You can use the tctl command below from your local machine with tsh access to examine what roles exist on your system:

tctl get roles --format=text

You will see something like the output below on a fresh install of Teleport with the preset roles and the addition of the hostcert-bot role created previously—your own cluster may have different roles. In this example, you want to give the bot the hostcert-bot role to allow the bot to generate host certificates.

Role    Allowed to login as                           Node Labels Access to resources
------- --------------------------------------------- ----------- ----------------------------------------
access  {{internal.logins}}                           <all nodes> event:list,read,session:read,list
auditor no-login-6566121f-b602-47f1-a118-c9c618ee5aec             session:list,read,event:list,read
editor
hostcert-bot                                                      host_cert:create                               

Your Machine ID host can now join the Teleport Cluster using a number of supported methods. This guide will highlight both the token and the IAM method which uses identity tokens bound to an AWS account. Note that the following commands may be entered on any device with tctl access to the cluster.

While any role with permissions for creating host certificates will work, this guide will make use of the hostcert-bot role created previously to ensure that the configuration adheres to the principle of least privilege.

tctl bots add robot --roles=hostcert-bot

kind: token
version: v2
metadata:
  # The token name is not a secret because instances must prove that they are
  # running in your AWS account to use this token.
  name: iam-token
spec:
  # Only allow bots to join using this token.
  roles: [Bot]

  # Set the join method to be IAM.
  join_method: iam

  # Define the name of the bot that will be allowed to use this token.
  bot_name: robot

  allow:
  # Restrict the AWS account and (optionally) ARN that can use this token.
  # This information can be obtained from running the
  # "aws sts get-caller-identity" command from the CLI.
  - aws_account: "111111111111"
    aws_arn: "arn:aws:sts::111111111111:assumed-role/teleport-bot-role/i-*"

$ tctl create -f iam-token.yaml

$ tctl bots add robot --token=iam-token --roles=hostcert-bot

Generating a configuration file

Machine ID may now be configured using the tbot configure command on the Machine ID host.

First, create a basic configuration file using the following parameters:

tbot configure \   --data-dir=/var/lib/teleport/bot \   --token=abcd123-insecure-do-not-use-this \   --join-method=token \   --certificate-ttl=1h0m0s \   --ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \   --auth-server=example.teleport.sh:443

tbot configure \   --data-dir=/var/lib/teleport/bot \   --token=iam-token \   --join-method=iam \   --certificate-ttl=1h0m0s \   --ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \   --auth-server=example.teleport.sh:443

Replace the following fields with values from your own cluster:

• token is the token output by the tctl bots add command or the name of your IAM join token.
• ca-pin is the CA Pin for your Teleport cluster, and is output by the tctl bots add command.
• data-dir is where Machine ID writes its private data, including its own short-lived renewable certificates. These should not be used by applications and tools.
• certificate-ttl is the option that will determine the time until expiration for any tbot issued certificates, including host certificates. The default TTL is for one hour, meaning that all generated certificates will expire after one hour's time.
• auth-server is the address of your Teleport Proxy or Teleport Cloud tenant, for example tele.example.com:443.

Using a text editor of your choice, write the output generated by the tbot configure command to a configuration YAML file. In this example we'll be using tbot.yaml.

Configuring the output

You now need to add an output to the tbot configuration. An output specifies where tbot should write certificates and what types of certificate should be written.

Add an ssh_host output to tbot.yaml, specifying the directory you wish the credentials to be written to and the name which should be included in the host certificate:

outputs:
  - type: ssh_host
    destination:
      type: directory
      path: /opt/machine-id
    principals: [nodename.my.domain.com]

Once completed, your tbot.yaml configuration should resemble the following:

version: v2
onboarding:
  token: "1234abcd5678efgh9"
  ca_path: ""
  ca_pins:
  - sha256:1234abcd5678efgh910ijklmnop
  join_method: token
storage:
  type: directory
  path: /var/lib/teleport/bot
  symlinks: secure
  acls: try
outputs:
  - type: ssh_host
    destination:
      type: directory
      path: /opt/machine-id
    principals: [nodename.my.domain.com]
debug: false
auth_server: example.teleport.sh:443
certificate_ttl: 1h0m0s
renewal_interval: 20m0s
oneshot: false

Note the renewal_interval field, which will determine the cadence in which new certificates will be generated. Machine ID uses a default of 20 minutes for the renewal_interval, however this field may be adjusted as needed.

Step 4/5 Start Machine ID and create a host certificate

Using your configuration file, the tbot start command will start running Machine ID on your Machine ID host in a loop, writing renewable certificates to /var/lib/teleport/bot according to your configuration, and the short-lived certificates including the host certificate your bot will use to /opt/machine-id.

In a production environment you will want to run Machine ID in the background using a service manager like systemd. However, in this guide you will run it in the foreground to better understand how it works.

export TELEPORT_ANONYMOUS_TELEMETRY=1
tbot start -c tbot.yaml

TELEPORT_ANONYMOUS_TELEMETRY enables the submission of anonymous usage telemetry. This helps us shape the future development of tbot. You can disable this by omitting this.

Now that Machine ID has successfully started, let's investigate the /opt/machine-id directory to see what was written to disk.

ls -1 /opt/machine-id
identitykeykey-cert.pubkey.pubknown_hostsssh_configssh_hostssh_host-cert.pubssh_host-user-ca.pubteleport-database-ca.crtteleport-host-ca.crtteleport-user-ca.crttlscert

Included in the output is the host certificate, private key, and user CA file Machine ID will have generated. The ssh_host-cert.pub file is the public host certificate, the ssh_host file is a private key file bound to the certificate, and the ssh_host-user-ca.pub file is the public user CA file. To confirm that the outputted files have been created correctly, view the public host certificate's contents by entering the following command:

sudo ssh-keygen -L -f /opt/machine-id/key-cert.pub

The contents of the public certificate will appear, and will contain a principals field which will contain the nodename.my.domain.com domain entered in a previous step. Your output will resemble the following:

        Type: [email protected] host certificate
        Public key: RSA-CERT SHA256:abcde123efghijk567lmnop
        Signing CA: RSA SHA256:abcde123efghijk567lmnop (using rsa-sha2-512)
        Key ID: ""
        Serial: 0
        Valid: from 2022-09-30T13:16:36 to 2022-09-30T17:17:36
        Principals:
                nodename.my.domain.com
        Critical Options: (none)
        Extensions:
                x-teleport-authority UNKNOWN OPTION (len 55)
                x-teleport-role UNKNOWN OPTION (len 8)

Step 5/5 Configuring SSHD and connecting to the Teleport Cluster

In order to ensure that the Machine ID host is able to authorize and authenticate itself to the cluster using sshd, sshd must now be configured to use the generated certificate files.

Using a text editor of your choice, open the sshd configuration file, usually found at /etc/ssh/sshd_config, and add the following lines:

## Path to the private Key File affiliated with the host certificate.
HostKey /opt/machine-id/ssh_host

## Ensures that the OpenSSH server is offering the generated public host certificate when connections are attempted.
HostCertificate /opt/machine-id/ssh_host-cert.pub

## Ensures that the server will trust certificates signed by the remote CA
TrustedUserCAKeys /opt/machine-id/ssh_host-user-ca.pub

Once saved, restart sshd to apply the new configuration:

systemctl restart sshd

The Machine ID host is now fully configured to accept connections from a tsh user. To confirm this, enter the following command from your local machine logged into the cluster using tsh:

tsh ssh -p 22 [email protected]

The connection should successfully complete without the need to manually enter any additional credentials.

Next steps

Now that host certificates have been created with Teleport, they can be applied to OpenSSH configurations as needed. For more information regarding the next steps to connect to OpenSSH with Teleport, see the following documentation:

• Using Teleport With OpenSSH
• Using SSH Host Certificates
• Machine ID Configuration Reference
• Joining Nodes using the IAM method

More information about TELEPORT_ANONYMOUS_TELEMETRY.