Fork me on GitHub

Teleport

Generating Short-Lived Host Certificates with Machine ID

  • Available for:
  • OpenSource
  • Team
  • Cloud
  • Enterprise

This version of the guide uses the v2 tbot configuration. This version is only supported by Teleport 14 and beyond. Change the selected version of the documentation to view the guide for previous Teleport versions.

Host certificates are generally created on Teleport for access to OpenSSH servers that can not otherwise join a Teleport cluster. While long-lived certificates may be applied for this purpose, short-lived and regularly rotated host certificates generated through Teleport help to provide a number of security benefits over their long-lived counterparts. Regular certificate rotation reduces risk by ensuring that any potentially stolen certificates are usable for a shorter period of time. Additionally, when coupled with Teleport's RBAC support and host certificate Principals and Predicates, you can apply limitations to both the process of creating host certificates, as well as the host certificates themselves.

In this guide, you will learn how to create host certificates through Machine ID for OpenSSH servers, allowing for the benefits of host certificates with OpenSSH nodes, and reducing risk by ensuring that short-lived certificates adhering to the principle of least privilege can be successfully applied to Teleport clusters.

Prerequisites

  • A Teleport Team account. If you don't have an account, sign up to begin your free trial.

  • The Enterprise tctl admin tool and tsh client tool, version >= 14.0.0.

    You can download these tools by visiting your Teleport account workspace.

  • A running Teleport cluster. For details on how to set this up, see the Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 14.0.0.

    See Installation for details.

  • A running Teleport Enterprise cluster. For details on how to set this up, see the Enterprise Getting Started guide.

  • The Enterprise tctl admin tool and tsh client tool version >= 14.0.0.

    You can download these tools by visiting your Teleport account workspace.

Cloud is not available for Teleport v.
Please use the latest version of Teleport Enterprise documentation.

To check version information, run the tctl version and tsh version commands. For example:

tctl version

Teleport Enterprise v13.3.9 git:api/14.0.0-gd1e081e go1.21


tsh version

Teleport v13.3.9 go1.21

Proxy version: 13.3.9Proxy: teleport.example.com

To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands on your administrative workstation using your current credentials. For example:

tsh login --proxy=teleport.example.com --user=[email protected]
tctl status

Cluster teleport.example.com

Version 14.0.0

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

  • A Linux based host that can support Machine ID. This "Machine ID Host" is the host that will be used to create Host Certificates using tbot. This host should additionally have OpenSSH server sshd version 6.9 or above running. The SSH port on this host must be open to traffic from the Teleport Proxy Service host.

    ssh -V

Step 1/5. Download and install Teleport

Select an edition, then follow the instructions for that edition to install Teleport.

Teleport Edition

curl https://goteleport.com/static/install.sh | bash -s 13.3.9

Before installing a teleport binary with a version besides v13, read our compatibility rules to ensure that the binary is compatible with Teleport Cloud.

When running multiple teleport binaries within a cluster, the following rules apply:

  • Patch and minor versions are always compatible, for example, any 8.0.1 component will work with any 8.0.3 component and any 8.1.0 component will work with any 8.3.0 component.
  • Servers support clients that are 1 major version behind, but do not support clients that are on a newer major version. For example, an 8.x.x Proxy Service is compatible with 7.x.x resource services and 7.x.x tsh, but we don't guarantee that a 9.x.x resource service will work with an 8.x.x Proxy Service. This also means you must not attempt to upgrade from 6.x.x straight to 8.x.x. You must upgrade to 7.x.x first.
  • Proxy Services and resource services do not support Auth Services that are on an older major version, and will fail to connect to older Auth Services by default. This behavior can be overridden by passing --skip-version-check when starting Proxy Services and resource services.
curl https://goteleport.com/static/install.sh | bash -s 14.0.0

Download Teleport's PGP public key

sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc

Source variables about OS version

source /etc/os-release

Add the Teleport APT repository for v14. You'll need to update this

file for each major release of Teleport.

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v14" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport-ent

For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

sudo apt-get install teleport-ent-fips

Source variables about OS version

source /etc/os-release

Add the Teleport YUM repository for v14. You'll need to update this

file for each major release of Teleport.

First, get the major version from $VERSION_ID so this fetches the correct

package version.

VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
sudo yum-config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v14/teleport.repo")"
sudo yum install teleport-ent

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

sudo yum install teleport-ent-fips

Source variables about OS version

source /etc/os-release

Add the Teleport Zypper repository for v14. You'll need to update this

file for each major release of Teleport.

First, get the OS major version from $VERSION_ID so this fetches the correct

package version.

VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

Use zypper to add the teleport RPM repo

sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/cloud/teleport-zypper.repo")
sudo yum install teleport-ent

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

sudo yum install teleport-ent-fips

Source variables about OS version

source /etc/os-release

Add the Teleport YUM repository for v14. You'll need to update this

file for each major release of Teleport.

First, get the major version from $VERSION_ID so this fetches the correct

package version.

VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

Use the dnf config manager plugin to add the teleport RPM repo

sudo dnf config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v14/teleport.repo")"

Install teleport

sudo dnf install teleport-ent

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

sudo dnf install teleport-ent-fips

Source variables about OS version

source /etc/os-release

Add the Teleport Zypper repository.

First, get the OS major version from $VERSION_ID so this fetches the correct

package version.

VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

Use Zypper to add the teleport RPM repo

sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v14/teleport-zypper.repo")

Install teleport

sudo zypper install teleport-ent

For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

sudo zypper install teleport-ent-fips

In the example commands below, update $SYSTEM_ARCH with the appropriate value (amd64, arm64, or arm). All example commands using this variable will update after one is filled out.

curl https://get.gravitational.com/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://cdn.teleport.dev/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz
shasum -a 256 teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz

Verify that the checksums match

tar -xvf teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-bin.tar.gz
cd teleport-ent
sudo ./install

For FedRAMP/FIPS-compliant installations of Teleport Enterprise, package URLs will be slightly different:

curl https://get.gravitational.com/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://cdn.teleport.dev/teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz
shasum -a 256 teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz

Verify that the checksums match

tar -xvf teleport-ent-v14.0.0-linux-$SYSTEM_ARCH-fips-bin.tar.gz
cd teleport-ent
sudo ./install

OS repository channels

The following channels are available for APT, YUM, and Zypper repos. They may be used in place of stable/v14 anywhere in the Teleport documentation.

Channel nameDescription
stable/<major>Receives releases for the specified major release line, i.e. v14
stable/cloudRolling channel that receives releases compatible with current Cloud version
stable/rollingRolling channel that receives all published Teleport releases
Cloud is not available for Teleport v.
Please use the latest version of Teleport Enterprise documentation.

Next, log in to your cluster from your local machine using tsh to ensure that your device will be able to interact with the cluster:

tsh login --proxy=myteleportcluster.com --user=teleport-username

Step 2/5. Create a role to issue tbot certificates

In order to create certificates tbot requires a role that provides the necessary permissions. When we apply a role to a tbot user in this case, we restrict a bot user to only create host certificates.

Teleport roles can additionally limit the creation of host certificates to a desired principal. For example, normally when creating an SSH host certificate a principal represents a user or host that the certificate is explicitly applied to.

By checking that a domain name matches the principal on a host certificate in the example configuration we use in this guide, access can be limited only to the desired server or servers. On Teleport, a predicate should be applied to a principal to further define the scope of permissions applied to the principal.

By using the format of where: 'predicate(host_cert.principals, "domain.name")', a Teleport user can create a role that will only be able to able to create host certificates with the designated principal. A list of supported predicates, their behavior, and example configurations are listed below:

PredicateBehaviorExample
all_equalAll requested principals must be equal to the set string. If the set string is foo.example.com, host certificates limit access only to foo.example.com.all_equal(host_cert.principals, "foo.example.com")
all_end_withAll requested principals must end with the set string. If the set string is .example.com, host certificates limit access only to *.example.com.all_end_with(host_cert.principals, ".example.com")
is_subsetAll requested principals must be contained in the listed set.is_subset(host_cert.principals, "foo.example.com", "bar.example.com")

In the case of an example role for use within this guide, use a text editor of your choice from your local administrative device with tsh access to create a role YAML named tbotrole.yaml with the following configuration options, replacing nodename.my.domain.com with the domain name associated with your OpenSSH server:

kind: role
metadata:
  name: hostcert-bot
version: v5
spec:
  allow:
    rules:
      - resources:
          - host_cert
        verbs:
          - create
        where: 'is_subset(host_cert.principals, "nodename.my.domain.com")'

Once complete, create your role to apply to the tbot configuration in the next step:

tctl create tbotrole.yaml

Step 3/5 Create Your tbot Configuration

Before you create a bot user, you need to determine which role(s) you want to assign to it. You can use the tctl command below from your local machine with tsh access to examine what roles exist on your system:

tctl get roles --format=text

You will see something like the output below on a fresh install of Teleport with the preset roles and the addition of the hostcert-bot role created previously—your own cluster may have different roles. In this example, you want to give the bot the hostcert-bot role to allow the bot to generate host certificates.

Role    Allowed to login as                           Node Labels Access to resources
------- --------------------------------------------- ----------- ----------------------------------------
access  {{internal.logins}}                           <all nodes> event:list,read,session:read,list
auditor no-login-6566121f-b602-47f1-a118-c9c618ee5aec             session:list,read,event:list,read
editor
hostcert-bot                                                      host_cert:create                               

Your Machine ID host can now join the Teleport Cluster using a number of supported methods. This guide will highlight both the token and the IAM method which uses identity tokens bound to an AWS account. Note that the following commands may be entered on any device with tctl access to the cluster.

While any role with permissions for creating host certificates will work, this guide will make use of the hostcert-bot role created previously to ensure that the configuration adheres to the principle of least privilege.

tctl bots add robot --roles=hostcert-bot

First, create an IAM method token that specifies the AWS account from which the bot can join. Create the below file as iam-token.yaml:

kind: token
version: v2
metadata:
  # The token name is not a secret because instances must prove that they are
  # running in your AWS account to use this token.
  name: iam-token
spec:
  # Only allow bots to join using this token.
  roles: [Bot]

  # Set the join method to be IAM.
  join_method: iam

  # Define the name of the bot that will be allowed to use this token.
  bot_name: robot

  allow:
  # Restrict the AWS account and (optionally) ARN that can use this token.
  # This information can be obtained from running the
  # "aws sts get-caller-identity" command from the CLI.
  - aws_account: "111111111111"
    aws_arn: "arn:aws:sts::111111111111:assumed-role/teleport-bot-role/i-*"

With the iam-token-yaml file created, enter the following command:

$ tctl create -f iam-token.yaml

Next, create the bot user.

$ tctl bots add robot --token=iam-token --roles=hostcert-bot

Generating a configuration file

Machine ID may now be configured using the tbot configure command on the Machine ID host.

First, create a basic configuration file using the following parameters:

tbot configure \ --data-dir=/var/lib/teleport/bot \ --token=abcd123-insecure-do-not-use-this \ --join-method=token \ --certificate-ttl=1h0m0s \ --ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \ --auth-server=example.teleport.sh:443
tbot configure \ --data-dir=/var/lib/teleport/bot \ --token=iam-token \ --join-method=iam \ --certificate-ttl=1h0m0s \ --ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \ --auth-server=example.teleport.sh:443

Replace the following fields with values from your own cluster:

  • token is the token output by the tctl bots add command or the name of your IAM join token.
  • ca-pin is the CA Pin for your Teleport cluster, and is output by the tctl bots add command.
  • data-dir is where Machine ID writes its private data, including its own short-lived renewable certificates. These should not be used by applications and tools.
  • certificate-ttl is the option that will determine the time until expiration for any tbot issued certificates, including host certificates. The default TTL is for one hour, meaning that all generated certificates will expire after one hour's time.
  • auth-server is the address of your Teleport Proxy or Teleport Cloud tenant, for example tele.example.com:443.

Using a text editor of your choice, write the output generated by the tbot configure command to a configuration YAML file. In this example we'll be using tbot.yaml.

Configuring the output

You now need to add an output to the tbot configuration. An output specifies where tbot should write certificates and what types of certificate should be written.

Add an ssh_host output to tbot.yaml, specifying the directory you wish the credentials to be written to and the name which should be included in the host certificate:

outputs:
  - type: ssh_host
    destination:
      type: directory
      path: /opt/machine-id
    principals: [nodename.my.domain.com]

Once completed, your tbot.yaml configuration should resemble the following:

version: v2
onboarding:
  token: "1234abcd5678efgh9"
  ca_path: ""
  ca_pins:
  - sha256:1234abcd5678efgh910ijklmnop
  join_method: token
storage:
  type: directory
  path: /var/lib/teleport/bot
  symlinks: secure
  acls: try
outputs:
  - type: ssh_host
    destination:
      type: directory
      path: /opt/machine-id
    principals: [nodename.my.domain.com]
debug: false
auth_server: example.teleport.sh:443
certificate_ttl: 1h0m0s
renewal_interval: 20m0s
oneshot: false

Note the renewal_interval field, which will determine the cadence in which new certificates will be generated. Machine ID uses a default of 20 minutes for the renewal_interval, however this field may be adjusted as needed.

Step 4/5 Start Machine ID and create a host certificate

Using your configuration file, the tbot start command will start running Machine ID on your Machine ID host in a loop, writing renewable certificates to /var/lib/teleport/bot according to your configuration, and the short-lived certificates including the host certificate your bot will use to /opt/machine-id.

In a production environment you will want to run Machine ID in the background using a service manager like systemd. However, in this guide you will run it in the foreground to better understand how it works.

export TELEPORT_ANONYMOUS_TELEMETRY=1
tbot start -c tbot.yaml

TELEPORT_ANONYMOUS_TELEMETRY enables the submission of anonymous usage telemetry. This helps us shape the future development of tbot. You can disable this by omitting this.

Now that Machine ID has successfully started, let's investigate the /opt/machine-id directory to see what was written to disk.

ls -1 /opt/machine-id
identitykeykey-cert.pubkey.pubknown_hostsssh_configssh_hostssh_host-cert.pubssh_host-user-ca.pubteleport-database-ca.crtteleport-host-ca.crtteleport-user-ca.crttlscert

Included in the output is the host certificate, private key, and user CA file Machine ID will have generated. The ssh_host-cert.pub file is the public host certificate, the ssh_host file is a private key file bound to the certificate, and the ssh_host-user-ca.pub file is the public user CA file. To confirm that the outputted files have been created correctly, view the public host certificate's contents by entering the following command:

sudo ssh-keygen -L -f /opt/machine-id/key-cert.pub

The contents of the public certificate will appear, and will contain a principals field which will contain the nodename.my.domain.com domain entered in a previous step. Your output will resemble the following:

        Type: [email protected] host certificate
        Public key: RSA-CERT SHA256:abcde123efghijk567lmnop
        Signing CA: RSA SHA256:abcde123efghijk567lmnop (using rsa-sha2-512)
        Key ID: ""
        Serial: 0
        Valid: from 2022-09-30T13:16:36 to 2022-09-30T17:17:36
        Principals:
                nodename.my.domain.com
        Critical Options: (none)
        Extensions:
                x-teleport-authority UNKNOWN OPTION (len 55)
                x-teleport-role UNKNOWN OPTION (len 8)

Step 5/5 Configuring SSHD and connecting to the Teleport Cluster

In order to ensure that the Machine ID host is able to authorize and authenticate itself to the cluster using sshd, sshd must now be configured to use the generated certificate files.

Using a text editor of your choice, open the sshd configuration file, usually found at /etc/ssh/sshd_config, and add the following lines:

## Path to the private Key File affiliated with the host certificate.
HostKey /opt/machine-id/ssh_host

## Ensures that the OpenSSH server is offering the generated public host certificate when connections are attempted.
HostCertificate /opt/machine-id/ssh_host-cert.pub

## Ensures that the server will trust certificates signed by the remote CA
TrustedUserCAKeys /opt/machine-id/ssh_host-user-ca.pub

Once saved, restart sshd to apply the new configuration:

systemctl restart sshd

The Machine ID host is now fully configured to accept connections from a tsh user. To confirm this, enter the following command from your local machine logged into the cluster using tsh:

tsh ssh -p 22 [email protected]

The connection should successfully complete without the need to manually enter any additional credentials.

Next steps

Now that host certificates have been created with Teleport, they can be applied to OpenSSH configurations as needed. For more information regarding the next steps to connect to OpenSSH with Teleport, see the following documentation:

More information about TELEPORT_ANONYMOUS_TELEMETRY.