Fork me on GitHub

Teleport

Adding Nodes to your Cluster

Improve

This guide explains how to add Teleport Nodes to your cluster.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 11.1.0.

    tctl version

    Teleport v11.1.0 go1.19

    tsh version

    Teleport v11.1.0 go1.19

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 11.1.0, which you can download by visiting the customer portal.

    tctl version

    Teleport v11.1.0 go1.19

    tsh version

    Teleport v11.1.0 go1.19

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.3.8. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.3.8 go1.19

    tsh version

    Teleport v10.3.8 go1.19

  • A Linux server that you will use to host your Teleport Node.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 11.1.0

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 10.3.8

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Step 1/3. Install Teleport on your Node

On the host where you will run your Teleport Node, follow the instructions for your environment to install Teleport.

Download Teleport's PGP public key

sudo curl https://apt.releases.teleport.dev/gpg \ -o /usr/share/keyrings/teleport-archive-keyring.asc

Source variables about OS version

source /etc/os-release

Add the Teleport APT repository for v11. You'll need to update this

file for each major release of Teleport.

Note: if using a fork of Debian or Ubuntu you may need to use '$ID_LIKE'

and the codename your distro was forked from instead of '$ID' and '$VERSION_CODENAME'.

Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L42-L67

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \ https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v11" \ | sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport

Source variables about OS version

source /etc/os-release

Add the Teleport YUM repository for v11. You'll need to update this

file for each major release of Teleport.

Note: if using a fork of RHEL/CentOS or Amazon Linux you may need to use '$ID_LIKE'

and the codename your distro was forked from instead of '$ID'

Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L133-L153

sudo yum-config-manager --add-repo $(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v11/teleport.repo")
sudo yum install teleport

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

Optional: Using DNF on newer distributions

$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo

$ sudo dnf install teleport

curl https://get.gravitational.com/teleport-v11.1.0-linux-amd64-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v11.1.0-linux-amd64-bin.tar.gz
shasum -a 256 teleport-v11.1.0-linux-amd64-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v11.1.0-linux-amd64-bin.tar.gz
cd teleport
sudo ./install
curl https://get.gravitational.com/teleport-v11.1.0-linux-arm-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v11.1.0-linux-arm-bin.tar.gz
shasum -a 256 teleport-v11.1.0-linux-arm-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v11.1.0-linux-arm-bin.tar.gz
cd teleport
sudo ./install
curl https://get.gravitational.com/teleport-v11.1.0-linux-arm64-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v11.1.0-linux-arm64-bin.tar.gz
shasum -a 256 teleport-v11.1.0-linux-arm64-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v11.1.0-linux-arm64-bin.tar.gz
cd teleport
sudo ./install

Using this APT repo may result in breaking upgrades upon "apt upgrade" as all major versions will be

published under the same component. We recommend following the instructions in the

"Debian/Ubuntu (DEB)" tab instead.

Download Teleport's PGP public key

sudo curl https://deb.releases.teleport.dev/teleport-pubkey.asc \ -o /usr/share/keyrings/teleport-archive-keyring.asc

Add the Teleport APT repository

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] https://deb.releases.teleport.dev/ stable main" \ | sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport
sudo yum-config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo
sudo yum install teleport

Optional: Using DNF on newer distributions

$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo

$ sudo dnf install teleport

Step 2/3. Join your Node to the cluster

In this section, we will join your Node to the Teleport cluster by:

  • Obtaining information stored on the Auth Service
  • Starting Teleport on your Node with the information we obtained

Obtain a CA pin

In a zero-trust environment, you must assume that an attacker can hijack the IP address of the Auth Service.

To prevent this from happening, you need to supply every new Node with information about the Auth Service. This technique is called CA pinning. It works by asking the Auth Service to produce a CA pin, a hash value of the SPKI header in a certificate. This way, an attacker cannot easily forge a matching private key.

If a CA pin is not provided, the Teleport Node will join a cluster but it will print a warning message.

The CA pin becomes invalid if a Teleport administrator performs the CA rotation by executing tctl auth rotate.

On you local machine, retrieve the CA pin of the Auth Service :

export CA_PIN=$(tctl status | awk '/CA pin/{print $3}')

Generate a token

Teleport only allows access to Nodes that have joined the cluster.

Once a Node joins, it receives a host certificate signed by the cluster's Auth Service. To receive a host certificate upon joining a cluster, a new Teleport host must present an invite token.

An invite token also defines which role a new host can assume within a cluster (e.g., auth, proxy, node, or kube).

Administrators can generate tokens as they are needed. A token can be used multiple times until its time to live (TTL) expires.

On your local machine, use the tctl tool to generate a new token. In the following example, a new token is created with a TTL of five minutes:

Generate a short-lived invite token for a new node:

export INVITE_TOKEN=$(tctl nodes add --ttl=5m --roles=node | grep "invite token:" | grep -Eo "[0-9a-z]{32}")
echo ${INVITE_TOKEN}

abcd123-insecure-do-not-use-this

You can also list all generated non-expired tokens:

tctl tokens ls

Token Type Expiry Time

------------------------ ----------- ---------------

abcd123-insecure-do-not-use-this Node 25 Sep 18 00:21 UTC

Rather than automatically generating a token, you can create one with a known value by using the --token flag:

tctl nodes add --ttl=5m --roles=node,proxy --token=secret-value

The invite token: secret-value

The value of --token should be cryptographically secure. For example, you can create an SHA256 hash to reduce the risk that a malicious actor will produce a token with the same value as yours:

head -n 1 /dev/random | sha256sum
Warning

Use short-lived tokens instead of long-lived static tokens. Static tokens are easier to steal, guess, and leak.

Static tokens are defined ahead of time by an administrator and stored in the auth server's config file:

# Config section in `/etc/teleport.yaml` file for the auth server
auth_service:
    enabled: true
    tokens:
    # This static token allows new hosts to join the cluster as "proxy" or "node"
    - "proxy,node:secret-token-value"
    # A token can also be stored in a file. In this example the token for adding
    # new auth servers are stored in /path/to/tokenfile
    - "auth:/path/to/tokenfile"

Start your Node with the invite token and CA pin

Execute the following commands on the host where you will run your Node so you can use the CA_PIN and INVITE_TOKEN variables to start Teleport.

export CA_PIN=$<CA pin>
export INVITE_TOKEN=<invite token>

Still on the host where you will run your Node, assign the address of your Auth Service to an environment variable, including the domain name (or IP address) and port:

export ADDR=teleport.example.com:3025

If you plan to use Teleport Node Tunneling, which we will explain below, set ADDR to the domain name (or IP address) and web_listen_addr port of your Teleport Proxy Service.

Teleport Node Tunneling lets you add a remote Node to an existing Teleport Cluster through a reverse tunnel. This is useful when the Auth Service is inaccessible from the network where the Node is running, e.g., for IoT applications.

We recommend setting up a Trusted Cluster if you have workloads split across different networks or clouds.

Using the ports in Teleport's default configuration, the Node needs to be able to talk to ports 3080 and 3024 on the Proxy Service. Port 3080 is used to initially fetch the credentials (SSH and TLS certificates) and for discovering the reverse tunnel. Port 3024 is used to establish a connection to the Auth Service through the Proxy.

For those using ACME, port 443 is also required.

To enable multiplexing so only one port is used, simply set the tunnel_listen_addr to the same value as web_listen_addr within the proxy_service section of your configuration file. Teleport will automatically recognize that the Proxy Service is using the same port for both addresses and enable multiplexing.

If your log setting is set to DEBUG, you will see multiplexing enabled in the server log.

DEBU [PROC:1]    Setup Proxy: Reverse tunnel proxy and web proxy listen on the same port, multiplexing is on. service/service.go:1944
Load Balancers

The setup above also works even if the cluster uses multiple Proxy Service instances behind a load balancer (LB) or a DNS entry with multiple values. In this case, the Node establishes a tunnel to every proxy.

This requires that an LB uses a round-robin or a similar balancing algorithm. Do not use sticky load balancing algorithms (a.k.a. "session affinity") with Teleport Proxy Service instances.

Execute the following command on your Node to add it to a cluster:

sudo teleport start \ --roles=node \ --token=${INVITE_TOKEN?} \ --ca-pin=${CA_PIN?} \ --auth-server=${ADDR?}

As new Nodes come online, they start sending ping requests every few seconds to the Auth Service. This allows users to explore cluster membership and size.

Run the following command on your local machine to see all of the Teleport Nodes in your cluster:

tctl nodes ls

Node Name Node ID Address Labels

--------- ------- ------- ------

turing d52527f9-b260-41d0-bb5a-e23b0cfe0f8f 10.1.0.5:3022 distro:ubuntu

dijkstra c9s93fd9-3333-91d3-9999-c9s93fd98f43 10.1.0.6:3022 distro:debian

Step 3/3. Revoke an invitation

Tokens used for joining Nodes to a cluster can be revoked before they are used.

Run the following command on your local machine to create a token for a new Proxy Service:

tctl nodes add --ttl=5m --roles=proxy

The invite token: abcd123-insecure-do-not-use-this.

This token will expire in 5 minutes.

Run this on the new node to join the cluster:

> teleport start \

--roles=proxy \

--token=abcd123-insecure-do-not-use-this \

--ca-pin=sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 \

--auth-server=123.123.123.123:3025

Please note:

- This invitation token will expire in 5 minutes

- 123.123.123.123 must be reachable from the new node

Next, run the following command to see a list of outstanding tokens:

$ tctl tokens ls

# Token                                Role       Expiry Time (UTC)
# -----                                ----       -----------------
# abcd123-insecure-do-not-use-this     Proxy      never
# efgh456-insecure-do-not-use-this     Node       17 May 16 03:51 UTC
# ijkl789-insecure-do-not-use-this     Signup     17 May 16 04:24 UTC
Signup tokens

The output of tctl tokens ls includes tokens used for adding users alongside tokens used for adding Nodes to your cluster.

In this example, the first token has a never expiry date because it is a static token configured via a config file.

The token with the Node role was generated to invite a new Node to this cluster. And the third token was generated to invite a new user to sign up.

Tokens created via tctl can be deleted (revoked) via the tctl tokens del command. Run the following command to delete a token:

tctl tokens del abcd123-insecure-do-not-use-this

Token abcd123-insecure-do-not-use-this has been deleted