Fork me on GitHub
Teleport

Adding Nodes to the cluster

This guide explains how to add nodes to Open Source, Enterprise Teleport, self-hosted or cloud editions.

Prerequisites

Verify that your Teleport client is connected:

$ tctl status

# Cluster  tele.example.com
# Version  7.1.3
# CA pin   sha256:sha-hash-here
Connecting to the cloud

To try this flow in the cloud, login into your cluster using tsh, then use tctl remotely:

$ tsh login --proxy=myinstance.teleport.sh
$ tctl status
Note

For cloud, login with a teleport user with editor privileges:

tsh logs you in and receives short-lived certificates

tsh login --proxy=myinstance.teleport.sh [email protected]

try out the connection

tctl get nodes

Adding Nodes to the cluster

Teleport only allows access to nodes that have joined the cluster.

Once a node joins, it receives its own host certificate signed by the cluster's auth server.

To receive a host certificate upon joining a cluster, a new Teleport host must present an invite token.

An invite token also defines which role a new host can assume within a cluster: auth, proxy, node, app, kube, or db.

There are two ways to create invitation tokens:

  • Short-lived Dynamic Tokens are more secure but require more planning.
  • Static Tokens are easy to use, but less secure.

Short-lived dynamic tokens

Administrators can generate tokens as they are needed.

Such token can be used multiple times until its time to live (TTL) expires.

Use the tctl tool to register a new invitation token (or it can also generate a new token for you). In the following example a new token is created with a TTL of 5 minutes:

tctl nodes add --ttl=5m --roles=node,proxy --token=secret-value

The invite token: secret-value

If --token is not provided, tctl will generate one:

Generate a short-lived invitation token for a new node:

tctl nodes add --ttl=5m --roles=node,app

The invite token: abcdefgh-do-not-use-this-token-123

You can also list all generated non-expired tokens:

tctl tokens ls

Token Type Expiry Time

------------------------ ----------- ---------------

abcd123-insecure-do-not-use-this Node 25 Sep 18 00:21 UTC

... or revoke an invitation before it's used:

tctl tokens rm abcd123-insecure-do-not-use-this

Using Node invitation tokens

Both static and short-lived dynamic tokens are used the same way. Execute the following command on a new node to add it to a cluster:

Adding a new regular SSH node to the cluster:

sudo teleport start --roles=node --token=abcd123-insecure-do-not-use-this --auth-server=10.0.10.5

Adding a new regular SSH node using Teleport Node Tunneling:

sudo teleport start --roles=node --token=abcd123-insecure-do-not-use-this --auth-server=teleport-proxy.example.com:3080

Adding a new proxy service on the cluster:

sudo teleport start --roles=proxy --token=abcd123-insecure-do-not-use-this --auth-server=10.0.10.5

As new nodes come online, they start sending ping requests every few seconds to the CA of the cluster. This allows users to explore cluster membership and size:

tctl nodes ls

Node Name Node ID Address Labels

--------- ------- ------- ------

turing d52527f9-b260-41d0-bb5a-e23b0cfe0f8f 10.1.0.5:3022 distro:ubuntu

dijkstra c9s93fd9-3333-91d3-9999-c9s93fd98f43 10.1.0.6:3022 distro:debian

Revoking invitations

As you have seen above, Teleport uses tokens to invite users to a cluster (sign-up tokens) or to add new nodes to it (provisioning tokens).

Both types of tokens can be revoked before they can be used. To see a list of outstanding tokens, run this command:

tctl tokens ls

Token Role Expiry Time (UTC)

----- ---- -----------------

abcd123-insecure-do-not-use-this Proxy never

efgh456-insecure-do-not-use-this Node 17 May 16 03:51 UTC

ijkl789-insecure-do-not-use-this Signup 17 May 16 04:24 UTC

In this example, the first token has a "never" expiry date because it is a static token configured via a config file.

The 2nd token with the "Node" role was generated to invite a new node to this cluster. And the 3rd token was generated to invite a new user.

The latter two tokens can be deleted (revoked) via tctl tokens del command:

tctl tokens del abcd123-insecure-do-not-use-this

Token abcd123-insecure-do-not-use-this has been deleted

Adding a Node located behind NAT

Note
This feature is sometimes called "Teleport IoT" or node tunneling.

With the current setup, you've only been able to add nodes that have direct access to the auth server and within the internal IP range of the cluster. We recommend setting up a Trusted Cluster if you have workloads split across different networks/clouds.

Teleport Node Tunneling lets you add a remote node to an existing Teleport Cluster through a tunnel. This can be useful for IoT applications, or for managing a couple of servers in a different network.

Similar to Adding Nodes to the Cluster, use tctl to create a single-use token for a node, but this time you'll replace the auth server IP with the URL of the proxy server. In the example below, we've replaced the auth server IP with the proxy web endpoint teleport-proxy.example.com:3080.

tctl tokens add --type=node | grep -oP '(?<=token:\s).*' > token.file

This will save the token to a file.

Run this on the new node to join the cluster:

sudo teleport start --roles=node --token=/path/to/token.file --auth-server=teleport-proxy.example.com:3080

Using the ports in the default configuration, the node needs to be able to talk to ports 3080 and 3024 on the proxy. Port 3080 is used to initially fetch the credentials (SSH and TLS certificates) and for discovery (where is the reverse tunnel running, in this case, 3024). For those using ACME, port 443 is also required. Port 3024 is used to establish a connection to the auth server through the proxy.

To enable multiplexing so only one port is used, simply set the tunnel_listen_addr the same as the web_listen_addr respectively within the proxy_service. Teleport will automatically recognize using the same port and enable multiplexing. If the log setting is set to DEBUG you will see multiplexing enabled in the server log.

DEBU [PROC:1]    Setup Proxy: Reverse tunnel proxy and web proxy listen on the same port, multiplexing is on. service/service.go:1944
Load Balancers
The setup above also works even if the cluster uses multiple proxies behind a load balancer (LB) or a DNS entry with multiple values. This works by the node establishing a tunnel to every proxy. This requires that an LB uses a round-robin or a similar balancing algorithm. Do not use sticky load balancing algorithms (a.k.a. "session affinity") with Teleport proxies.

Next Steps

Static tokens

Static tokens are defined ahead of time by an administrator and stored in the auth server's config file:

# Config section in `/etc/teleport.yaml` file for the auth server
auth_service:
    enabled: true
    tokens:
    # This static token allows new hosts to join the cluster as "proxy" or "node"
    - "proxy,node:secret-token-value"
    # A token can also be stored in a file. In this example the token for adding
    # new auth servers are stored in /path/to/tokenfile
    - "auth:/path/to/tokenfile"

Untrusted auth servers

Teleport nodes use the HTTPS protocol to offer the join tokens to the auth server running on 10.0.10.5 in the example above. In a zero-trust environment, you must assume that an attacker can hijack the IP address of the auth server e.g. 10.0.10.5.

To prevent this from happening, you need to supply every new node with an additional bit of information about the auth server. This technique is called "CA Pinning". It works by asking the auth server to produce a "CA Pin", which is a hashed value of its public key, i.e. for which an attacker can't forge a matching private key.

On the auth server:

tctl status

Cluster staging.example.com

User CA never updated

Host CA never updated

CA pin sha256:7e12c17c20d9cb504bbcb3f0236be3f446861f1396dcbb44425fe28ec1c108f1

The "CA pin" at the bottom needs to be passed to the new nodes when they're starting for the first time, i.e. when they join a cluster:

Via CLI:

sudo teleport start \ --roles=node \ --token=abcd123-insecure-do-not-use-this \ --ca-pin=sha256:7e12c17c20d9cb504bbcb3f0236be3f446861f1396dcbb44425fe28ec1c108f1 \ --auth-server=10.12.0.6:3025

or via /etc/teleport.yaml on a node:

teleport:
  auth_token: "abcd123-insecure-do-not-use-this"
  ca_pin: "sha256:7e12c17c20d9cb504bbcb3f0236be3f446861f1396dcbb44425fe28ec1c108f1"
  auth_servers:
    - "10.12.0.6:3025"
Warning
If a CA pin is not provided, the Teleport Node will join a cluster but it will print a WARN message (warning) into its standard error output.
Warning
The CA pin becomes invalid if a Teleport administrator performs the CA rotation by executing tctl auth rotate .
Have a suggestion or can’t find something?
IMPROVE THE DOCS