Adding Nodes to the cluster
This guide explains how to add nodes to Open Source, Enterprise Teleport, self-hosted or cloud editions.
Verify that your Teleport client is connected:
$ tctl status # Cluster tele.example.com # Version 7.1.3 # CA pin sha256:sha-hash-here
To try this flow in the cloud, login into your cluster using tsh, then use tctl remotely:
$ tsh login --proxy=myinstance.teleport.sh $ tctl status
For cloud, login with a teleport user with editor privileges:
tsh logs you in and receives short-lived certificatestsh login --proxy=myinstance.teleport.sh [email protected]
try out the connectiontctl get nodes
Teleport only allows access to nodes that have joined the cluster.
Once a node joins, it receives its own host certificate signed by the cluster's auth server.
To receive a host certificate upon joining a cluster, a new Teleport host must present an invite token.
An invite token also defines which role a new host can assume within a cluster:
There are two ways to create invitation tokens:
- Short-lived Dynamic Tokens are more secure but require more planning.
- Static Tokens are easy to use, but less secure.
Administrators can generate tokens as they are needed.
Such token can be used multiple times until its time to live (TTL) expires.
tctl tool to register a new invitation token (or
it can also generate a new token for you). In the following example a new token
is created with a TTL of 5 minutes:
tctl nodes add --ttl=5m --roles=node,proxy --token=secret-value
The invite token: secret-value
--token is not provided,
tctl will generate one:
Generate a short-lived invitation token for a new node:tctl nodes add --ttl=5m --roles=node,app
The invite token: abcdefgh-do-not-use-this-token-123
You can also list all generated non-expired tokens:tctl tokens ls
Token Type Expiry Time
------------------------ ----------- ---------------
abcd123-insecure-do-not-use-this Node 25 Sep 18 00:21 UTC
... or revoke an invitation before it's used:tctl tokens rm abcd123-insecure-do-not-use-this
Both static and short-lived dynamic tokens are used the same way. Execute the following command on a new node to add it to a cluster:
Adding a new regular SSH node to the cluster:sudo teleport start --roles=node --token=abcd123-insecure-do-not-use-this --auth-server=10.0.10.5
Adding a new regular SSH node using Teleport Node Tunneling:sudo teleport start --roles=node --token=abcd123-insecure-do-not-use-this --auth-server=teleport-proxy.example.com:3080
Adding a new proxy service on the cluster:sudo teleport start --roles=proxy --token=abcd123-insecure-do-not-use-this --auth-server=10.0.10.5
As new nodes come online, they start sending ping requests every few seconds to the CA of the cluster. This allows users to explore cluster membership and size:
tctl nodes ls
Node Name Node ID Address Labels
--------- ------- ------- ------
turing d52527f9-b260-41d0-bb5a-e23b0cfe0f8f 10.1.0.5:3022 distro:ubuntu
dijkstra c9s93fd9-3333-91d3-9999-c9s93fd98f43 10.1.0.6:3022 distro:debian
As you have seen above, Teleport uses tokens to invite users to a cluster (sign-up tokens) or to add new nodes to it (provisioning tokens).
Both types of tokens can be revoked before they can be used. To see a list of outstanding tokens, run this command:
tctl tokens ls
Token Role Expiry Time (UTC)
----- ---- -----------------
abcd123-insecure-do-not-use-this Proxy never
efgh456-insecure-do-not-use-this Node 17 May 16 03:51 UTC
ijkl789-insecure-do-not-use-this Signup 17 May 16 04:24 UTC
In this example, the first token has a "never" expiry date because it is a static token configured via a config file.
The 2nd token with the "Node" role was generated to invite a new node to this cluster. And the 3rd token was generated to invite a new user.
The latter two tokens can be deleted (revoked) via
tctl tokens del abcd123-insecure-do-not-use-this
Token abcd123-insecure-do-not-use-this has been deleted
With the current setup, you've only been able to add nodes that have direct access to the auth server and within the internal IP range of the cluster. We recommend setting up a Trusted Cluster if you have workloads split across different networks/clouds.
Teleport Node Tunneling lets you add a remote node to an existing Teleport Cluster through a tunnel. This can be useful for IoT applications, or for managing a couple of servers in a different network.
Similar to Adding Nodes to the Cluster, use
create a single-use token for a node, but this time you'll replace the auth
server IP with the URL of the proxy server. In the example below, we've
replaced the auth server IP with the proxy web endpoint
tctl tokens add --type=node | grep -oP '(?<=token:\s).*' > token.file
This will save the token to a file.
Run this on the new node to join the cluster:sudo teleport start --roles=node --token=/path/to/token.file --auth-server=teleport-proxy.example.com:3080
Using the ports in the default configuration, the node needs to be able to talk to ports
3024 on the proxy. Port
3080 is used to initially fetch the credentials (SSH and TLS certificates) and for discovery (where is the reverse tunnel running, in this case,
3024). For those using ACME, port
443 is also required. Port
3024 is used to establish a connection to the auth server through the proxy.
To enable multiplexing so only one port is used, simply set the
tunnel_listen_addr the same as the
web_listen_addr respectively within the
proxy_service. Teleport will automatically recognize using the same port and enable multiplexing. If the log setting is set to DEBUG you will see multiplexing enabled in the server log.
DEBU [PROC:1] Setup Proxy: Reverse tunnel proxy and web proxy listen on the same port, multiplexing is on. service/service.go:1944
Static tokens are defined ahead of time by an administrator and stored in the auth server's config file:
# Config section in `/etc/teleport.yaml` file for the auth server auth_service: enabled: true tokens: # This static token allows new hosts to join the cluster as "proxy" or "node" - "proxy,node:secret-token-value" # A token can also be stored in a file. In this example the token for adding # new auth servers are stored in /path/to/tokenfile - "auth:/path/to/tokenfile"
Teleport nodes use the HTTPS protocol to offer the join tokens to the auth
server running on
10.0.10.5 in the example above. In a zero-trust environment,
you must assume that an attacker can hijack the IP address of the auth server
To prevent this from happening, you need to supply every new node with an additional bit of information about the auth server. This technique is called "CA Pinning". It works by asking the auth server to produce a "CA Pin", which is a hashed value of its public key, i.e. for which an attacker can't forge a matching private key.
On the auth server:
User CA never updated
Host CA never updated
CA pin sha256:7e12c17c20d9cb504bbcb3f0236be3f446861f1396dcbb44425fe28ec1c108f1
The "CA pin" at the bottom needs to be passed to the new nodes when they're starting for the first time, i.e. when they join a cluster:
sudo teleport start \ --roles=node \ --token=abcd123-insecure-do-not-use-this \ --ca-pin=sha256:7e12c17c20d9cb504bbcb3f0236be3f446861f1396dcbb44425fe28ec1c108f1 \ --auth-server=10.12.0.6:3025
/etc/teleport.yaml on a node:
teleport: auth_token: "abcd123-insecure-do-not-use-this" ca_pin: "sha256:7e12c17c20d9cb504bbcb3f0236be3f446861f1396dcbb44425fe28ec1c108f1" auth_servers: - "10.12.0.6:3025"
WARNmessage (warning) into its standard error output.
tctl auth rotate.