Teleport Recording Proxy Mode
- Version 15.x
- Version 14.x
- Version 13.x
- Version 12.x
- Older Versions
- Available for:
Teleport Recording Proxy Mode was added to allow Teleport users
to enable session recording for servers running
sshd, which is helpful
when gradually transitioning large server fleets to Teleport.
Teleport Cloud only supports session recording at the Node level. If you are interested in setting up session recording, read our Server Access Getting Started Guide so you can start replacing your OpenSSH servers with Teleport Nodes.
We consider Recording Proxy Mode to be less secure than recording at the Node level for two reasons:
- It grants additional privileges to the Teleport Proxy Service. In the default Node Recording mode, the Proxy Service stores no secrets and cannot "see" the decrypted data. This makes a Proxy Server less critical to the security of the overall cluster. But if an attacker gains physical access to a Proxy Server running in Proxy Recording mode, they will be able to see the decrypted traffic and client keys stored in the Proxy Server's process memory.
- Recording Proxy Mode requires the use of SSH agent forwarding. Agent forwarding is required because without it, a Proxy Server will not be able to establish a second connection to the destination node.
The Teleport Proxy Service should be available to clients and set up with TLS.
A running Teleport Enterprise cluster. For details on how to set this up, see our Enterprise Getting Started guide.
tctladmin tool and
tshclient tool version >= 14.2.1, which you can download by visiting your Teleport account.tctl version
Teleport Enterprise v14.2.1 go1.21tsh version
Teleport v14.2.1 go1.21
- A host where you will run an OpenSSH server.
- To check that you can connect to your Teleport cluster, sign in with
tsh login, then verify that you can run
tctlcommands using your current credentials.
tctlis supported on macOS and Linux machines. For example:If you can connect to the cluster and run thetsh login --proxy=teleport.example.com --user=[email protected]tctl status
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
tctl statuscommand, you can use your current credentials to run subsequent
tctlcommands from your workstation. If you host your own Teleport cluster, you can also run
tctlcommands on the computer that hosts the Teleport Auth Service for full permissions.
When running Teleport in production, you should adhere to the following best practices to avoid security incidents:
- Avoid using
sudoin production environments unless it's necessary.
- Create new, non-root, users and use test instances for experimenting with Teleport.
- Run Teleport's services as a non-root user unless required. Only the SSH
Service requires root access. Note that you will need root permissions (or
CAP_NET_BIND_SERVICEcapability) to make Teleport listen on a port numbered <
- Follow the principle of least privilege. Don't give users
permissive roles when more a restrictive role will do.
For example, don't assign users the built-in
access,editorroles, which give them permissions to access and edit all cluster resources. Instead, define roles with the minimum required permissions for each user and configure access requests to provide temporary elevated permissions.
- When you enroll Teleport resources—for example, new databases or applications—you
should save the invitation token to a file.
If you enter the token directly on the command line, a malicious user could view
it by running the
historycommand on a compromised system.
You should note that these practices aren't necessarily reflected in the examples used in documentation. Examples in the documentation are primarily intended for demonstration and for development environments.
Backing up production instances, environments, and/or settings before making permanent modifications is encouraged as a best practice. Doing so allows you to roll back to an existing state if needed.
To enable session recording for
sshd nodes, the cluster must be switched to
Recording Proxy Mode. In this mode, the recording will be done on the Proxy level.
Edit the Auth Service configuration file as follows:
# snippet from /etc/teleport.yaml auth_service: # Session Recording must be set to Proxy to work with OpenSSH session_recording: "proxy" # can also be "off" and "node" (default)
When in recording mode, Teleport will check that the host certificate of any Node a user connects to is signed by a Teleport CA. By default, this is a strict check. If the Node presents just a key or a certificate signed by a different CA, Teleport will reject this connection with the error message:
ssh: handshake failed: remote host presented a public key, expected a host certificate
You can disable strict host checks as shown below. However, this opens the possibility for Person-in-the-Middle attacks and is not recommended.
# snippet from /etc/teleport.yaml auth_service: proxy_checks_host_keys: no
sshd must be told to allow users to log in with certificates generated
by the Teleport User CA. Start by exporting the Teleport CA public key.
On your Teleport Node, export the Teleport Certificate Authority certificate into a file and update your SSH configuration to trust Teleport's CA. Assign proxy to the address of your Teleport Proxy Service:
curl 'https://proxy/webapi/auth/export?type=user' | sed s/cert-authority\ // > teleport_user_ca.pubsudo mv ./teleport_user_ca.pub /etc/ssh/teleport_user_ca.pubecho "TrustedUserCAKeys /etc/ssh/teleport_user_ca.pub" | sudo tee -a /etc/ssh/sshd_config
sshd will trust users who present a Teleport-issued certificate.
The next step is to configure host authentication.
The recommended solution is to ask Teleport to issue valid host certificates for all OpenSSH nodes. To generate a host certificate, run this on your Teleport Auth Server:
Creating host certs, with an array of every host to be accessed.
Wildcard certs aren't supported by OpenSSH. The domain must be fully
Management of the host certificates can become complex. This is another
reason we recommend using Teleport SSH on nodes.sudo tctl auth sign \ --host=api.example.com,ssh.example.com,188.8.131.52,184.108.40.206 \ --format=openssh \ --out=api.example.com
The credentials have been written to api.example.com, api.example.com-cert.pub
You can use ssh-keygen to verify the contents.ssh-keygen -L -f api.example.com-cert.pub
Type: [email protected] host certificate
Public key: RSA-CERT SHA256:ireEc5HWFjhYPUhmztaFud7EgsopO8l+GpxNMd3wMSk
Signing CA: RSA SHA256:/6HSHsoU5u+r85M26Ut+M9gl+HventwSwrbTvP/cmvo
Key ID: ""
Valid: after 2020-07-29T20:26:24
Critical Options: (none)
x-teleport-authority UNKNOWN OPTION (len 47)
x-teleport-role UNKNOWN OPTION (len 8)
Then add the following lines to
/etc/ssh/sshd_config on all OpenSSH nodes, and
HostKey /etc/ssh/api.example.com HostCertificate /etc/ssh/api.example.com-cert.pub
Now you can use the
tsh ssh command to log in to any
sshd node in the
cluster, and the session will be recorded.
If you want to use the OpenSSH
ssh client for logging into
sshd servers behind a proxy
in "recording mode", you have to tell the
ssh client to use a jump host and
enable SSH agent forwarding, otherwise, a recording proxy will not be able to
terminate the SSH connection to record it:
Note that agent forwarding is enabled twice: one from a client to a proxy
(mandatory if using a recording proxy), and then optionally from a proxy
to the end server if you want your agent running on the end server
To avoid typing all this and use the usual
ssh [email protected], users can update their
Verify that a Teleport certificate is loaded into the agent after logging in:
Login as Joetsh login --proxy=proxy.example.com --user=joe
see if the certificate is present (look for "teleport:joe") at the end of the certssh-add -L
It is well known that the Gnome Keyring SSH agent, used by many popular Linux desktops like Ubuntu, and
gpg-agent from GnuPG do not support SSH
certificates. We recommend using the
ssh-agent from OpenSSH.
Alternatively, you can disable the SSH agent integration entirely using the
--no-use-local-ssh-agent flag or
environment variable with
When using a Teleport proxy in "recording mode", be aware of OpenSSH's built-in rate-limiting. On large numbers of Proxy Service connections, you may encounter errors like:
channel 0: open failed: connect failed: ssh: handshake failed: EOF
MaxStartups setting in
man sshd_config. This setting means that by
default, OpenSSH only allows 10 unauthenticated connections at a time and starts
dropping connections 30% of the time when the number of connections goes over 10.
When it hits 100 authentication connections, all new connections are
To increase the concurrency level, increase the value to something like
MaxStartups 50:30:100. This allows 50 concurrent connections and a max of 100.