Teleport Session Recording
- Available for:
This page provides an overview of how Teleport records sessions and the various configuration options that apply to session recordings.
The content that is captured depends on the type of the session.
Teleport captures the entire pseudo-terminal (PTY) output of the session. The intention is for session recording to document what a user saw when they ran a session.
Note that the design of session recording carries some security risks. Namely,
users can conceal terminal commands by encoding them (e.g., using
running scripts from disk or the internet, or changing the terminal settings. If
this presents an issue for your environment, consider using the BPF-based
Enhanced Session Recording
Teleport captures the entire PTY output for
kubectl exec invocations.
Kubernetes session recording carries the same security risks as SSH session recording.
Teleport captures the contents of the desktop screen and any mouse input. Teleport does not capture keystrokes in the remote desktop.
Teleport captures a stream of
app.session.request audit events related to
application access. The audit events are bucketed into 5-minute time intervals
informally referred to as "chunks."
Teleport captures a stream of audit events related to the database being accessed.
Teleport supports two different recording configurations: synchronous and asynchronous recording. Additionally, Teleport administrators can configure where the recording takes place: at the node or at the proxy.
This results in 4 possible session recording configurations:
node-sync: synchronous recording performed by an SSH Service instance
node: asynchronous recording performed by an SSH Service instance
proxy-sync: synchronous recording performed on a Proxy Service instance
proxy: asynchronous recording performed on a Proxy Service instance
This is a cluster-wide configuration option and applies to the entire
Teleport cluster. It can be configured by setting
auth_service section of your
teleport.yaml, or dynamically using
session_recording_config resource. If you need to apply different
recording configurations to different sets of resources, you can set up
trusted clusters with their own
Windows, database, and Kubernetes sessions are always recorded at the host running the Teleport service for this session type, since there is no Teleport software running on the "node" (desktop, database, or Kubernetes cluster).
For this reason, the Teleport Windows, Database, and Kubernetes Services treat
proxy identically (perform asynchronous recording) and
proxy-sync identically (perform synchronous recording).
By default, Teleport performs recording at the SSH node. This is because Teleport's Proxy Server cannot see the SSH traffic to the node, it is encrypted end-to-end (from the SSH client to the SSH server).
In some cases, it is desirable to perform the recording at Teleport's Proxy Server.
For example, sessions to non-Teleport servers (like OpenSSH's
sshd) can only be
recorded at a Proxy Service instance. This is referred to as Recording Proxy Mode.
In this mode, the Proxy Server terminates (decrypts) the SSH connection and establishes its own SSH connection to the destination server, effectively becoming an authorized "man in the middle."
We consider this mode to be less secure, as it grants additional privileges to the Proxy Server. Since the Proxy Server needs credentials to decrypt the SSH connection, it must be properly secured and is a higher value target for an attacker than a proxy that cannot decrypt the data flowing through it.
Additionally, the credentials that the Proxy Server uses to decrypt the SSH connection are provided via SSH Agent Forwarding, so Agent Forwarding must be enabled to record at the Proxy Server.
When synchronous recording is enabled, the Teleport component doing the recording (which may be the Node or the Proxy Service instance depending on your configuration) submits each recording event to Teleport's Auth Server as it occurs. In this mode, failure to emit a recording event is considered fatal - the session will be terminated if an event cannot be recorded. This makes synchronous recording best suited for highly regulated environments where you need to be confident that all data is recorded. This also means that you need a reliable and low-latency connection to the Auth Server for the duration of the session to ensure that the session isn't interrupted or terminated due to temporary connection loss.
In synchronous recording modes, the Auth Server receives a stream of recording events and is responsible for assembling them into the final artifact and uploading it to the storage backend. Since data is streamed directly to the Auth Server, Teleport administrators don't need to be concerned with disk space on their Nodes and Proxy Service instances, as no recording data is written to those disks.
When asynchronous, recording events are written to the local filesystem during the session. When the session completes, Teleport assembles the parts into a complete recording and submits the entire recording to the Auth Server for storage.
Since recording data is flushed to disk, administrators should be careful to ensure that the system has enough disk space to accommodate the expected number of Teleport sessions. Additionally, since recording data is temporarily stored on disk, there is a greater chance that it can be tampered with, deleted, or otherwise corrupted before the upload completes.
The advantage of asynchronous recording is that it doesn't require a persistent connection to the Auth Server. For example, an SSH session can continue to operate even if Teleport's Auth Server goes down. When the session completes Teleport will attempt to upload the recording to the Auth Server. If the Auth Server is still unavailable, Teleport has built-in retry and backoff mechanisms that will upload the artifact when the Auth Server comes back online. Additionally, asynchronous recording is well-suited towards recording sessions that are extra chatty or in environments where the connection to the auth server is unreliable or high-latency.
Session recordings are stored in Teleport's audit sessions backend, which is
specified by the
audit_sessions_uri field in the
teleport.yaml configuration file.
Teleport current supports the following session storage backends:
- File: stores recordings on the local filesystem. Suitable for dev environments, demos, and small home environments.
- S3: stores recordings in an AWS S3 bucket, or S3-compatible storage. Suitable for production deployments.
- GCS: stores recordings in Google Cloud Storage. Suitable for production deployments.
The Teleport Auth Service is the only component that writes directly to the storage backend.
Note that the session storage backend is distinct from Teleport's cluster state or audit log storage, which supports a different set of services (SQLite, DynamoDB, Firestore, etc).
A Teleport session recording is an ordered sequence of structured events associated with a session. Each event is an encoded Protocol Buffer and the complete session is compressed with gzip before being written to the storage backend.
These recordings have a
.tar extension for backwards compatibility reasons, but it
should be noted that they are not TAR archives and cannot be read using the
SSH and Kubernetes sessions can be played in Teleport's Web UI or by using
tsh play command. Desktop session
recordings can only be played back in the Web UI.
In the Web UI, the session recordings page is populated by querying Teleport's audit log for session end events, which contain metadata about the recording. This is sometimes surprising to Teleport users, because even though the recordings and audit log are stored in separate backends, both need to be operational in order to play recordings.
Every Teleport process runs a service called the upload completer which periodically checks for abandoned uploads and completes them if there is not an active session tracker for the session associated with the recording. By default, the upload completer runs every 5 minutes, and session trackers have a 30 minute expiration period. This means it can take up to ~35 minutes after the service comes back online for an abandoned upload to be completed.
In asynchronous recording modes, if the node goes down during the session, the partially completed recording will sit on the node's disk. The node's upload completer will eventually detect the abandoned upload and stream it to the Teleport Auth Server where it will be written to the storage backend.
In synchronous recording modes, Teleport's Auth Server is streaming the recording directly to storage. If the Auth Server goes down during a session, the uncompleted upload will sit as a series of parts (in cloud storage or on the Auth Server's disk) and it is the responsibility of the Auth Server's upload completer to detect the abandoned upload and complete it.