

Enhanced SSH Session Recording with BPF
Length: 05:01
This guide explains Enhanced Session Recording for SSH with BPF and how to set it up in your Teleport cluster.
Teleport's default SSH and Kubernetes session recording feature captures what is echoed to a terminal.
This has inherent advantages. For example, because no input is captured, Teleport session recordings typically do not contain passwords that were entered into a terminal.
The disadvantage is that there are several techniques for rendering session recordings less useful:
- Obfuscation. For example, even though the command
echo Y3VybCBodHRwOi8vd3d3LmV4YW1wbGUuY29tCg== | base64 --decode | sh
does not containcurl http://www.example.com
, when decoded, that is what is run. - Shell scripts. For example, if a user uploads and executes a script, the commands run within the script are not captured, only the output.
- Terminal controls. Terminals support a wide variety of controls including the ability for users to disable terminal echo. This is frequently used when requesting credentials. Disabling terminal echo allows commands to be run without being captured.
Furthermore, due to their unstructured nature, session recordings are difficult to ingest and perform monitoring and alerting on.
Teleport Enhanced Session Recording mitigates all three concerns by providing advanced security and greater logging capabilities, and better correlates a user with their activities.
Teleport Enhanced Session Recording does not provide a secure environment on its own and is not a substitute for a Linux Security Module (SELinux, AppArmor, etc.). It must be paired with reasonable system hardening practices to enforce a trusted host environment, including proper access control on core system binaries and libraries and well-designed user management.
Note that privileged users (either as root or via sudo
) can interfere with
session recording activities (such as unloading/disabling the necessary
libraries, altering how Teleport is run, tampering with kernel functionality,
creating tunnels, or just performing actions outside of the restricted
session). Also, a local user with both monitored and unmonitored console
sessions or ptrace privileges may not be fully captured in recordings.
Commands executed via daemons (systemd, crond, atd, etc.) could be outside of the recorded session scope. Proper network-based restrictions for ingress traffic must also be implemented to prevent possible unauthorized data transfer.
Additionally, certain forensic information such as full binary paths (accounting for any potential symbolic links), any modifications via shared library preloading, and environment variables may not be captured in session recordings.
Prerequisites
-
A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.
-
The
tctl
admin tool andtsh
client tool version >= 13.0.3.tctl versionTeleport v13.0.3 go1.20
tsh versionTeleport v13.0.3 go1.20
See Installation for details.
-
A running Teleport Enterprise cluster. For details on how to set this up, see our Enterprise Getting Started guide.
-
The Enterprise
tctl
admin tool andtsh
client tool version >= 13.0.3, which you can download by visiting your Teleport account.tctl versionTeleport Enterprise v13.0.3 go1.20
tsh versionTeleport v13.0.3 go1.20
Please use the latest version of Teleport Enterprise documentation.
-
At least one host that you will use to run the Teleport Node Service. The host must run Linux kernel 5.8 (or above).
You can check your kernel version using the
uname
command. The output should look something like the following.uname -r5.8.17
See below for more details on the required versions for your Linux kernel and distribution.
Our Standard Session Recording works with older Linux kernels. View Teleport Nodes for more details.
Linux distributions and supported kernels
Distro name | Distro version | Kernel Version |
---|---|---|
Ubuntu "Groovy Gorilla" | 20.10 | 5.8+ |
Fedora | 33 | 5.8+ |
Archlinux | 2020.09.01 | 5.8.5+ |
Flatcar | 2765.2.2 | 5.10.25+ |
Amazon Linux | 2 | 5.10+ |
- Make sure you can connect to Teleport. Log in to your cluster using
tsh
, then usetctl
remotely:tsh login --proxy=teleport.example.com [email protected]tctl statusCluster teleport.example.com
Version 13.0.3
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
You can run subsequent
tctl
commands in this guide on your local machine.For full privileges, you can also run
tctl
commands on your Auth Service host.
Step 1/2. Configure a Teleport Node
Install Teleport on your Node
On the host where you will run your Teleport Node, follow the instructions for your environment to install Teleport.
Use the appropriate commands for your environment to install your package.
Teleport Edition
Download Teleport's PGP public key
sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.ascSource variables about OS version
source /etc/os-releaseAdd the Teleport APT repository for v13. You'll need to update this
file for each major release of Teleport.
echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v13" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/nullsudo apt-get updatesudo apt-get install teleport
Source variables about OS version
source /etc/os-releaseAdd the Teleport YUM repository for v13. You'll need to update this
file for each major release of Teleport.
sudo yum-config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v13/teleport.repo")"sudo yum install teleportTip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path
Source variables about OS version
source /etc/os-releaseAdd the Teleport YUM repository for v13. You'll need to update this
file for each major release of Teleport.
Use the dnf config manager plugin to add the teleport RPM repo
sudo dnf config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v13/teleport.repo")"Install teleport
sudo dnf install teleportTip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path
In the example commands below, update $SYSTEM_ARCH
with the appropriate
value (amd64
, arm64
, or arm
). All example commands using this variable
will update after one is filled out.
curl https://get.gravitational.com/teleport-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gz.sha256<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gzshasum -a 256 teleport-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gzVerify that the checksums match
tar -xvf teleport-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gzcd teleportsudo ./install
Download Teleport's PGP public key
sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.ascSource variables about OS version
source /etc/os-releaseAdd the Teleport APT repository for v13. You'll need to update this
file for each major release of Teleport.
echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v13" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/nullsudo apt-get updatesudo apt-get install teleport-ent
For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips
package instead:
sudo apt-get install teleport-ent-fips
Source variables about OS version
source /etc/os-releaseAdd the Teleport YUM repository for v13. You'll need to update this
file for each major release of Teleport.
sudo yum-config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v13/teleport.repo")"sudo yum install teleport-entTip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path
For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips
package instead:
sudo yum install teleport-ent-fips
Source variables about OS version
source /etc/os-releaseAdd the Teleport YUM repository for v13. You'll need to update this
file for each major release of Teleport.
Use the dnf config manager plugin to add the teleport RPM repo
sudo dnf config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v13/teleport.repo")"Install teleport
sudo dnf install teleport-entTip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path
For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips
package instead:
sudo dnf install teleport-ent-fips
In the example commands below, update $SYSTEM_ARCH
with the appropriate
value (amd64
, arm64
, or arm
). All example commands using this variable
will update after one is filled out.
curl https://get.gravitational.com/teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gz.sha256<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gzshasum -a 256 teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gzVerify that the checksums match
tar -xvf teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-bin.tar.gzcd teleport-entsudo ./install
For FedRAMP/FIPS-compliant installations of Teleport Enterprise, package URLs will be slightly different:
curl https://get.gravitational.com/teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-fips-bin.tar.gz.sha256<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-fips-bin.tar.gzshasum -a 256 teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-fips-bin.tar.gzVerify that the checksums match
tar -xvf teleport-ent-v13.0.3-linux-$SYSTEM_ARCH-fips-bin.tar.gzcd teleport-entsudo ./install
Please use the latest version of Teleport Enterprise documentation.
Generate a token
Use the tctl
tool to generate an invite token that your Node will use to join
the cluster. In the following example, a new token is created with a TTL of five
minutes:
Generate a short-lived invitation token for a new node:
tctl nodes add --ttl=5m --roles=nodeThe invite token: abcd123-insecure-do-not-use-this
You can also list all generated non-expired tokens:
tctl tokens lsToken Type Expiry Time
------------------------ ----------- ---------------
abcd123-insecure-do-not-use-this Node 25 Sep 18 00:21 UTC
... or revoke an invitation token before it's used:
tctl tokens rm abcd123-insecure-do-not-use-this
Create a configuration file
Set up your Teleport Node with the following content in /etc/teleport.yaml
.
# Example config to be saved as etc/teleport.yaml
version: v3
teleport:
nodename: graviton-node
# The token you created earlier
auth_token: abcd123-insecure-do-not-use-this
# Replace with the address of the Teleport Auth Service
auth_server: 127.0.0.1:3025
# Or specify the Proxy Service address.
proxy_server: 127.0.0.1:3080
data_dir: /var/lib/teleport
proxy_service:
enabled: false
auth_service:
enabled: false
ssh_service:
enabled: true
enhanced_recording:
# Enable or disable enhanced auditing for this node. Default value: false.
enabled: true
# Optional: command_buffer_size is optional with a default value of 8 pages.
command_buffer_size: 8
# Optional: disk_buffer_size is optional with default value of 128 pages.
disk_buffer_size: 128
# Optional: network_buffer_size is optional with default value of 8 pages.
network_buffer_size: 8
# Optional: Controls where cgroupv2 hierarchy is mounted. Default value:
# /cgroup2.
cgroup_path: /cgroup2
Start Teleport on your Node
teleport start
Step 2/2. Inspect the audit log
Enhanced session recording events will be shown in Teleport's audit log, which you can inspect by visiting Teleport's Web UI.
Events emitted via Enhanced Session Recording will include the
"enhanced_recording": true
field in events similar to the following:
{
"code": "T2004I",
"ei": 23,
"enhanced_recording": true,
"event": "session.end",
"interactive": true,
"namespace": "default",
"participants": [
"benarent"
],
"server_id": "585fc225-5cf9-4e9f-8ff6-1b0fd6885b09",
"sid": "ca82b98d-1d30-11ea-8244-cafde5327a6c",
"time": "2019-12-12T22:44:46.218Z",
"uid": "83e67464-a93a-4c7c-8ce6-5a3d8802c3b2",
"user": "benarent"
}
If your Teleport cluster uses a file-based event log, you can examine your audit log on the Teleport Auth Service host.
Teleport's session recordings backend is configured via the
teleport.storage.audit_sessions_uri
field. If a provided URI includes a scheme
that belongs to a cloud-based service (e.g., s3://
or dynamodb://
), you will
not be able to inspect session recordings in the filesystem of your Auth Service
host.
Examine the contents of /var/lib/teleport/log
as shown below:
teleport-auth ~: tree /var/lib/teleport/log/var/lib/teleport/log
├── 1048a649-8f3f-4431-9529-0c53339b65a5
│ ├── 2020-01-13.00:00:00.log
│ └── sessions
│ └── default
│ ├── fad07202-35bb-11ea-83aa-125400432324-0.chunks.gz
│ ├── fad07202-35bb-11ea-83aa-125400432324-0.events.gz
│ ├── fad07202-35bb-11ea-83aa-125400432324-0.session.command-events.gz
│ ├── fad07202-35bb-11ea-83aa-125400432324-0.session.network-events.gz
│ └── fad07202-35bb-11ea-83aa-125400432324.index
├── events.log -> /var/lib/teleport/log/1048a649-8f3f-4431-9529-0c53339b65a5/2020-01-13.00:00:00.log
├── playbacks
│ └── sessions
│ └── default
└── upload
└── sessions
└── default
To quickly check the status of the audit log, you can simply tail the logs with
tail -f /var/lib/teleport/log/events.log
. The resulting capture from Teleport will
be a JSON log for each command and network request.
Please use the latest version of Teleport Enterprise documentation.
Next steps
- Read more about session recording.
- See all configuration options for Enhanced Session Recording in our Configuration Reference.