Enhanced Session Recording with BPF

This guide explains Enhanced Session Recording for SSH with BPF and how to set it up in your Teleport cluster.

Teleport's default SSH and Kubernetes session recording feature captures what is echoed to a terminal.

This has inherent advantages. For example, because no input is captured, Teleport session recordings typically do not contain passwords that were entered into a terminal.

The disadvantage is that there are several techniques for rendering session recordings less useful:

• Obfuscation. For example, even though the command echo Y3VybCBodHRwOi8vd3d3LmV4YW1wbGUuY29tCg== | base64 --decode | sh does not contain curl http://www.example.com, when decoded, that is what is run.
• Shell scripts. For example, if a user uploads and executes a script, the commands run within the script are not captured, only the output.
• Terminal controls. Terminals support a wide variety of controls including the ability for users to disable terminal echo. This is frequently used when requesting credentials. Disabling terminal echo allows commands to be run without being captured.

Furthermore, due to their unstructured nature, session recordings are difficult to ingest and perform monitoring and alerting on.

Teleport Enhanced Session Recording mitigates all three concerns by providing advanced security and greater logging capabilities, and better correlates a user with their activities.

Prerequisites

• At least one host that you will use to run the Teleport Node Service. The host must run Linux kernel 5.8 (or above).

  You can check your kernel version using the uname command. The output should look something like the following.

  uname -r
  5.8.17

  See below for more details on the required versions for your Linux kernel and distribution.

Linux distributions and supported kernels

tsh login --proxy=teleport.example.com [email protected]
tctl status
Cluster  teleport.example.com
Version  11.3.1
CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status
Cluster  myinstance.teleport.sh
Version  11.2.1
CA pin   sha256:sha-hash-here

Step 1/2. Configure a Teleport Node

Install Teleport on your Node

On the host where you will run your Teleport Node, follow the instructions for your environment to install Teleport.

Next, use the appropriate commands for your environment to install your package.

Download Teleport's PGP public key
sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc
Source variables about OS version
source /etc/os-release
Add the Teleport APT repository for v11. You'll need to update this
file for each major release of Teleport.
Note: if using a fork of Debian or Ubuntu you may need to use '$ID_LIKE'
and the codename your distro was forked from instead of '$ID' and '$VERSION_CODENAME'.
Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L42-L67
echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v11" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null
sudo apt-get update
sudo apt-get install teleport

Source variables about OS version
source /etc/os-release
Add the Teleport YUM repository for v11. You'll need to update this
file for each major release of Teleport.
Note: if using a fork of RHEL/CentOS or Amazon Linux you may need to use '$ID_LIKE'
and the codename your distro was forked from instead of '$ID'
Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L133-L153
sudo yum-config-manager --add-repo $(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v11/teleport.repo")
sudo yum install teleport
Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path
Optional:  Use DNF on newer distributions
$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo
$ sudo dnf install teleport

curl https://get.gravitational.com/teleport-v11.3.1-linux-
-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-v11.3.1-linux-
-bin.tar.gz
shasum -a 256 teleport-v11.3.1-linux-
-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-v11.3.1-linux-
-bin.tar.gz
cd teleport
sudo ./install

dpkg -i ~/Downloads/teleport-ent_11.3.1_
.deb
Selecting previously unselected package teleport-ent.
(Reading database ... 30810 files and directories currently installed.)
Preparing to unpack teleport-ent_11.3.1_$SYSTEM_ARCH.deb ...
Unpacking teleport-ent 11.3.1 ...
Setting up teleport-ent 11.3.1 ...

rpm -i ~/Downloads/teleport-ent-11.3.1.
.rpm
warning: teleport-ent-11.3.1.$SYSTEM-ARCH.rpm: Header V4 RSA/SHA512 Signature, key ID 6282c411: NOKEY

curl https://get.gravitational.com/teleport-ent-v11.3.1-linux-
-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-v11.3.1-linux-
-bin.tar.gz
shasum -a 256 teleport-v11.3.1-linux-
-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-v11.3.1-linux-
-bin.tar.gz
cd teleport
sudo ./install

curl https://get.gravitational.com/teleport-ent-v11.3.1-linux-
-fips-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v11.3.1-linux-
-fips-bin.tar.gz
shasum -a 256 teleport-ent-v11.3.1-linux-
-fips-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-ent-v11.3.1-linux-
-fips-bin.tar.gz
cd teleport-ent
sudo ./install

dpkg -i ~/Downloads/teleport-ent_11.2.1_
.deb
Selecting previously unselected package teleport-ent.
(Reading database ... 30810 files and directories currently installed.)
Preparing to unpack teleport-ent_11.2.1_$SYSTEM_ARCH.deb ...
Unpacking teleport-ent 11.2.1 ...
Setting up teleport-ent 11.2.1 ...

rpm -i ~/Downloads/teleport-ent-11.2.1.
.rpm
warning: teleport-ent-11.2.1.$SYSTEM-ARCH.rpm: Header V4 RSA/SHA512 Signature, key ID 6282c411: NOKEY

curl https://get.gravitational.com/teleport-ent-v11.2.1-linux-
-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-v11.2.1-linux-amd64-bin.tar.gz
shasum -a 256 teleport-v11.2.1-linux-amd64-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-v11.2.1-linux-amd64-bin.tar.gz
cd teleport
sudo ./install

Generate a token

Use the tctl tool to generate an invite token that your Node will use to join the cluster. In the following example, a new token is created with a TTL of five minutes:

Generate a short-lived invitation token for a new node:
tctl nodes add --ttl=5m --roles=node
The invite token: abcd123-insecure-do-not-use-this
You can also list all generated non-expired tokens:
tctl tokens ls
Token                            Type            Expiry Time
------------------------         -----------     ---------------
abcd123-insecure-do-not-use-this         Node            25 Sep 18 00:21 UTC
... or revoke an invitation token before it's used:
tctl tokens rm abcd123-insecure-do-not-use-this

Create a configuration file

Set up your Teleport Node with the following content in /etc/teleport.yaml.

# Example config to be saved as etc/teleport.yaml
version: v3
teleport:
  nodename: graviton-node
  # The token you created earlier
  auth_token: abcd123-insecure-do-not-use-this

  # Replace with the address of the Teleport Auth Service
  auth_server: 127.0.0.1:3025
  # Or specify the Proxy Service address.
  proxy_server: 127.0.0.1:3080

  data_dir: /var/lib/teleport
proxy_service:
  enabled: false
auth_service:
  enabled: false
ssh_service:
  enabled: true
  enhanced_recording:
    # Enable or disable enhanced auditing for this node. Default value: false.
    enabled: true

    # Optional: command_buffer_size is optional with a default value of 8 pages.
    command_buffer_size: 8

    # Optional: disk_buffer_size is optional with default value of 128 pages.
    disk_buffer_size: 128

    # Optional: network_buffer_size is optional with default value of 8 pages.
    network_buffer_size: 8

    # Optional: Controls where cgroupv2 hierarchy is mounted. Default value:
    # /cgroup2.
    cgroup_path: /cgroup2

Start Teleport on your Node

teleport start

Step 2/2. Inspect the audit log

{
  "code": "T2004I",
  "ei": 23,
  "enhanced_recording": true,
  "event": "session.end",
  "interactive": true,
  "namespace": "default",
  "participants": [
    "benarent"
  ],
  "server_id": "585fc225-5cf9-4e9f-8ff6-1b0fd6885b09",
  "sid": "ca82b98d-1d30-11ea-8244-cafde5327a6c",
  "time": "2019-12-12T22:44:46.218Z",
  "uid": "83e67464-a93a-4c7c-8ce6-5a3d8802c3b2",
  "user": "benarent"
}

teleport-auth ~:  tree /var/lib/teleport/log
/var/lib/teleport/log
├── 1048a649-8f3f-4431-9529-0c53339b65a5
│   ├── 2020-01-13.00:00:00.log
│   └── sessions
│       └── default
│           ├── fad07202-35bb-11ea-83aa-125400432324-0.chunks.gz
│           ├── fad07202-35bb-11ea-83aa-125400432324-0.events.gz
│           ├── fad07202-35bb-11ea-83aa-125400432324-0.session.command-events.gz
│           ├── fad07202-35bb-11ea-83aa-125400432324-0.session.network-events.gz
│           └── fad07202-35bb-11ea-83aa-125400432324.index
├── events.log -> /var/lib/teleport/log/1048a649-8f3f-4431-9529-0c53339b65a5/2020-01-13.00:00:00.log
├── playbacks
│   └── sessions
│       └── default
└── upload
└── sessions
└── default

{
  "code": "T2004I",
  "ei": 23,
  "enhanced_recording": true,
  "event": "session.end",
  "interactive": true,
  "namespace": "default",
  "participants": [
    "benarent"
  ],
  "server_id": "585fc225-5cf9-4e9f-8ff6-1b0fd6885b09",
  "sid": "ca82b98d-1d30-11ea-8244-cafde5327a6c",
  "time": "2019-12-12T22:44:46.218Z",
  "uid": "83e67464-a93a-4c7c-8ce6-5a3d8802c3b2",
  "user": "benarent"
}

Next steps

• Read more about session recording.
• See all configuration options for Enhanced Session Recording in our Configuration Reference.