Fork me on GitHub
Teleport

Enhanced Session Recording with BPF

Improve
Enhanced SSH Session Recording with BPF

Enhanced SSH Session Recording with BPF

Length: 05:01

This guide explains Enhanced Session Recording for SSH with BPF and how to set it up in your Teleport cluster.

Teleport's default SSH and Kubernetes session recording feature captures what is echoed to a terminal.

This has inherent advantages. For example, because no input is captured, Teleport session recordings typically do not contain passwords that were entered into a terminal.

The disadvantage is that there are several techniques for rendering session recordings less useful:

  • Obfuscation. For example, even though the command echo Y3VybCBodHRwOi8vd3d3LmV4YW1wbGUuY29tCg== | base64 --decode | sh does not contain curl http://www.example.com, when decoded, that is what is run.
  • Shell scripts. For example, if a user uploads and executes a script, the commands run within the script are not captured, only the output.
  • Terminal controls. Terminals support a wide variety of controls including the ability for users to disable terminal echo. This is frequently used when requesting credentials. Disabling terminal echo allows commands to be run without being captured.

Furthermore, due to their unstructured nature, session recordings are difficult to ingest and perform monitoring and alerting on.

Teleport Enhanced Session Recording mitigates all three concerns by providing advanced security and greater logging capabilities, and better correlates a user with their activities.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 9.2.3.

    tctl version

    Teleport v9.2.3 go1.17

    tsh version

    Teleport v9.2.3 go1.17

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 9.2.3, which you can download by visiting the customer portal.

    tctl version

    Teleport v9.2.3 go1.17

    tsh version

    Teleport v9.2.3 go1.17

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 9.1.2. To download these tools, visit the Downloads page.

    tctl version

    Teleport v9.1.2 go1.17

    tsh version

    Teleport v9.1.2 go1.17

  • At least one host that you will use to run the Teleport Node Service. The host must run Linux kernel 5.8 (or above).

    You can check your kernel version using the uname command. The output should look something like the following.

    uname -r

    5.8.17

    See below for more details on the required versions for your Linux kernel and distribution.

Our Standard Session Recording works with older Linux kernels. View Teleport Nodes for more details.

Linux distributions and supported kernels

Distro nameDistro versionKernel version
Ubuntu "Groovy Gorilla"20.105.8+
Fedora335.8+
Archlinux2020.09.015.8.5+
Flatcar2765.2.25.10.25+

Verify that your Teleport client is connected by running the following on your Auth Service host:

tctl status

Cluster tele.example.com

Version 9.2.3

CA pin sha256:sha-hash-here

Remain logged in to your Auth Service host so you can run subsequent tctl commands in this guide.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 9.1.2

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Step 1/2. Configure a Teleport Node

Install Teleport on your Node

On the host where you will run your Teleport Node, follow the instructions for your environment to install Teleport.

Download Teleport's PGP public key

sudo curl https://deb.releases.teleport.dev/teleport-pubkey.asc \ -o /usr/share/keyrings/teleport-archive-keyring.asc

Add the Teleport APT repository

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] https://deb.releases.teleport.dev/ stable main" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null
sudo apt-get update
sudo apt-get install teleport
sudo yum-config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo
sudo yum install teleport

Optional: Using DNF on newer distributions

$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo

$ sudo dnf install teleport

curl https://get.gravitational.com/teleport-v9.2.3-linux-amd64-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v9.2.3-linux-amd64-bin.tar.gz
shasum -a 256 teleport-v9.2.3-linux-amd64-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v9.2.3-linux-amd64-bin.tar.gz
cd teleport
sudo ./install
curl https://get.gravitational.com/teleport-v9.2.3-linux-arm-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v9.2.3-linux-arm-bin.tar.gz
shasum -a 256 teleport-v9.2.3-linux-arm-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v9.2.3-linux-arm-bin.tar.gz
cd teleport
sudo ./install
curl https://get.gravitational.com/teleport-v9.2.3-linux-arm64-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://get.gravitational.com/teleport-v9.2.3-linux-arm64-bin.tar.gz
shasum -a 256 teleport-v9.2.3-linux-arm64-bin.tar.gz

Verify that the checksums match

tar -xzf teleport-v9.2.3-linux-arm64-bin.tar.gz
cd teleport
sudo ./install

Generate a token

Use the tctl tool to generate an invite token that your Node will use to join the cluster. In the following example, a new token is created with a TTL of five minutes:

Generate a short-lived invitation token for a new node:

tctl nodes add --ttl=5m --roles=node

The invite token: abcd123-insecure-do-not-use-this

You can also list all generated non-expired tokens:

tctl tokens ls

Token Type Expiry Time

------------------------ ----------- ---------------

abcd123-insecure-do-not-use-this Node 25 Sep 18 00:21 UTC

... or revoke an invitation token before it's used:

tctl tokens rm abcd123-insecure-do-not-use-this

Create a configuration file

Set up your Teleport Node with the following content in /etc/teleport.yaml.

# Example config to be saved as etc/teleport.yaml
teleport:
  nodename: graviton-node
  # The token you created earlier
  auth_token: abcd123-insecure-do-not-use-this
  auth_servers:
  # Replace with the address of the Teleport Auth Service or Proxy Service.
  - 127.0.0.1:3025
  data_dir: /var/lib/teleport
proxy_service:
  enabled: false
auth_service:
  enabled: false
ssh_service:
  enabled: true
  enhanced_recording:
    # Enable or disable enhanced auditing for this node. Default value: false.
    enabled: true

    # Optional: command_buffer_size is optional with a default value of 8 pages.
    command_buffer_size: 8

    # Optional: disk_buffer_size is optional with default value of 128 pages.
    disk_buffer_size: 128

    # Optional: network_buffer_size is optional with default value of 8 pages.
    network_buffer_size: 8

    # Optional: Controls where cgroupv2 hierarchy is mounted. Default value:
    # /cgroup2.
    cgroup_path: /cgroup2

Start Teleport on your Node

teleport start

Step 2/2. Inspect the audit log

Enhanced session recording events will be shown in Teleport's audit log, which you can inspect by visiting Teleport's Web UI.

Events emitted via Enhanced Session Recording will include the "enhanced_recording": true field in events similar to the following:

{
  "code": "T2004I",
  "ei": 23,
  "enhanced_recording": true,
  "event": "session.end",
  "interactive": true,
  "namespace": "default",
  "participants": [
    "benarent"
  ],
  "server_id": "585fc225-5cf9-4e9f-8ff6-1b0fd6885b09",
  "sid": "ca82b98d-1d30-11ea-8244-cafde5327a6c",
  "time": "2019-12-12T22:44:46.218Z",
  "uid": "83e67464-a93a-4c7c-8ce6-5a3d8802c3b2",
  "user": "benarent"
}

If your Teleport cluster uses a file-based event log, you can examine your audit log on the Teleport Auth Service host.

Teleport's session recordings backend is configured via the teleport.storage.audit_sessions_uri field. If a provided URI includes a scheme that belongs to a cloud-based service (e.g., s3:// or dynamodb://), you will not be able to inspect session recordings in the filesystem of your Auth Service host.

Examine the contents of /var/lib/teleport/log as shown below:

teleport-auth ~: tree /var/lib/teleport/log

/var/lib/teleport/log

├── 1048a649-8f3f-4431-9529-0c53339b65a5

│   ├── 2020-01-13.00:00:00.log

│   └── sessions

│   └── default

│   ├── fad07202-35bb-11ea-83aa-125400432324-0.chunks.gz

│   ├── fad07202-35bb-11ea-83aa-125400432324-0.events.gz

│   ├── fad07202-35bb-11ea-83aa-125400432324-0.session.command-events.gz

│   ├── fad07202-35bb-11ea-83aa-125400432324-0.session.network-events.gz

│   └── fad07202-35bb-11ea-83aa-125400432324.index

├── events.log -> /var/lib/teleport/log/1048a649-8f3f-4431-9529-0c53339b65a5/2020-01-13.00:00:00.log

├── playbacks

│   └── sessions

│   └── default

└── upload

└── sessions

└── default

To quickly check the status of the audit log, you can simply tail the logs with tail -f /var/lib/teleport/log/events.log. The resulting capture from Teleport will be a JSON log for each command and network request.

Enhanced session recording events will be shown in Teleport's audit log, which you can inspect by visiting Teleport's Web UI.

Sessions with Enhanced Session Recording will include the "enhanced_recording": true field in events similar to the following:

{
  "code": "T2004I",
  "ei": 23,
  "enhanced_recording": true,
  "event": "session.end",
  "interactive": true,
  "namespace": "default",
  "participants": [
    "benarent"
  ],
  "server_id": "585fc225-5cf9-4e9f-8ff6-1b0fd6885b09",
  "sid": "ca82b98d-1d30-11ea-8244-cafde5327a6c",
  "time": "2019-12-12T22:44:46.218Z",
  "uid": "83e67464-a93a-4c7c-8ce6-5a3d8802c3b2",
  "user": "benarent"
}

Next steps