Export Teleport Audit Events to Panther

Panther is a cloud-native security analytics platform. In this guide, we'll explain how to forward Teleport audit events to Panther using Fluentd.

How it works

The Teleport Event Handler is designed to communicate with Fluentd using mTLS to establish a secure channel. In this setup, the Event Handler sends events to Fluentd, which forwards them to S3 to be ingested by Panther.

Prerequisites

• A running Teleport cluster version 15.3.1 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

• The tctl admin tool and tsh client tool.

  On Teleport Enterprise, you must use the Enterprise version of tctl, which you can download from your Teleport account workspace. Otherwise, visit Installation for instructions on downloading tctl and tsh for Teleport Community Edition.

Recommended: Configure Machine ID to provide short-lived Teleport credentials to the plugin. Before following this guide, follow a Machine ID deployment guide to run the tbot binary on your infrastructure.

• A Panther account.
• A server, virtual machine, Kubernetes cluster, or Docker environment to run the Event Handler. The instructions below assume a local Docker container for testing.
• Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new fluent.conf file you can integrate into an existing Fluentd system, or use with a fresh setup.
• An S3 bucket to store the logs. Panther will ingest the logs from this bucket.

The instructions below demonstrate a local test of the Event Handler plugin on VM. You will need to adjust paths, ports, and domains for other environments.

• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:

  tsh login --proxy=teleport.example.com --user=[email protected]
  tctl status
  Cluster  teleport.example.com
  Version  15.3.1
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/7. Install the Event Handler plugin

The Teleport event handler runs alongside the Fluentd forwarder, receives events from Teleport's events API, and forwards them to Fluentd.

curl -L -O https://get.gravitational.com/teleport-event-handler-v15.3.1-linux-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v15.3.1-linux-amd64-bin.tar.gz
sudo ./teleport-event-handler/install

curl -L -O https://get.gravitational.com/teleport-event-handler-v15.3.1-darwin-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v15.3.1-darwin-amd64-bin.tar.gz
sudo ./teleport-event-handler/install

docker pull public.ecr.aws/gravitational/teleport-plugin-event-handler:15.3.1

helm repo add teleport https://charts.releases.teleport.dev

helm repo update

git clone https://github.com/gravitational/teleport-plugins.git --depth 1
cd teleport-plugins/event-handler/build.assets
make build

git clone https://github.com/gravitational/teleport-plugins.git --depth 1
cd teleport-plugins/event-handler
go build

Step 2/7. Generate a plugin configuration

teleport-event-handler configure . mytenant.teleport.sh:443

teleport-event-handler configure . teleport.example.com:443

TELEPORT_CLUSTER_ADDRESS=mytenant.teleport.sh:443
docker run -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:15.3.1 configure . ${TELEPORT_CLUSTER_ADDRESS?}

kubectl create secret generic teleport-event-handler-client-tls --from-file=ca.crt=ca.crt,client.crt=client.crt,client.key=client.key

You'll see the following output:

Teleport event handler 15.3.1

[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml

The plugin generates several setup files:

ls -l
-rw------- 1 bob bob     1038 Jul  1 11:14 ca.crt
-rw------- 1 bob bob     1679 Jul  1 11:14 ca.key
-rw------- 1 bob bob     1042 Jul  1 11:14 client.crt
-rw------- 1 bob bob     1679 Jul  1 11:14 client.key
-rw------- 1 bob bob      541 Jul  1 11:14 fluent.conf
-rw------- 1 bob bob     1078 Jul  1 11:14 server.crt
-rw------- 1 bob bob     1766 Jul  1 11:14 server.key
-rw------- 1 bob bob      260 Jul  1 11:14 teleport-event-handler-role.yaml
-rw------- 1 bob bob      343 Jul  1 11:14 teleport-event-handler.toml

teleport-event-handler configure --cn=handler.example.com --dns-names=forwarder.example.com

Step 3/7. Create a user and role for reading audit events

The teleport-event-handler configure command generated a file called teleport-event-handler-role.yaml. This file defines a teleport-event-handler role and a user with read-only access to the event API:

kind: role
metadata:
  name: teleport-event-handler
spec:
  allow:
    rules:
      - resources: ['event', 'session']
        verbs: ['list','read']
version: v5
---
kind: user
metadata:
  name: teleport-event-handler
spec:
  roles: ['teleport-event-handler']
version: v2

Move this file to your workstation (or recreate it by pasting the snippet above) and use tctl on your workstation to create the role and the user:

tctl create -f teleport-event-handler-role.yaml
user "teleport-event-handler" has been created
role 'teleport-event-handler' has been created

Step 4/7. Create teleport-event-handler credentials

In order for the Event Handler plugin to forward events from your Teleport cluster, it needs signed credentials from the cluster's certificate authority. The teleport-event-handler user cannot request this itself, and should use Teleport Machine ID to obtain the credentials.

Teleport Machine ID bot should leverage the teleport-event-handler role to request the credentials. Machine ID already impersonates roles when requesting credentials from the Teleport cluster.

Export an identity file for the Event Handler plugin user

Give the plugin access to a Teleport identity file. We recommend using Machine ID for this in order to produce short-lived identity files that are less dangerous if exfiltrated, though in demo deployments, you can generate longer-lived identity files with tctl:

outputs:
- type: identity
  destination:
    type: directory
    # For this guide, /opt/machine-id is used as the destination directory.
    # You may wish to customize this. Multiple outputs cannot share the same
    # destination.
    path: /opt/machine-id

outputs:
  - type: identity
    destination:
      type: kubernetes_secret
      name: teleport-event-handler-identity

tctl auth sign --user=teleport-event-handler --out=identity

tsh login --proxy=teleport.example.com --ttl=60060

sudo mkdir -p /var/lib/teleport/api-credentials
sudo mv identity /var/lib/teleport/plugins/api-credentials

kubectl -n teleport create secret generic --from-file=identity teleport-event-handler-identity

Step 5/7. Create a Dockerfile with Fluentd and the S3 plugin

To send logs to Panther, you need to use the Fluentd output plugin for S3. Create a Dockerfile with the following content:

FROM fluent/fluentd:edge
USER root
RUN fluent-gem install fluent-plugin-s3
USER fluent

Build the Docker image:

$ docker build -t fluentd-s3 .

docker run -u $(id -u root):$(id -g root) -p 8888:8888 -v $(pwd):/keys -v \$(pwd)/fluent.conf:/fluentd/etc/fluent.conf --entrypoint=/bin/sh -i --tty  fluentd-s3

Configure Fluentd for Panther

When you run the Teleport Event Handler, it will create a fluent.conf file. This file needs to be updated to send logs to Panther. This means adding a <filter> and <match> section to the file. These sections will filter and format the logs before sending them to S3, The record_transformer is important to send the right date and time format for Panther.

<!--
# Below code is commented out as it's autogenerated in step 4 by teleport-event-handler
fluent.conf 
This is a sample configuration file for Fluentd to send logs to S3. 
Created by the Teleport Event Handler plugin. 
Add the <filter> and <match> sections to the file.
 <source>
     @type http
     port 8888

     <transport tls>
         client_cert_auth true
         ca_path "/keys/ca.crt"
         cert_path "/keys/server.crt"
         private_key_path "/keys/server.key"
         private_key_passphrase "AUTOGENERATED"
     </transport>

     <parse>
       @type json
       json_parser oj

       # This time format is used by Teleport Event Handler.
       time_type string
       time_format %Y-%m-%dT%H:%M:%S
     </parse>

     # If the number of events is high, fluentd will start failing the ingestion
     # with the following error message: buffer space has too many data errors.
     # The following configuration prevents data loss in case of a restart and
     # overcomes the limitations of the default fluentd buffer configuration.
     # This configuration is optional.
     # See https://docs.fluentd.org/configuration/buffer-section for more details.
     <buffer>
       @type file
       flush_thread_count 8
       flush_interval 1s
       chunk_limit_size 10M
       queue_limit_length 16
       retry_max_interval 30
       retry_forever true
  </buffer>
</source>
-->
<filter test.log>
  @type record_transformer
  enable_ruby true
  <record>
    time ${time.utc.strftime("%Y-%m-%dT%H:%M:%SZ")}
  </record>
</filter>
<match test.log>
  @type s3
  aws_key_id  REPLACE_aws_access_key
  aws_sec_key  REPLACE_aws_secret_access_key
  s3_bucket  REPLACE_s3_bucket
  s3_region us-west-2
  path teleport/logs
  <buffer>
    @type file
    path /var/log/fluent/buffer/s3-events
    timekey 60
    timekey_wait 0
    timekey_use_utc true
    chunk_limit_size 256m
  </buffer>
  time_slice_format %Y%m%d%H%M%S
  <format>
    @type json
    </format>
</match>
<match session.*>
  @type stdout
</match>

Start the Fluentd container:

$ docker run -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluentd-s3

This will start the Fluentd container and expose port 8888 for the Teleport Event Handler to send logs to.

Step 6/7. Run the Teleport Event Handler plugin

Configure the Event Handler

In this section, you will configure the Teleport Event Handler for your environment.

storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"

[forward.fluentd]
ca = "/home/bob/event-handler/ca.crt"
cert = "/home/bob/event-handler/client.crt"
key = "/home/bob/event-handler/client.key"
url = "https://fluentd.example.com:8888/test.log"
session-url = "https://fluentd.example.com:8888/session"

[teleport]
addr = "example.teleport.com:443"
identity = "identity"

eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  namespace: "default"

teleport:
  address: "example.teleport.com:443"
  identitySecretName: teleport-event-handler-identity
  identitySecretPath: identity

fluentd:
  url: "https://fluentd.fluentd.svc.cluster.local/events.log"
  sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

Next, modify the configuration file as follows:

If you are providing credentials to the Event Handler using a tbot binary that runs on a Linux server, make sure the value of identity in the Event Handler configuration is the same as the path of the identity file you configured tbot to generate, /opt/machine-id/identity.

Start the Teleport Event Handler

Start the Teleport Teleport Event Handler by following the instructions below.

[Unit]
Description=Teleport Event Handler
After=network.target

[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled true
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-event-handler.pid

[Install]
WantedBy=multi-user.target

sudo systemctl enable teleport-event-handler
sudo systemctl start teleport-event-handler

teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

sudo journalctl -u teleport-event-handler
DEBU   Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-2118:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140...

helm install teleport-plugin-event-handler teleport/teleport-plugin-event-handler \  --values teleport-plugin-event-handler-values.yaml \  --version 15.3.1

The Logs view in Panther should now report your Teleport cluster events.

Step 7/7. Configure Panther to ingest logs from S3

Once logs are being sent to S3, you can configure Panther to ingest them. Follow the Panther documentation to set up the S3 bucket as a data source.

Troubleshooting connection issues

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

• The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the --ttl flag in the tctl auth sign command, which is 12 hours by default.
• Ensure that in your Teleport Event Handler configuration file (teleport-event-handler.toml), you have provided the correct host and port for the Teleport Proxy Service.
• Start the FluentD container prior to starting the Teleport Event Handler. The Event Handler will attempt to connect to FluentD immediately upon startup.

Next steps

• Read more about impersonation here.
• Learn more about the Panther Detections, Alerts and Notifications.
• To see all of the options you can set in the values file for the teleport-plugin-event-handler Helm chart, consult our reference guide.