Export Events with Fluentd

Fluentd is an open-source data collector for a unified logging layer. In this guide, we will:

• Set up Teleport's Event Handler plugin.
• Forward events with Fluentd.

This guide also serves as an explanation for the Teleport Event Handler plugin, using Fluentd as the target service. We'll create a local Docker container as a destination for the Event Handler:

You can follow the instructions below for a local proof-of-concept demo, or use any of the additional installation instructions to configure the Teleport Event Handler to integrate with your infrastructure.

Prerequisites

• Fluentd version v1.12.4 or greater.

• Docker version v20.10.7.

• Make sure you can connect to Teleport. Log in to your cluster using tsh, then use tctl remotely:

  tsh login --proxy=teleport.example.com [email protected]
  tctl status
  Cluster  teleport.example.com
  Version  13.0.3
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  You can run subsequent tctl commands in this guide on your local machine.

  For full privileges, you can also run tctl commands on your Auth Service host.

• On your workstation, create a folder called event-handler, to hold configuration files and plugin state:

  mkdir -p event-handler
  cd event-handler

Step 1/6. Install the event handler plugin

The Teleport event handler runs alongside the Fluentd forwarder, receives events from Teleport's events API, and forwards them to Fluentd.

curl -L -O https://get.gravitational.com/teleport-event-handler-v13.0.3-linux-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v13.0.3-linux-amd64-bin.tar.gz

curl -L -O https://get.gravitational.com/teleport-event-handler-v13.0.3-darwin-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v13.0.3-darwin-amd64-bin.tar.gz

docker pull public.ecr.aws/gravitational/teleport-plugin-event-handler:13.0.3

helm repo add teleport https://charts.releases.teleport.dev

helm repo update

git clone https://github.com/gravitational/teleport-plugins.git --depth 1
cd teleport-plugins/event-handler/build.assets
make build

git clone https://github.com/gravitational/teleport-plugins.git --depth 1
cd teleport-plugins/event-handler
go build

Step 2/6. Configure the plugin

./teleport-event-handler configure . teleport.example.com:443

TELEPORT_CLUSTER_ADDRESS=mytenant.teleport.sh:443
docker run -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:13.0.3 configure . ${TELEPORT_CLUSTER_ADDRESS?}

kubectl create secret generic teleport-event-handler-client-tls --from-file=ca.crt=ca.crt,client.crt=client.crt,client.key=client.key

You'll see the following output:

Teleport event handler 0.0.1 07617b0ad0829db043fe779faf1669defdc8d84e

[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml

Follow-along with our getting started guide:

https://goteleport.com/setup/guides/fluentd

The plugin generates several setup files:

ls -l
-rw------- 1 bob bob     1038 Jul  1 11:14 ca.crt
-rw------- 1 bob bob     1679 Jul  1 11:14 ca.key
-rw------- 1 bob bob     1042 Jul  1 11:14 client.crt
-rw------- 1 bob bob     1679 Jul  1 11:14 client.key
-rw------- 1 bob bob      541 Jul  1 11:14 fluent.conf
-rw------- 1 bob bob     1078 Jul  1 11:14 server.crt
-rw------- 1 bob bob     1766 Jul  1 11:14 server.key
-rw------- 1 bob bob      260 Jul  1 11:14 teleport-event-handler-role.yaml
-rw------- 1 bob bob      343 Jul  1 11:14 teleport-event-handler.toml

If you have an existing Fluentd setup with TLS, issue a client certificate and key from the same certificate authority for the Teleport Event Handler to use.

Step 3/6. Create a user and role for reading audit events

The teleport-event-handler configure command generated a file called teleport-event-handler-role.yaml. This file defines a teleport-event-handler role and a user with read-only access to the event API:

kind: role
metadata:
  name: teleport-event-handler
spec:
  allow:
    rules:
      - resources: ['event', 'session']
        verbs: ['list','read']
version: v5
---
kind: user
metadata:
  name: teleport-event-handler
spec:
  roles: ['teleport-event-handler']
version: v2

Move this file to your workstation (or recreate it by pasting the snippet above) and use tctl on your workstation to create the role and the user:

tctl create -f teleport-event-handler-role.yaml
user "teleport-event-handler" has been created
role 'teleport-event-handler' has been created

Step 4/6. Create teleport-event-handler credentials

Enable impersonation of the Fluentd plugin user

In order for the Fluentd plugin to forward events from your Teleport cluster, it needs a signed identity file from the cluster's certificate authority. The Fluentd user cannot request this itself, and requires another user to impersonate this account in order to request a certificate.

Create a role that enables your user to impersonate the Fluentd user. First, paste the following YAML document into a file called teleport-event-handler-impersonator.yaml:

kind: role
version: v5
metadata:
  name: teleport-event-handler-impersonator
spec:
  # SSH options used for user sessions
  options:
    # max_session_ttl defines the TTL (time to live) of SSH certificates
    # issued to the users with this role.
    max_session_ttl: 10h

  # allow section declares a list of resource/verb combinations that are
  # allowed for the users of this role. by default nothing is allowed.
  allow:
    impersonate:
      users: ["teleport-event-handler"]
      roles: ["teleport-event-handler"]

Next, create the role:

tctl create -f teleport-event-handler-impersonator.yaml

Assign the teleport-event-handler-impersonator role to your Teleport user by running the following commands, depending on whether you authenticate as a local Teleport user or via the github, saml, or oidc authentication connectors:

tctl get users/$(tsh status -f json | jq -r '.active.username') > out.yaml

  roles:
   - access
   - auditor
   - editor
+  - teleport-event-handler-impersonator

tctl create -f out.yaml

tctl get github/github --with-secrets > github.yaml

  teams_to_roles:
    - organization: octocats
      team: admins
      roles:
        - access
+       - teleport-event-handler-impersonator

tctl create -f github.yaml

tctl get --with-secrets saml/mysaml > saml.yaml

  attributes_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - teleport-event-handler-impersonator

tctl create -f saml.yaml

tctl get oidc/myoidc --with-secrets > oidc.yaml

  claims_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - teleport-event-handler-impersonator

tctl create -f oidc.yaml

Log out of your Teleport cluster and log in again to assume the new role.

Export an identity file for the Fluentd plugin user

The Fluentd Teleport plugin uses the teleport-event-handler role and user to read events. We export an identity file for the user with the tctl auth sign command.

tctl auth sign --user=teleport-event-handler --out=identity

Step 5/6. Start the Fluentd forwarder

The Fluentd plugin will send events to your Fluentd instance using keys generated on the previous step.

The fluent.conf file generated earlier configures your Fluentd instance to accept events using TLS and print them:

<source>
    @type http
    port 8888

    <transport tls>
        client_cert_auth true

        # We are going to run fluentd in Docker. /keys will be mounted from the host file system.
        ca_path /keys/ca.crt
        cert_path /keys/server.crt
        private_key_path /keys/server.key
        private_key_passphrase ********** # Passphrase generated along with the keys
    </transport>

    <parse>
      @type json
      json_parser oj

      # This time format is used by the plugin. This field is required.
      time_type string
      time_format %Y-%m-%dT%H:%M:%S
    </parse>

    # If the number of events is high, fluentd will start failing the ingestion
    # with the following error message: buffer space has too many data errors.
    # The following configuration prevents data loss in case of a restart and
    # overcomes the limitations of the default fluentd buffer configuration.
    # This configuration is optional.
    # See https://docs.fluentd.org/configuration/buffer-section for more details.
    <buffer>
      @type file
      flush_thread_count 8
      flush_interval 1s
      chunk_limit_size 10M
      queue_limit_length 16
      retry_max_interval 30
      retry_forever true
    </buffer>
</source>

# Events sent to test.log will be dumped to STDOUT.
<match test.log>
  @type stdout
</match>

To try out this Fluentd configuration, start your fluentd instance:

docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge

This will consume your current terminal, so open a new one to continue following along.

Step 6/6. Start the event handler plugin

Earlier, we generated a file called teleport-event-handler.toml to configure the Fluentd event handler. This file includes setting similar to the following:

storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"

[forward.fluentd]
ca = "/home/bob/event-handler/ca.crt"
cert = "/home/bob/event-handler/client.crt"
key = "/home/bob/event-handler/client.key"
url = "https://localhost:8888/test.log"

[teleport]
addr = "example.teleport.com:443"
identity = "identity"

./teleport-event-handler start --config teleport-event-handler.toml

eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  namespace: "default"

teleport:
  address: "example.teleport.com:443"
  identitySecretName: teleport-event-handler-identity

fluentd:
  url: "https://fluentd.fluentd.svc.cluster.local/events.log"
  sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

helm install teleport-plugin-event-handler teleport/teleport-plugin-event-handler \  --values teleport-plugin-event-handler-values.yaml \  --version 13.0.3

./teleport-event-handler start --config teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

Once the handler starts, you will see notifications about scanned and forwarded events:

INFO[0046] Event sent id=0b5f2a3e-faa5-4d77-ab6e-362bca0994fc ts="2021-06-08 11:00:56.034 +0000 UTC" type=user.login
...

Troubleshooting connection issues

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

• The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the --ttl flag in the tctl auth sign command, which is 12 hours by default.
• Ensure that in your Teleport Event Handler configuration file (teleport-event-handler.toml), you have provided the correct host and port for the Teleport Proxy Service or Auth Service.

Next Steps

Read more about impersonation here.

While this guide uses the tctl auth sign command to issue credentials for the Teleport Event Handler, production clusters should use Machine ID for safer, more reliable renewals. Read our guide to getting started with Machine ID.

To see all of the options you can set in the values file for the teleport-plugin-event-handler Helm chart, consult our reference guide.