Fork me on GitHub

Teleport

Forwarding events with Fluentd

Improve
Forwarding Access Logs using FluentD

Forwarding Access Logs using FluentD

Length: 18:36

In this guide, we will explain how to:

  • Set up Teleport's event handler.
  • Forward events with Fluentd.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 10.3.1.

    tctl version

    Teleport v10.3.1 go1.18

    tsh version

    Teleport v10.3.1 go1.18

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 10.3.1, which you can download by visiting the customer portal.

    tctl version

    Teleport v10.3.1 go1.18

    tsh version

    Teleport v10.3.1 go1.18

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.2.2. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.2.2 go1.18

    tsh version

    Teleport v10.2.2 go1.18

  • Fluentd version v1.12.4.
  • Docker version v20.10.7.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 10.3.1

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 10.2.2

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Create a folder called fluentd to hold configuration and plugin state:

mkdir -p event-handler
cd event-handler

Step 1/6. Install the event handler plugin

The Teleport event handler runs alongside the Fluentd forwarder, receives events from Teleport's events API, and forwards them to Fluentd.

curl -L -O https://get.gravitational.com/teleport-event-handler-v10.3.1-linux-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v10.3.1-linux-amd64-bin.tar.gz
curl -L -O https://get.gravitational.com/teleport-event-handler-v10.3.1-darwin-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v10.3.1-darwin-amd64-bin.tar.gz

We currently only build the event handler plugin for amd64 machines. If your macOS machine uses Apple silicon, you will need to install Rosetta before you can run the event handler plugin. You can also build from source.

docker pull public.ecr.aws/gravitational/teleport-plugin-event-handler:10.3.1
helm repo add teleport https://charts.releases.teleport.dev/

Ensure that you have Docker installed and running.

Run the following commands to build the plugin:

git clone https://github.com/gravitational/teleport-plugins.git --depth 1
cd teleport-plugins/event-handler/build.assets
make build

You can find the compiled binary within your clone of the teleport-plugins repo, with the file path, event-handler/build/teleport-event-handler.

Step 2/6. Configure the plugin

Run the configure command to generate a sample configuration. Replace teleport.example.com:443 with the DNS name and HTTPS port of Teleport's Proxy Service:

./teleport-event-handler configure . teleport.example.com:443

Run the configure command to generate a sample configuration. Replace mytenant.teleport.sh with the DNS name of your Teleport Cloud tenant:

./teleport-event-handler configure . mytenant.teleport.sh

Run the configure command to generate a sample configuration. Assign TELEPORT_CLUSTER_ADDRESS to the DNS name and port of your Teleport Auth Service or Proxy Service:

TELEPORT_CLUSTER_ADDRESS=mytenant.teleport.sh:443
docker run -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin quay.io/gravitational/teleport-plugin-event-handler:10.3.1 configure . ${TELEPORT_CLUSTER_ADDRESS?}

In order to connect to Fluentd, you'll need to have the root certificate and the client credentials available as a secret. Use the following command to create that secret in Kubernetes:

kubectl create secret generic teleport-event-handler-client-tls --from-file=ca.crt=ca.crt,client.crt=client.crt,client.key=client.key

This will pack the content of ca.crt, client.crt, and client.key into the secret, so the Helm Chart can mount them to their appropriate path.

You'll see the following output:

Teleport event handler 0.0.1 07617b0ad0829db043fe779faf1669defdc8d84e

[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml

Follow-along with our getting started guide:

https://goteleport.com/setup/guides/fluentd

The plugin generates several setup files:

ls -l

-rw------- 1 bob bob 1038 Jul 1 11:14 ca.crt

-rw------- 1 bob bob 1679 Jul 1 11:14 ca.key

-rw------- 1 bob bob 1042 Jul 1 11:14 client.crt

-rw------- 1 bob bob 1679 Jul 1 11:14 client.key

-rw------- 1 bob bob 541 Jul 1 11:14 fluent.conf

-rw------- 1 bob bob 1078 Jul 1 11:14 server.crt

-rw------- 1 bob bob 1766 Jul 1 11:14 server.key

-rw------- 1 bob bob 260 Jul 1 11:14 teleport-event-handler-role.yaml

-rw------- 1 bob bob 343 Jul 1 11:14 teleport-event-handler.toml

File(s)Purpose
ca.crt and ca.keySelf-signed CA certificate and private key for Fluentd
server.crt and server.keyFluentd server certificate and key
client.crt and client.keyFluentd client certificate and key, all signed by the generated CA
teleport-event-handler-role.yamluser and role resource definitions for Teleport's event handler
fluent.confFluentd plugin configuration

Step 3/6. Create a user and role for reading audit events

The configure command generates a file called teleport-event-handler-role.yaml that defines a teleport-event-handler role and a user with read-only access to the event API:

kind: user
metadata:
  name: teleport-event-handler
spec:
  roles: ['teleport-event-handler']
version: v2
---
kind: role
metadata:
  name: teleport-event-handler
spec:
  allow:
    rules:
      - resources: ['event']
        verbs: ['list','read']
version: v5

Use tctl to create the role and the user:

tctl create -f teleport-event-handler-role.yaml

user "teleport-event-handler" has been created

role 'teleport-event-handler' has been created

Step 4/6. Create teleport-event-handler credentials

Enable impersonation of the Fluentd plugin user

In order for the Fluentd plugin to forward events from your Teleport cluster, it needs a signed identity file from the cluster's certificate authority. The Fluentd user cannot request this itself, and requires another user to impersonate this account in order to request a certificate.

Create a role that enables your user to impersonate the Fluentd user. First, paste the following YAML document into a file called teleport-event-handler-impersonator.yaml:

kind: role
version: v5
metadata:
  name: teleport-event-handler-impersonator
spec:
  # SSH options used for user sessions
  options:
    # max_session_ttl defines the TTL (time to live) of SSH certificates
    # issued to the users with this role.
    max_session_ttl: 10h

  # allow section declares a list of resource/verb combinations that are
  # allowed for the users of this role. by default nothing is allowed.
  allow:
    impersonate:
      users: ["teleport-event-handler"]
      roles: ["teleport-event-handler"]

Next, create the role:

tctl create teleport-event-handler-impersonator.yaml

Assign this role to the current user by running the following command, substituting the user as appropriate:

tctl get users/<USER> > out.yaml

Now edit out.yaml, adding teleport-event-handler-impersonator to the list of existing roles and update via tctl again:

$ tctl create -f out.yaml

Log in to your Teleport cluster again to assume the new role.

Export an identity file for the Fluentd plugin user

The Fluentd Teleport plugin uses the teleport-event-handler role and user to read events. We export an identity file for the user with the tctl auth sign command.

tctl auth sign --format=tls --user=teleport-event-handler --out=auth

This command should result in three PEM-encoded files: auth.crt, auth.key, and auth.cas (certificate, private key, and CA certs, respectively).

tctl auth sign --user=teleport-event-handler --out=identity

The above sequence should result in one PEM-encoded file, identity.

If you are planning to use the Helm Chart, you'll need to generate the keys with the file format, then create a secret in Kubernetes.

Create the identity using the following command:

tctl auth sign --format=file --user=teleport-event-handler --out=identity

Then create the Kubernetes secret:

kubectl create secret generic teleport-event-handler-identity --from-file=auth_id=identity

These commands should result in a PEM-encoded file, identity, and a secret in Kubernetes with the name teleport-event-handler-identity.

Step 5/6. Start the Fluentd forwarder

The Fluentd plugin will send events to your Fluentd instance using keys generated on the previous step.

The fluent.conf file generated earlier configures your Fluentd instance to accept events using TLS and print them:

<source>
    @type http
    port 8888

    <transport tls>
        client_cert_auth true

        # We are going to run fluentd in Docker. /keys will be mounted from the host file system.
        ca_path /keys/ca.crt
        cert_path /keys/server.crt
        private_key_path /keys/server.key
        private_key_passphrase ********** # Passphrase generated along with the keys
    </transport>

    <parse>
      @type json
      json_parser oj

      # This time format is used by the plugin. This field is required.
      time_type string
      time_format %Y-%m-%dT%H:%M:%S
    </parse>
</source>

# Events sent to test.log will be dumped to STDOUT.
<match test.log>
  @type stdout
</match>

To try out this Fluentd configuration, start your fluentd instance:

docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge

Step 6/6. Start the event handler plugin

Earlier, we generated a file called teleport-event-handler.toml to configure the Fluentd event handler. This file includes setting similar to the following:

storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"

[forward.fluentd]
ca = "/home/sasha/scripts/event-handler/ca.crt"
cert = "/home/sasha/scripts/event-handler/client.crt"
key = "/home/sasha/scripts/event-handler/client.key"
url = "https://localhost:8888/test.log"

[teleport]
addr = "example.teleport.com:443"
identity = "identity"

To start the event handler, run the following command:

./teleport-event-handler start --config teleport-event-handler.toml
storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"

[forward.fluentd]
ca = "/home/sasha/scripts/event-handler/ca.crt"
cert = "/home/sasha/scripts/event-handler/client.crt"
key = "/home/sasha/scripts/event-handler/client.key"
url = "https://localhost:8888/test.log"

[teleport]
addr = "example.teleport.com:443"
client_key = "auth.key"
client_crt = "auth.crt"
root_cas = "auth.cas"

To start the event handler, run the following command:

./teleport-event-handler start --config teleport-event-handler.toml

Use the following template to create teleport-plugin-event-handler-values.yaml:

eventHandler:
  storagePath: "./storage"
  timeout: "10s"
  batch: 20
  namespace: "default"

teleport:
  address: "example.teleport.com:443"
  identitySecretName: teleport-event-handler-identity

fluentd:
  url: "https://fluentd.fluentd.svc.cluster.local/events.log"
  sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
  certificate:
    secretName: "teleport-event-handler-client-tls"
    caPath: "ca.crt"
    certPath: "client.crt"
    keyPath: "client.key"

persistentVolumeClaim:
  enabled: true

To start the event handler in Kubernetes, run the following command:

helm install teleport-plugin-event-handler teleport/teleport-plugin-event-handler --values teleport-plugin-event-handler-values.yaml
Note

This example will start exporting from May 5th 2021:

./teleport-event-handler start --config teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

The start time can be set only once, on the first run of the tool.

If you want to change the time frame later, remove the plugin state directory that you specified in the storage field of the handler's configuration file.

Once the handler starts, you will see notifications about scanned and forwarded events:

INFO[0046] Event sent id=0b5f2a3e-faa5-4d77-ab6e-362bca0994fc ts="2021-06-08 11:00:56.034 +0000 UTC" type=user.login
...

Next Steps