Fork me on GitHub
Teleport

Forwarding events with Fluentd

This section will cover:

  • Setting up Teleport's event handler.
  • Forwarding events with Fluentd for Cloud, Enterprise and Open Source editions.

Prerequisites

  • Teleport v7.1.3 Cloud, Open Source or Enterprise
  • Teleport admin tool tctl configured to access the cluster.
  • Fluentd version v1.12.4.
  • Docker version v20.10.7.
terraform version

Terraform v0.12.0

tctl version

Teleport v7.1.3 go(teleport.golang)

Test connectivity and admin permissions

tctl get roles

Create a folder fluentd to hold configuration and plugin state:

mkdir -p event-handler
cd event-handler

Step 1/6. Install event handler plugin

Teleport event handler runs alongside fluentd forwarder, receives events from Teleport's events API and forwards them to fluentd.

curl -L -O https://get.gravitational.com/teleport-event-handler-v7.1.3-linux-amd64-bin.tar.gz
tar -zxvf teleport-event-handler-v7.1.3-linux-amd64-bin.tar.gz -C .

Step 2/6. Generate configuration

Run configure command to generate sample configuration. Replace teleport.example.com:443 with the DNS name and web proxy port of Teleport's proxy:

./teleport-event-handler configure . teleport.example.com:443

You'll see the following output:

Teleport event handler 0.0.1 07617b0ad0829db043fe779faf1669defdc8d84e

[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml

Follow-along with our getting started guide:

https://goteleport.com/setup/guides/fluentd

Plugin has generated several setup files:

ls -l

-rw------- 1 bob bob 1038 Jul 1 11:14 ca.crt

-rw------- 1 bob bob 1679 Jul 1 11:14 ca.key

-rw------- 1 bob bob 1042 Jul 1 11:14 client.crt

-rw------- 1 bob bob 1679 Jul 1 11:14 client.key

-rw------- 1 bob bob 541 Jul 1 11:14 fluent.conf

-rw------- 1 bob bob 1078 Jul 1 11:14 server.crt

-rw------- 1 bob bob 1766 Jul 1 11:14 server.key

-rw------- 1 bob bob 260 Jul 1 11:14 teleport-event-handler-role.yaml

-rw------- 1 bob bob 343 Jul 1 11:14 teleport-event-handler.toml

  • ca.crt and ca.key is a fluentd self-signed CA certificate and a private key.
  • server.crt and server.key is a fluentd server certificate and key.
  • client.crt and client.key is a fluentd client certificate and key, all signed by the generated CA.
  • teleport-event-handler-role.yaml is a Teleport's event handler client user and role.
  • fluent.conf is a fluentd plugin configuration.

Step 3/6. Create user and role for reading audit events

configure command generated teleport-event-handler-role.yaml that defines a teleport-event-handler role and a user with read-only access to the event API:

kind: user
metadata:
  name: teleport-event-handler
spec:
  roles: ['teleport-event-handler']
version: v2
---
kind: role
metadata:
  name: teleport-event-handler
spec:
  allow:
    rules:
      - resources: ['event']
        verbs: ['list','read']
version: v4

Use tctl to create the role and the user:

tctl create -f teleport-event-handler-role.yaml

user "teleport-event-handler" has been created

role 'teleport-event-handler' has been created

Step 4/6. Create teleport-event-handler credentials

Teleport Plugin use the fluentd role and user to read the events. We export the identity files, using tctl auth sign.

tctl auth sign --out identity --user teleport-event-handler

The credentials have been written to identity

This will generate identity which contains TLS certificates and will be used to connect plugin to your Teleport instance.

Note

Clients, for example cloud interactive users, could be missing impersonation privileges when trying to use tctl auth sign, will get the following error:

ERROR: access denied: impersonation is not allowed

Create the following file with role: teleport-event-handler-impersonator.yaml:

kind: role
version: v4
metadata:
  name: teleport-event-handler-impersonator
spec:
  # SSH options used for user sessions 
  options:
    # max_session_ttl defines the TTL (time to live) of SSH certificates 
    # issued to the users with this role.
    max_session_ttl: 10h

  # allow section declares a list of resource/verb combinations that are
  # allowed for the users of this role. by default nothing is allowed.
  allow:
    impersonate:
      users: ['teleport-event-handler']
      roles: ['teleport-event-handler']

  # the deny section uses the identical format as the 'allow' section.
  # the deny rules always override allow rules.
  deny:
    node_labels:
      '*': '*'
tctl create teleport-event-handler-impersonator.yaml

Assign this role to the current user. Re-login to assume the new role and try to issue certificate for terraform user again.

Step 5/6. Start fluentd forwarder

The plugin will send events to the fluentd instance using keys generated on the previous step.

Generated sample fluent.conf file sets accepts events using TLS and prints them:

<source>
    @type http
    port 8888

    <transport tls>
        client_cert_auth true

        # We are going to run fluentd in Docker. /keys will be mounted from the host file system.
        ca_path /keys/ca.crt
        cert_path /keys/server.crt
        private_key_path /keys/server.key
        private_key_passphrase ********** # Passphrase generated along with the keys
    </transport>

    <parse>
      @type json
      json_parser oj

      # This time format is used by the plugin. This field is required.
      time_type string
      time_format %Y-%m-%dT%H:%M:%S
    </parse>
</source>

# Events sent to test.log will be dumped to STDOUT.
<match test.log> 
  @type stdout
</match>

To try it out, start fluentd instance:

docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge

Step 6/6. Start the event handler plugin

configure command generated teleport-event-handler.toml configuration file for the event handler:

storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"

[forward.fluentd]
ca = "/home/sasha/scripts/event-handler/ca.crt"
cert = "/home/sasha/scripts/event-handler/client.crt"
key = "/home/sasha/scripts/event-handler/client.key"
url = "https://localhost:8888/test.log"

[teleport]
addr = "example.teleport.com:443"
identity = "identity"

Start an export from the current moment:

./teleport-event-handler start --config teleport-event-handler.toml
Note

This example will start export from May 5th 2021:

./teleport-event-handler start --config teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

Start time can be set only once, on the first run of the tool.

If you want to change the time frame later, remove plugin state dir which you had specified in storage-dir argument.

Once handler starts, you will see notifications about scanned and forwarded events:

INFO[0046] Event sent id=0b5f2a3e-faa5-4d77-ab6e-362bca0994fc ts="2021-06-08 11:00:56.034 +0000 UTC" type=user.login
...

Next Steps

Have a suggestion or can’t find something?
IMPROVE THE DOCS