

Forwarding Access Logs using FluentD
Length: 18:36
Fluentd is an open-source data collector for a unified logging layer. In this guide, we will:
- Set up Teleport's Event Handler plugin.
- Forward events with Fluentd.
This guide also serves as an explanation for the Teleport Event Handler plugin, using Fluentd as the target service. We'll create a local Docker container as a destination for the Event Handler:
You can follow the instructions below for a local proof-of-concept demo, or use any of the additional installation instructions to configure the Teleport Event Handler to integrate with your infrastructure.
Prerequisites
-
A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.
-
The
tctl
admin tool andtsh
client tool version >= 13.0.3.tctl versionTeleport v13.0.3 go1.20
tsh versionTeleport v13.0.3 go1.20
See Installation for details.
-
A running Teleport Enterprise cluster. For details on how to set this up, see our Enterprise Getting Started guide.
-
The Enterprise
tctl
admin tool andtsh
client tool version >= 13.0.3, which you can download by visiting your Teleport account.tctl versionTeleport Enterprise v13.0.3 go1.20
tsh versionTeleport v13.0.3 go1.20
Please use the latest version of Teleport Enterprise documentation.
-
Fluentd version v1.12.4 or greater.
-
Docker version v20.10.7.
-
Make sure you can connect to Teleport. Log in to your cluster using
tsh
, then usetctl
remotely:tsh login --proxy=teleport.example.com [email protected]tctl statusCluster teleport.example.com
Version 13.0.3
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
You can run subsequent
tctl
commands in this guide on your local machine.For full privileges, you can also run
tctl
commands on your Auth Service host. -
On your workstation, create a folder called
event-handler
, to hold configuration files and plugin state:mkdir -p event-handlercd event-handler
Step 1/6. Install the event handler plugin
The Teleport event handler runs alongside the Fluentd forwarder, receives events from Teleport's events API, and forwards them to Fluentd.
curl -L -O https://get.gravitational.com/teleport-event-handler-v13.0.3-linux-amd64-bin.tar.gztar -zxvf teleport-event-handler-v13.0.3-linux-amd64-bin.tar.gz
We currently only build the Event Handler plugin for amd64 machines. For ARM architecture, you can build from source.
curl -L -O https://get.gravitational.com/teleport-event-handler-v13.0.3-darwin-amd64-bin.tar.gztar -zxvf teleport-event-handler-v13.0.3-darwin-amd64-bin.tar.gz
We currently only build the event handler plugin for amd64 machines. If your macOS machine uses Apple silicon, you will need to install Rosetta before you can run the event handler plugin. You can also build from source.
Ensure that you have Docker installed and running.
docker pull public.ecr.aws/gravitational/teleport-plugin-event-handler:13.0.3
To allow Helm to install charts that are hosted in the Teleport Helm repository, use helm repo add
:
helm repo add teleport https://charts.releases.teleport.dev
To update the cache of charts from the remote repository, run helm repo update
:
helm repo update
Ensure that you have Docker installed and running.
Run the following commands to build the plugin:
git clone https://github.com/gravitational/teleport-plugins.git --depth 1cd teleport-plugins/event-handler/build.assetsmake build
You can find the compiled binary within your clone of the teleport-plugins
repo, with the file path, event-handler/build/teleport-event-handler
.
You will need Go >= 1.20 installed.
Run the following commands on your Universal Forwarder host:
git clone https://github.com/gravitational/teleport-plugins.git --depth 1cd teleport-plugins/event-handlergo build
The resulting executable will have the name event-handler
. To follow the
rest of this guide, rename this file to teleport-event-handler
and move it
to /usr/local/bin
.
Step 2/6. Configure the plugin
Please use the latest version of Teleport Enterprise documentation.
Run the configure
command to generate a sample configuration. Replace
teleport.example.com:443
with the DNS name and HTTPS port of Teleport's Proxy
Service:
./teleport-event-handler configure . teleport.example.com:443
Run the configure
command to generate a sample configuration. Assign
TELEPORT_CLUSTER_ADDRESS
to the DNS name and port of your Teleport Auth
Service or Proxy Service:
TELEPORT_CLUSTER_ADDRESS=mytenant.teleport.sh:443docker run -v `pwd`:/opt/teleport-plugin -w /opt/teleport-plugin public.ecr.aws/gravitational/teleport-plugin-event-handler:13.0.3 configure . ${TELEPORT_CLUSTER_ADDRESS?}
In order to export audit events, you'll need to have the root certificate and the client credentials available as a secret. Use the following command to create that secret in Kubernetes:
kubectl create secret generic teleport-event-handler-client-tls --from-file=ca.crt=ca.crt,client.crt=client.crt,client.key=client.key
This will pack the content of ca.crt
, client.crt
, and client.key
into the
secret so the Helm chart can mount them to their appropriate path.
You'll see the following output:
Teleport event handler 0.0.1 07617b0ad0829db043fe779faf1669defdc8d84e
[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml
Follow-along with our getting started guide:
https://goteleport.com/setup/guides/fluentd
The plugin generates several setup files:
ls -l-rw------- 1 bob bob 1038 Jul 1 11:14 ca.crt
-rw------- 1 bob bob 1679 Jul 1 11:14 ca.key
-rw------- 1 bob bob 1042 Jul 1 11:14 client.crt
-rw------- 1 bob bob 1679 Jul 1 11:14 client.key
-rw------- 1 bob bob 541 Jul 1 11:14 fluent.conf
-rw------- 1 bob bob 1078 Jul 1 11:14 server.crt
-rw------- 1 bob bob 1766 Jul 1 11:14 server.key
-rw------- 1 bob bob 260 Jul 1 11:14 teleport-event-handler-role.yaml
-rw------- 1 bob bob 343 Jul 1 11:14 teleport-event-handler.toml
File(s) | Purpose |
---|---|
ca.crt and ca.key | Self-signed CA certificate and private key for Fluentd |
server.crt and server.key | Fluentd server certificate and key |
client.crt and client.key | Fluentd client certificate and key, all signed by the generated CA |
teleport-event-handler-role.yaml | user and role resource definitions for Teleport's event handler |
fluent.conf | Fluentd plugin configuration |
If you have an existing Fluentd setup with TLS, issue a client certificate and key from the same certificate authority for the Teleport Event Handler to use.
Step 3/6. Create a user and role for reading audit events
The teleport-event-handler configure
command generated a file called
teleport-event-handler-role.yaml
. This file defines a teleport-event-handler
role and a user with read-only access to the event
API:
kind: role
metadata:
name: teleport-event-handler
spec:
allow:
rules:
- resources: ['event', 'session']
verbs: ['list','read']
version: v5
---
kind: user
metadata:
name: teleport-event-handler
spec:
roles: ['teleport-event-handler']
version: v2
Move this file to your workstation (or recreate it by pasting the snippet above)
and use tctl
on your workstation to create the role and the user:
tctl create -f teleport-event-handler-role.yamluser "teleport-event-handler" has been created
role 'teleport-event-handler' has been created
Step 4/6. Create teleport-event-handler credentials
Enable impersonation of the Fluentd plugin user
In order for the Fluentd plugin to forward events from your Teleport cluster, it needs a signed identity file from the cluster's certificate authority. The Fluentd user cannot request this itself, and requires another user to impersonate this account in order to request a certificate.
Create a role that enables your user to impersonate the Fluentd user. First,
paste the following YAML document into a file called
teleport-event-handler-impersonator.yaml
:
kind: role
version: v5
metadata:
name: teleport-event-handler-impersonator
spec:
# SSH options used for user sessions
options:
# max_session_ttl defines the TTL (time to live) of SSH certificates
# issued to the users with this role.
max_session_ttl: 10h
# allow section declares a list of resource/verb combinations that are
# allowed for the users of this role. by default nothing is allowed.
allow:
impersonate:
users: ["teleport-event-handler"]
roles: ["teleport-event-handler"]
Next, create the role:
tctl create -f teleport-event-handler-impersonator.yaml
Assign the teleport-event-handler-impersonator
role to your Teleport user by running the following
commands, depending on whether you authenticate as a local Teleport user or via
the github
, saml
, or oidc
authentication connectors:
Retrieve your local user's configuration resource:
tctl get users/$(tsh status -f json | jq -r '.active.username') > out.yaml
Edit out.yaml
, adding teleport-event-handler-impersonator
to the list of existing roles:
roles:
- access
- auditor
- editor
+ - teleport-event-handler-impersonator
Apply your changes:
tctl create -f out.yaml
Retrieve your github
configuration resource:
tctl get github/github --with-secrets > github.yaml
Edit github.yaml
, adding teleport-event-handler-impersonator
to the
teams_to_roles
section. The team you will map to this role will depend on how
you have designed your organization's RBAC, but it should be the smallest team
possible within your organization. This team must also include your user.
Here is an example:
teams_to_roles:
- organization: octocats
team: admins
roles:
- access
+ - teleport-event-handler-impersonator
Apply your changes:
tctl create -f github.yaml
Note the --with-secrets
flag in the tctl get
command. This adds the value of
spec.signing_key_pair.private_key
to github.yaml
. This is a sensitive value,
so take precautions when creating this file and remove it after updating the resource.
Retrieve your saml
configuration resource:
tctl get --with-secrets saml/mysaml > saml.yaml
Edit saml.yaml
, adding teleport-event-handler-impersonator
to the
attributes_to_roles
section. The attribute you will map to this role will
depend on how you have designed your organization's RBAC, but it should be the
smallest group possible within your organization. This group must also include
your user.
Here is an example:
attributes_to_roles:
- name: "groups"
value: "my-group"
roles:
- access
+ - teleport-event-handler-impersonator
Apply your changes:
tctl create -f saml.yaml
Note the --with-secrets
flag in the tctl get
command. This adds the value of
spec.signing_key_pair.private_key
to saml.yaml
. This is a sensitive value,
so take precautions when creating this file and remove it after updating the resource.
Retrieve your oidc
configuration resource:
tctl get oidc/myoidc --with-secrets > oidc.yaml
Edit oidc.yaml
, adding teleport-event-handler-impersonator
to the
claims_to_roles
section. The claim you will map to this role will depend on
how you have designed your organization's RBAC, but it should be the smallest
group possible within your organization. This group must also include your
user.
Here is an example:
claims_to_roles:
- name: "groups"
value: "my-group"
roles:
- access
+ - teleport-event-handler-impersonator
Apply your changes:
tctl create -f oidc.yaml
Note the --with-secrets
flag in the tctl get
command. This adds the value of
spec.signing_key_pair.private_key
to oidc.yaml
. This is a sensitive value,
so take precautions when creating this file and remove it after updating the resource.
Log out of your Teleport cluster and log in again to assume the new role.
Export an identity file for the Fluentd plugin user
The Fluentd Teleport plugin uses the teleport-event-handler
role and user to
read events. We export an identity file for the user with the tctl auth sign
command.
tctl auth sign --user=teleport-event-handler --out=identity
This command creates one PEM-encoded file, identity
. The identity file
includes both TLS and SSH credentials. The Event Handler plugin uses the SSH
credentials to connect to the Proxy Service, which establishes a reverse tunnel
connection to the Auth Service. The plugin uses this reverse tunnel, along with
your TLS credentials, to connect to the Auth Service's gRPC endpoint.
Please use the latest version of Teleport Enterprise documentation.
Please use the latest version of Teleport Enterprise documentation.
Step 5/6. Start the Fluentd forwarder
The Fluentd plugin will send events to your Fluentd instance using keys generated on the previous step.
The fluent.conf
file generated earlier configures your Fluentd instance to
accept events using TLS and print them:
<source>
@type http
port 8888
<transport tls>
client_cert_auth true
# We are going to run fluentd in Docker. /keys will be mounted from the host file system.
ca_path /keys/ca.crt
cert_path /keys/server.crt
private_key_path /keys/server.key
private_key_passphrase ********** # Passphrase generated along with the keys
</transport>
<parse>
@type json
json_parser oj
# This time format is used by the plugin. This field is required.
time_type string
time_format %Y-%m-%dT%H:%M:%S
</parse>
# If the number of events is high, fluentd will start failing the ingestion
# with the following error message: buffer space has too many data errors.
# The following configuration prevents data loss in case of a restart and
# overcomes the limitations of the default fluentd buffer configuration.
# This configuration is optional.
# See https://docs.fluentd.org/configuration/buffer-section for more details.
<buffer>
@type file
flush_thread_count 8
flush_interval 1s
chunk_limit_size 10M
queue_limit_length 16
retry_max_interval 30
retry_forever true
</buffer>
</source>
# Events sent to test.log will be dumped to STDOUT.
<match test.log>
@type stdout
</match>
To try out this Fluentd configuration, start your fluentd instance:
docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge
This will consume your current terminal, so open a new one to continue following along.
Step 6/6. Start the event handler plugin
Earlier, we generated a file called teleport-event-handler.toml
to configure
the Fluentd event handler. This file includes setting similar to the following:
Please use the latest version of Teleport Enterprise documentation.
storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"
[forward.fluentd]
ca = "/home/bob/event-handler/ca.crt"
cert = "/home/bob/event-handler/client.crt"
key = "/home/bob/event-handler/client.key"
url = "https://localhost:8888/test.log"
[teleport]
addr = "example.teleport.com:443"
identity = "identity"
To start the Event Handler, run the following command:
./teleport-event-handler start --config teleport-event-handler.toml
Use the following template to create teleport-plugin-event-handler-values.yaml
:
eventHandler:
storagePath: "./storage"
timeout: "10s"
batch: 20
namespace: "default"
teleport:
address: "example.teleport.com:443"
identitySecretName: teleport-event-handler-identity
fluentd:
url: "https://fluentd.fluentd.svc.cluster.local/events.log"
sessionUrl: "https://fluentd.fluentd.svc.cluster.local/session.log"
certificate:
secretName: "teleport-event-handler-client-tls"
caPath: "ca.crt"
certPath: "client.crt"
keyPath: "client.key"
persistentVolumeClaim:
enabled: true
To start the event handler in Kubernetes, run the following command:
helm install teleport-plugin-event-handler teleport/teleport-plugin-event-handler \ --values teleport-plugin-event-handler-values.yaml \ --version 13.0.3
This example will start exporting from May 5th 2021
:
./teleport-event-handler start --config teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"
The start time can be set only once, on the first run of the tool.
If you want to change the time frame later, remove the plugin state directory
that you specified in the storage
field of the handler's configuration file.
Once the handler starts, you will see notifications about scanned and forwarded events:
INFO[0046] Event sent id=0b5f2a3e-faa5-4d77-ab6e-362bca0994fc ts="2021-06-08 11:00:56.034 +0000 UTC" type=user.login
...
Troubleshooting connection issues
If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:
- The certificate the Teleport Event Handler is using to connect to your
Teleport cluster is not past its expiration date. This is the value of the
--ttl
flag in thetctl auth sign
command, which is 12 hours by default. - Ensure that in your Teleport Event Handler configuration file
(
teleport-event-handler.toml
), you have provided the correct host and port for the Teleport Proxy Service or Auth Service.
Next Steps
Read more about impersonation here.
While this guide uses the tctl auth sign
command to issue credentials for the
Teleport Event Handler, production clusters should use Machine ID for safer,
more reliable renewals. Read our guide
to getting started with Machine ID.
To see all of the options you can set in the values file for the
teleport-plugin-event-handler
Helm chart, consult our reference
guide.