Forwarding Access Logs using FluentD
Length: 18:36
In this guide, we will explain how to:
- Set up Teleport's event handler.
- Forward events with Fluentd.
Prerequisites
-
A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.
-
The
tctl
admin tool andtsh
client tool version >= 9.3.7.tctl versionTeleport v9.3.7 go1.17
tsh versionTeleport v9.3.7 go1.17
See Installation for details.
-
A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.
-
The
tctl
admin tool andtsh
client tool version >= 9.3.7, which you can download by visiting the customer portal.tctl versionTeleport v9.3.7 go1.17
tsh versionTeleport v9.3.7 go1.17
-
A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.
-
The
tctl
admin tool andtsh
client tool version >= 9.3.8. To download these tools, visit the Downloads page.tctl versionTeleport v9.3.8 go1.17
tsh versionTeleport v9.3.8 go1.17
- Fluentd version v1.12.4.
- Docker version v20.10.7.
To connect to Teleport, log in to your cluster using tsh
, then use tctl
remotely:
tsh login --proxy=teleport.example.com [email protected]tctl statusCluster teleport.example.com
Version 9.3.7
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
You can run subsequent tctl
commands in this guide on your local machine.
For full privileges, you can also run tctl
commands on your Auth Service host.
To connect to Teleport, log in to your cluster using tsh
, then use tctl
remotely:
tsh login --proxy=myinstance.teleport.sh [email protected]tctl statusCluster myinstance.teleport.sh
Version 9.3.8
CA pin sha256:sha-hash-here
You must run subsequent tctl
commands in this guide on your local machine.
Create a folder called fluentd
to hold configuration and plugin state:
mkdir -p event-handlercd event-handler
Step 1/6. Install the event handler plugin
The Teleport event handler runs alongside the Fluentd forwarder, receives events from Teleport's events API, and forwards them to Fluentd.
curl -L -O https://get.gravitational.com/teleport-event-handler-v9.3.7-linux-amd64-bin.tar.gztar -zxvf teleport-event-handler-v9.3.7-linux-amd64-bin.tar.gz
curl -L -O https://get.gravitational.com/teleport-event-handler-v9.3.7-darwin-amd64-bin.tar.gztar -zxvf teleport-event-handler-v9.3.7-linux-amd64-bin.tar.gz
docker pull quay.io/gravitational/teleport-plugin-event-handler:9.3.7
Step 2/6. Generate configuration
Run the configure
command to generate a sample configuration. Replace
teleport.example.com:443
with the DNS name and HTTPS port of Teleport's Proxy
Service:
./teleport-event-handler configure . teleport.example.com:443
Run the configure
command to generate a sample configuration. Replace
mytenant.teleport.sh
with the DNS name of your Teleport Cloud tenant:
./teleport-event-handler configure . mytenant.teleport.sh
You'll see the following output:
Teleport event handler 0.0.1 07617b0ad0829db043fe779faf1669defdc8d84e
[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml
Follow-along with our getting started guide:
https://goteleport.com/setup/guides/fluentd
The plugin generates several setup files:
ls -l-rw------- 1 bob bob 1038 Jul 1 11:14 ca.crt
-rw------- 1 bob bob 1679 Jul 1 11:14 ca.key
-rw------- 1 bob bob 1042 Jul 1 11:14 client.crt
-rw------- 1 bob bob 1679 Jul 1 11:14 client.key
-rw------- 1 bob bob 541 Jul 1 11:14 fluent.conf
-rw------- 1 bob bob 1078 Jul 1 11:14 server.crt
-rw------- 1 bob bob 1766 Jul 1 11:14 server.key
-rw------- 1 bob bob 260 Jul 1 11:14 teleport-event-handler-role.yaml
-rw------- 1 bob bob 343 Jul 1 11:14 teleport-event-handler.toml
File(s) | Purpose |
---|---|
ca.crt and ca.key | Self-signed CA certificate and private key for Fluentd |
server.crt and server.key | Fluentd server certificate and key |
client.crt and client.key | Fluentd client certificate and key, all signed by the generated CA |
teleport-event-handler-role.yaml | user and role resource definitions for Teleport's event handler |
fluent.conf | Fluentd plugin configuration |
Step 3/6. Create a user and role for reading audit events
The configure
command generates a file called
teleport-event-handler-role.yaml
that defines a teleport-event-handler
role
and a user with read-only access to the event
API:
kind: user
metadata:
name: teleport-event-handler
spec:
roles: ['teleport-event-handler']
version: v2
---
kind: role
metadata:
name: teleport-event-handler
spec:
allow:
rules:
- resources: ['event']
verbs: ['list','read']
version: v5
Use tctl
to create the role and the user:
tctl create -f teleport-event-handler-role.yamluser "teleport-event-handler" has been created
role 'teleport-event-handler' has been created
Step 4/6. Create teleport-event-handler credentials
Enable impersonation of the Fluentd plugin user
In order for the Fluentd plugin to forward events from your Teleport cluster, it needs a signed identity file from the cluster's certificate authority. The Fluentd user cannot request this itself, and requires another user to impersonate this account in order to request a certificate.
Create a role that enables your user to impersonate the Fluentd user. First,
paste the following YAML document into a file called
teleport-event-handler-impersonator.yaml
:
kind: role
version: v5
metadata:
name: teleport-event-handler-impersonator
spec:
# SSH options used for user sessions
options:
# max_session_ttl defines the TTL (time to live) of SSH certificates
# issued to the users with this role.
max_session_ttl: 10h
# allow section declares a list of resource/verb combinations that are
# allowed for the users of this role. by default nothing is allowed.
allow:
impersonate:
users: ["teleport-event-handler"]
roles: ["teleport-event-handler"]
Next, create the role:
tctl create teleport-event-handler-impersonator.yaml
Assign this role to the current user by running the following command, substituting the user as appropriate:
tctl get users/<USER> > out.yaml
Now edit out.yaml
, adding teleport-event-handler-impersonator
to the list of
existing roles and update via tctl
again:
$ tctl create -f out.yaml
Log in to your Teleport cluster again to assume the new role.
Export an identity file for the Fluentd plugin user
The Fluentd Teleport plugin uses the teleport-event-handler
role and user to
read events. We export an identity file for the user with the tctl auth sign
command.
tctl auth sign --format=tls --user=teleport-event-handler --out=auth
This command should result in three PEM-encoded files: auth.crt
, auth.key
,
and auth.cas
(certificate, private key, and CA certs, respectively).
tctl auth sign --user=teleport-event-handler --out=identity
The above sequence should result in one PEM-encoded file: terraform-identity
.
Step 5/6. Start the Fluentd forwarder
The Fluentd plugin will send events to your Fluentd instance using keys generated on the previous step.
The fluent.conf
file generated earlier configures your Fluentd instance to
accept events using TLS and print them:
<source>
@type http
port 8888
<transport tls>
client_cert_auth true
# We are going to run fluentd in Docker. /keys will be mounted from the host file system.
ca_path /keys/ca.crt
cert_path /keys/server.crt
private_key_path /keys/server.key
private_key_passphrase ********** # Passphrase generated along with the keys
</transport>
<parse>
@type json
json_parser oj
# This time format is used by the plugin. This field is required.
time_type string
time_format %Y-%m-%dT%H:%M:%S
</parse>
</source>
# Events sent to test.log will be dumped to STDOUT.
<match test.log>
@type stdout
</match>
To try out this Fluentd configuration, start your fluentd instance:
docker run -u $(id -u ${USER}):$(id -g ${USER}) -p 8888:8888 -v $(pwd):/keys -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf fluent/fluentd:edge
Step 6/6. Start the event handler plugin
Earlier, we generated a file called teleport-event-handler.toml
to configure
the Fluentd event handler. This file includes setting similar to the following:
storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"
[forward.fluentd]
ca = "/home/sasha/scripts/event-handler/ca.crt"
cert = "/home/sasha/scripts/event-handler/client.crt"
key = "/home/sasha/scripts/event-handler/client.key"
url = "https://localhost:8888/test.log"
[teleport]
addr = "example.teleport.com:443"
identity = "identity"
storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"
[forward.fluentd]
ca = "/home/sasha/scripts/event-handler/ca.crt"
cert = "/home/sasha/scripts/event-handler/client.crt"
key = "/home/sasha/scripts/event-handler/client.key"
url = "https://localhost:8888/test.log"
[teleport]
addr = "example.teleport.com:443"
client_key = "auth.key"
client_crt = "auth.crt"
root_cas = "auth.cas"
To start the event handler, run the following command:
./teleport-event-handler start --config teleport-event-handler.toml
This example will start exporting from May 5th 2021
:
./teleport-event-handler start --config teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"
The start time can be set only once, on the first run of the tool.
If you want to change the time frame later, remove the plugin state directory
that you specified in the storage
field of the handler's configuration file.
Once the handler starts, you will see notifications about scanned and forwarded events:
INFO[0046] Event sent id=0b5f2a3e-faa5-4d77-ab6e-362bca0994fc ts="2021-06-08 11:00:56.034 +0000 UTC" type=user.login
...
Next Steps
- Read more about impersonation here.