Skip to main content
Version: 17.x (unreleased)

Export Teleport Audit Events with Datadog

Datadog is a SAAS monitoring and security platform. In this guide, we'll explain how to forward Teleport audit events to Datadog using Fluentd.

How it works​

The Teleport Event Handler authenticates to the Teleport Auth Service to receive audit events over a gRPC stream, then sends those events to Fluentd as JSON payloads over a secure channel established via mutual TLS:

Architecture of the setup shown in this guide

Since the Datadog Agent can only receive logs from remote sources as JSON-encoded bytes over a TCP or UDP connection, the Teleport Event Handler needs to send its HTTPS payloads without using the Datadog Agent. Fluentd handles authentication to the Datadog API.

Prerequisites​

  • A running Teleport cluster version 17.0.0-dev or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl admin tool and tsh client tool.

    Visit Installation for instructions on downloading tctl and tsh.

Recommended: Configure Machine ID to provide short-lived Teleport credentials to the plugin. Before following this guide, follow a Machine ID deployment guide to run the tbot binary on your infrastructure.

  • A Datadog account.
  • A server, virtual machine, Kubernetes cluster, or Docker environment to run the Event Handler. The instructions below assume a local Docker container for testing.
  • Fluentd version v1.12.4 or greater. The Teleport Event Handler will create a new fluent.conf file you can integrate into an existing Fluentd system, or use with a fresh setup.

The instructions below demonstrate a local test of the Event Handler plugin on your workstation. You will need to adjust paths, ports, and domains for other environments.

  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example: 
    $ tsh login --proxy=teleport.example.com [email protected]
    $ tctl status
    # Cluster  teleport.example.com
    # Version  17.0.0-dev
    # CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/6. Install the Event Handler plugin​

The Teleport Event Handler runs alongside the Fluentd forwarder, receives events from Teleport's events API, and forwards them to Fluentd.

The Event Handler plugin is provided in amd64 and arm64 binaries for downloading. Replace ARCH with your required version.

$ curl -L -O https://cdn.teleport.dev/teleport-event-handler-v17.0.0-dev-linux-ARCH-bin.tar.gz
$ tar -zxvf teleport-event-handler-v17.0.0-dev-linux-ARCH-bin.tar.gz
$ sudo ./teleport-event-handler/install

Step 2/6. Generate a plugin configuration​

Run the configure command to generate a sample configuration. Replace mytenant.teleport.sh with the DNS name of your Teleport Enterprise Cloud tenant:

$ teleport-event-handler configure . mytenant.teleport.sh:443

You'll see the following output:

Teleport event handler 17.0.0-dev

[1] mTLS Fluentd certificates generated and saved to ca.crt, ca.key, server.crt, server.key, client.crt, client.key
[2] Generated sample teleport-event-handler role and user file teleport-event-handler-role.yaml
[3] Generated sample fluentd configuration file fluent.conf
[4] Generated plugin configuration file teleport-event-handler.toml

The plugin generates several setup files:

$ ls -l
# -rw------- 1 bob bob     1038 Jul  1 11:14 ca.crt
# -rw------- 1 bob bob     1679 Jul  1 11:14 ca.key
# -rw------- 1 bob bob     1042 Jul  1 11:14 client.crt
# -rw------- 1 bob bob     1679 Jul  1 11:14 client.key
# -rw------- 1 bob bob      541 Jul  1 11:14 fluent.conf
# -rw------- 1 bob bob     1078 Jul  1 11:14 server.crt
# -rw------- 1 bob bob     1766 Jul  1 11:14 server.key
# -rw------- 1 bob bob      260 Jul  1 11:14 teleport-event-handler-role.yaml
# -rw------- 1 bob bob      343 Jul  1 11:14 teleport-event-handler.toml
File(s)Purpose
ca.crt and ca.keySelf-signed CA certificate and private key for Fluentd
server.crt and server.keyFluentd server certificate and key
client.crt and client.keyFluentd client certificate and key, all signed by the generated CA
teleport-event-handler-role.yamluser and role resource definitions for Teleport's event handler
fluent.confFluentd plugin configuration
Running the Event Handler separately from the log forwarder

This guide assumes that you are running the Event Handler on the same host or Kubernetes pod as your log forwarder. If you are not, you will need to instruct the Event Handler to generate mTLS certificates for subjects besides localhost. To do this, use the --cn and --dns-names flags of the teleport-event-handler configure command.

For example, if your log forwarder is addressable at forwarder.example.com and the Event Handler at handler.example.com, you would run the following configure command:

$ teleport-event-handler configure --cn=handler.example.com --dns-names=forwarder.example.com

The command generates client and server certificates with the subjects set to the value of --cn.

The --dns-names flag accepts a comma-separated list of DNS names. It will append subject alternative names (SANs) to the server certificate (the one you will provide to your log forwarder) for each DNS name in the list. The Event Handler looks up each DNS name before appending it as an SAN and exits with an error if the lookup fails.

Step 3/6. Create a user and role for reading audit events​

The teleport-event-handler configure command generated a file called teleport-event-handler-role.yaml. This file defines a teleport-event-handler role and a user with read-only access to the event API:

kind: role
metadata:
  name: teleport-event-handler
spec:
  allow:
    rules:
      - resources: ['event', 'session']
        verbs: ['list','read']
version: v5
---
kind: user
metadata:
  name: teleport-event-handler
spec:
  roles: ['teleport-event-handler']
version: v2

Move this file to your workstation (or recreate it by pasting the snippet above) and use tctl on your workstation to create the role and the user:

$ tctl create -f teleport-event-handler-role.yaml
# user "teleport-event-handler" has been created
# role 'teleport-event-handler' has been created

Step 4/6. Create teleport-event-handler credentials​

The Teleport Event Handler needs credentials to authenticate to the Teleport Auth Service. In this section, you will give the Event Handler access to these credentials.

Enable issuing of credentials for the Event Handler role​

With the role created, you now need to allow the Machine ID bot to produce credentials for this role.

This can be done with tctl, replacing my-bot with the name of your bot:

$ tctl bots update my-bot --add-roles teleport-event-handler

Export an identity file for the Event Handler plugin user​

Give the plugin access to a Teleport identity file. We recommend using Machine ID for this in order to produce short-lived identity files that are less dangerous if exfiltrated, though in demo deployments, you can generate longer-lived identity files with tctl:

Configure tbot with an output that will produce the credentials needed by the plugin. As the plugin will be accessing the Teleport API, the correct output type to use is identity.

For this guide, the directory destination will be used. This will write these credentials to a specified directory on disk. Ensure that this directory can be written to by the Linux user that tbot runs as, and that it can be read by the Linux user that the plugin will run as.

Modify your tbot configuration to add an identity output.

If running tbot on a Linux server, use the directory output to write identity files to the /opt/machine-id directory:

outputs:
- type: identity
  destination:
    type: directory
    # For this guide, /opt/machine-id is used as the destination directory.
    # You may wish to customize this. Multiple outputs cannot share the same
    # destination.
    path: /opt/machine-id

If running tbot on Kubernetes, write the identity file to Kubernetes secret instead:

outputs:
  - type: identity
    destination:
      type: kubernetes_secret
      name: teleport-event-handler-identity

If operating tbot as a background service, restart it. If running tbot in one-shot mode, execute it now.

You should now see an identity file under /opt/machine-id or a Kubernetes secret named teleport-event-handler-identity. This contains the private key and signed certificates needed by the plugin to authenticate with the Teleport Auth Service.

Step 5/6. Install the Fluentd output plugin for Datadog​

In order for Fluentd to communicate with Datadog, it requires the Fluentd output plugin for Datadog. Install the plugin on your Fluentd host using either gem or the td-agent, if installed:

# Using Gem
$ gem install fluent-plugin-datadog

# Using td-agent
$ /usr/sbin/td-agent-gem install fluent-plugin-datadog
Testing Locally?

If you're running Fluentd in a local Docker container for testing, you can adjust the entrypoint to an interactive shell as the root user, so you can install the plugin before starting Fluentd:

$ docker run -u $(id -u root):$(id -g root) -p 8888:8888 -v $(pwd):/keys -v \
$(pwd)/fluent.conf:/fluentd/etc/fluent.conf --entrypoint=/bin/sh -i --tty  fluent/fluentd:edge
# From the container shell:
$ gem install fluent-plugin-datadog
$ fluentd -c /fluentd/etc/fluent.conf

Configure Fluentd​

  1. Visit Datadog and generate an API key for Fluentd by following the Datadog documentation.

  2. Copy the API key and use it to add a new <match> block to fluent.conf:

    <match test.log>

      @type datadog
      @id awesome_agent
      api_key abcd123-insecure-do-not-use-this

      host http-intake.logs.us5.datadoghq.com

      # Optional parameters
      dd_source teleport

    </match>

  3. Edit your configuration as follows:

    • Add your API key to the api_key field.
    • Adjust the host value to match your Datadog site. See the Datadog Log Collection and Integrations guide to determine the correct value.
    • dd_source is an optional field you can use to filter these logs in the Datadog UI.
    • Adjust ca_path, cert_path and private_key_path to point to the credential files generated earlier. If you're testing locally, the Docker command above already mounted the current working directory to keys/ in the container.

  4. Restart Fluentd after saving the changes to fluent.conf.

Step 6/6. Run the Teleport Event Handler plugin​

In this section, you will modify the Event Handler configuration you generated and run the Event Handler to test your configuration.

Configure the Event Handler​

In this section, you will configure the Teleport Event Handler for your environment.

Earlier, we generated a file called teleport-event-handler.toml to configure the Fluentd event handler. This file includes setting similar to the following:

storage = "./storage"
timeout = "10s"
batch = 20
namespace = "default"
# The window size configures the duration of the time window for the event handler
# to request events from Teleport. By default, this is set to 24 hours.
# Reduce the window size if the events backend cannot manage the event volume 
# for the default window size.
# The window size should be specified as a duration string, parsed by Go's time.ParseDuration.
window-size = "24h"

[forward.fluentd]
ca = "/home/bob/event-handler/ca.crt"
cert = "/home/bob/event-handler/client.crt"
key = "/home/bob/event-handler/client.key"
url = "https://fluentd.example.com:8888/test.log"
session-url = "https://fluentd.example.com:8888/session"

[teleport]
addr = "example.teleport.com:443"
identity = "identity"

Modify the configuration to replace fluentd.example.com with the domain name of your Fluentd deployment.

Next, modify the configuration file as follows:

addr: Include the hostname and HTTPS port of your Teleport Proxy Service or Teleport Enterprise Cloud account (e.g., teleport.example.com:443 or mytenant.teleport.sh:443).

identity: Fill this in with the path to the identity file you exported earlier.

client_key, client_crt, root_cas: Comment these out, since we are not using them in this configuration.

If you are providing credentials to the Event Handler using a tbot binary that runs on a Linux server, make sure the value of identity in the Event Handler configuration is the same as the path of the identity file you configured tbot to generate, /opt/machine-id/identity.

Start the Teleport Event Handler​

Start the Teleport Teleport Event Handler by following the instructions below.

Copy the teleport-event-handler.toml file to /etc on your Linux server. Update the settings within the toml file to match your environment. Make sure to use absolute paths on settings such as identity and storage. Files and directories in use should only be accessible to the system user executing the teleport-event-handler service such as /var/lib/teleport-event-handler.

Next, create a systemd service definition at the path /usr/lib/systemd/system/teleport-event-handler.service with the following content:

[Unit]
Description=Teleport Event Handler
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/teleport-event-handler start --config=/etc/teleport-event-handler.toml --teleport-refresh-enabled=true
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-event-handler.pid

[Install]
WantedBy=multi-user.target

If you are not using Machine ID to provide short-lived credentials to the Event Handler, you can remove the --teleport-refresh-enabled true flag.

Enable and start the plugin:

$ sudo systemctl enable teleport-event-handler
$ sudo systemctl start teleport-event-handler
Choose when to start exporting events

You can configure when you would like the Teleport Event Handler to begin exporting events when you run the start command. This example will start exporting from May 5th, 2021:

$ teleport-event-handler start --config /etc/teleport-event-handler.toml --start-time "2021-05-05T00:00:00Z"

You can only determine the start time once, when first running the Teleport Event Handler. If you want to change the time frame later, remove the plugin state directory that you specified in the storage field of the handler's configuration file.

Once the Teleport Event Handler starts, you will see notifications about scanned and forwarded events:

$ sudo journalctl -u teleport-event-handler
DEBU   Event sent id:f19cf375-4da6-4338-bfdc-e38334c60fd1 index:0 ts:2022-09-21
18:51:04.849 +0000 UTC type:cert.create event-handler/app.go:140
...

The Logs view in Datadog should now report your Teleport cluster events:

Datadog Logs

Troubleshooting connection issues​

If the Teleport Event Handler is displaying error logs while connecting to your Teleport Cluster, ensure that:

  • The certificate the Teleport Event Handler is using to connect to your Teleport cluster is not past its expiration date. This is the value of the --ttl flag in the tctl auth sign command, which is 12 hours by default.
  • Ensure that in your Teleport Event Handler configuration file (teleport-event-handler.toml), you have provided the correct host and port for the Teleport Proxy Service.

Next steps​

  • Read more about impersonation here.
  • While this guide uses the tctl auth sign command to issue credentials for the Teleport Event Handler, production clusters should use Machine ID for safer, more reliable renewals. Read our guide to getting started with Machine ID.
  • To see all of the options you can set in the values file for the teleport-plugin-event-handler Helm chart, consult our reference guide.
  • Review the Fluentd output plugin for Datadog README file to learn how to customize the log format entering Datadog.