Distributed Tracing

Teleport leverages OpenTelemetry to generate traces and export them to any OpenTelemetry Protocol (OTLP) capable exporter. In the event that your telemetry backend doesn't support receiving OTLP traces, you may be able to leverage the OpenTelemetry Collector to proxy traces from OTLP to a format that your telemetry backend accepts.

Configure Teleport

In order to enable tracing for a teleport instance, add the following section to that instance's configuration file (/etc/teleport.yaml). For a detailed description of these configuration fields, see the configuration reference page.

tracing_service:
   enabled: yes
   exporter_url: grpc://collector.example.com:4317
   sampling_rate_per_million: 1000000

Sampling rate

It is important to choose the sampling rate wisely. Sampling at a rate of 100% could have a negative impact on the performance of your cluster. Teleport honors the sampling rate included in any incoming requests, which means that even when the tracing_service is enabled and the sampling rate is 0, if Teleport receives a request that has a span which is sampled, then Teleport will sample and export all spans that are generated in response to that request.

Exporter URL

The exporter_url setting indicates where Teleport should send spans to. Supported schemes are grpc://, http://, https://, and file:// (if no scheme is provided, then grpc:// is used).

When using file://, the url must be a path to a directory that Teleport has write permissions for. Spans will be saved to files within the provided directory, each file containing one proto encoded span per line. Files are rotated after exceeding 100MB, in order to override the default limit add ?limit=<desired_file_size_in_bytes> to the exporter_url (i.e. file:///var/lib/teleport/traces?limit=100).

By default the connection to the exporter is insecure, to support TLS add the following to the tracing_service configuration:

   # Optional path to CA certificates are used to validate the exporter.
  ca_certs:
    - /var/lib/teleport/exporter_ca.pem
  # Optional path tp TLS certificates are used to enable mTLS for the exporter
  https_keypairs:
    - key_file: /var/lib/teleport/exporter_key.pem
      cert_file: /var/lib/teleport/exporter_cert.pem

After updating teleport.yaml, start your teleport instance to apply the new configuration.

tsh

To capture traces from tsh simply add the --trace flag to your command. All traces generated by tsh --trace will be proxied to the exporter_url defined for the Auth Service of the cluster the command is being run on.

tsh --trace ssh root@myserver
tsh --trace ls

Exporting traces from tsh to a different exporter than the one defined in the Auth Service config is also possible via the --trace-exporter flag. A URL must be provided that adheres to the same format as the exporter_url of the tracing_service.

tsh --trace --trace-exporter=grpc://collector.example.com:4317 ssh root@myserver
tsh --trace --trace-exporter=file:///var/lib/teleport/traces ls