Fork me on GitHub

Teleport

Distributed Tracing

Improve

Teleport leverages OpenTelemetry to generate traces and export them to any OpenTelemetry Protocol (OTLP) capable exporter. In the event that your telemetry backend doesn't support receiving OTLP traces, you may be able to leverage the OpenTelemetry Collector to proxy traces from OTLP to a format that your telemetry backend accepts.

Configure Teleport

In order to enable tracing for a teleport instance, add the following section to that instance's configuration file (/etc/teleport.yaml). For a detailed description of these configuration fields, see the configuration reference page.

tracing_service:
   enabled: yes
   exporter_url: grpc://collector.example.com:4317
   sampling_rate_per_million: 1000000

Please note that tracing was added in the following versions: 8.3.18, 9.3.15, and 10.1.0. Enabling the above configuration on a version prior to the versions with tracing support will result in a failure to parse the config file.

Sampling rate

It is important to choose the sampling rate wisely. Sampling at a rate of 100% could have a negative impact on the performance of your cluster. Teleport honors the sampling rate included in any incoming requests, which means that even when the tracing_service is enabled and the sampling rate is 0, if Teleport receives a request that has a span which is sampled, then Teleport will sample and export all spans that are generated in response to that request.

Exporter URL

The exporter_url setting indicates where Teleport should send spans to. Supported schemes are grpc://, http://, https://, and file:// (if no scheme is provided, then grpc:// is used).

When using file://, the url must be a path to a directory that Teleport has write permissions for. Spans will be saved to files within the provided directory, each file containing one proto encoded span per line. Files are rotated after exceeding 100MB, in order to override the default limit add ?limit=<desired_file_size_in_bytes> to the exporter_url (i.e. file:///var/lib/teleport/traces?limit=100).

By default the connection to the exporter is insecure, to support TLS add the following to the tracing_service configuration:

   # Optional path to CA certificates are used to validate the exporter.
  ca_certs:
    - /var/lib/teleport/exporter_ca.pem
  # Optional path tp TLS certificates are used to enable mTLS for the exporter
  https_keypairs:
    - key_file: /var/lib/teleport/exporter_key.pem
      cert_file: /var/lib/teleport/exporter_cert.pem

After updating teleport.yaml, start your teleport instance to apply the new configuration.

tsh

To capture traces from tsh simply add the --trace flag to your command. All traces generated by tsh --trace will be proxied to the exporter_url defined for the Auth Service of the cluster the command is being run on.

tsh --trace ssh [email protected]
tsh --trace ls