Teleport

Profiling

Available for:
OpenSource
Enterprise
Cloud

Teleport leverages Go's diagnostic capabilities to collect and export profiling data. Profiles can help identify the cause of spikes in CPU, the source of memory leaks, or the reason for a deadlock.

Enable profiling

The profiling endpoint is only enabled if the --debug flag is supplied.

Teleport's diagnostic HTTP endpoints are disabled by default. You can enable them via:

Start a teleport instance with the --diag-addr flag set to the local address where the diagnostic endpoint will listen:

sudo teleport start --debug --diag-addr=127.0.0.1:3000

Edit a teleport instance's configuration file (/etc/teleport.yaml by default) to include the following:

teleport:
    diag_addr: 127.0.0.1:3000

To enable debug logs:

log:
    severity: DEBUG

Verify that Teleport is now serving the diagnostics endpoint:

curl http://127.0.0.1:3000/healthz

Collecting profiles

Go's standard profiling endpoints are served at http://127.0.0.1:3000/debug/pprof/. Retrieving a profile requires sending a request to the endpoint corresponding to the desired profile type. When debugging an issue it is helpful to collect a series of profiles over a period of time.

CPU

CPU profile shows execution statistics gathered over a user specified period:

Download the profile into a file:
curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile?seconds=30

Visualize the profile
go tool pprof -http : cpu.profile

Goroutine

Goroutine profiles show the stack traces for all running goroutines in the system:

Download the profile into a file:
curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutine

Visualize the profile
go tool pprof -http : goroutine.profile

Heap

Heap profiles show allocated objects in the system:

Download the profile into a file:
curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heap

Visualize the profile
go tool pprof  -http : heap.profile

Trace

Trace profiles capture scheduling, system calls, garbage collections, heap size, and other events that are collected by the Go runtime over a user specified period of time:

Download the profile into a file:
curl -o trace.out http://127.0.0.1:3000/debug/pprof/trace?seconds=5

Visualize the profile
go tool trace trace.out

Sign in to Teleport