
Teleport leverages Go's diagnostic capabilities to collect and export profiling data. Profiles can help identify the cause of spikes in CPU, the source of memory leaks, or the reason for a deadlock.
Enable profiling
The profiling endpoint is only enabled if the --debug
flag is supplied.
Teleport's diagnostic HTTP endpoints are disabled by default. You can enable them via:
Start a teleport
instance with the --diag-addr
flag set to the local address where the diagnostic endpoint will listen:
sudo teleport start --diag-addr=127.0.0.1:3000
Edit a teleport
instance's configuration file (/etc/teleport.yaml
by default) to include the following:
teleport:
diag_addr: 127.0.0.1:3000
Verify that Teleport is now serving the diagnostics endpoint:
curl http://127.0.0.1:3000
Collecting profiles
Go's standard profiling endpoints are served at http://127.0.0.1:3000/debug/pprof/
.
Retrieving a profile requires sending a request to the endpoint corresponding
to the desired profile type. When debugging an issue it is helpful to collect
a series of profiles over a period of time.
CPU
CPU profile shows execution statistics gathered over a user specified period:
Download the profile into a file:
curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile?seconds=30Visualize the profile
go tool pprof -http : cpu.profile
Goroutine
Goroutine profiles show the stack traces for all running goroutines in the system:
Download the profile into a file:
curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutineVisualize the profile
go tool pprof -http : goroutine.profile
Heap
Heap profiles show allocated objects in the system:
Download the profile into a file:
curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heapVisualize the profile
go tool pprof -http : heap.profile
Trace
Trace profiles capture scheduling, system calls, garbage collections, heap size, and other events that are collected by the Go runtime over a user specified period of time:
Download the profile into a file:
curl -o trace.out http://127.0.0.1:3000/debug/pprof/trace?seconds=5Visualize the profile
go tool trace trace.out
Further Reading
- More information about diagnostics in the Go ecosystem: https://go.dev/doc/diagnostics
- Go's profiling endpoints: https://golang.org/pkg/net/http/pprof/
- A deep dive on profiling Go programs: https://go.dev/blog/pprof