Teleport provides HTTP endpoints for monitoring purposes. They are disabled
by default, but you can enable them using the --diag-addr
flag to teleport start
:
$ teleport start --diag-addr=127.0.0.1:3000
Now you can see the monitoring information by visiting several endpoints:
http://127.0.0.1:3000/metrics
is the list of internal metrics Teleport is
tracking. It is compatible with Prometheus
collectors.
http://127.0.0.1:3000/healthz
returns "OK" if the process is healthy or
503
otherwise.
http://127.0.0.1:3000/readyz
is similar to /healthz
, but it returns "OK"
only after the node successfully joined the cluster, i.e.it draws the
difference between "healthy" and "ready".
http://127.0.0.1:3000/debug/pprof/
is Golang's standard profiler. It's only
available when -d
flag is given in addition to --diag-addr
Name | Type | Component | Description |
---|---|---|---|
backend_batch_read_requests_total | counter | cache | Number of read requests to the backend |
backend_batch_read_seconds | histogram | cache | Latency for batch read operations |
backend_batch_write_requests_total | counter | cache | Number of batch write requests to the backend |
backend_batch_read_seconds | histogram | cache | Latency for batch read operations |
backend_batch_write_requests_total | counter | cache | Number of batch write requests to the backend |
backend_batch_write_seconds | histogram | cache | Latency for backend batch write operations |
backend_read_requests_total | counter | cache | Number of read requests to the backend |
backend_read_seconds | histogram | cache | Latency for read operations |
backend_write_requests_total | counter | cache | Number of write requests to the backend |
backend_write_seconds | histogram | cache | Latency for backend write operations |
etcd_backend_batch_read_requests | counter | etcd | Number of read requests to the etcd database |
etcd_backend_batch_read_seconds | histogram | etcd | Latency for etcd read operations |
etcd_backend_read_requests | counter | etcd | Number of read requests to the etcd database |
etcd_backend_read_seconds | histogram | etcd | Latency for etcd read operations |
etcd_backend_tx_requests | counter | etcd | Number of transaction requests to the database |
etcd_backend_tx_seconds | histogram | etcd | Latency for etcd transaction operations |
etcd_backend_write_requests | counter | etcd | Number of write requests to the database |
etcd_backend_write_seconds | histogram | etcd | Latency for etcd write operations |
firestore_events_backend_batch_read_requests | counter | GCP Cloud Firestore | Number of batch read requests to Cloud Firestore events |
firestore_events_backend_batch_read_seconds | histogram | GCP Cloud Firestore | Latency for Cloud Firestore events batch read operations |
firestore_events_backend_batch_write_requests | counter | GCP Cloud Firestore | Number of batch write requests to Cloud Firestore events |
firestore_events_backend_batch_write_seconds | histogram | GCP Cloud Firestore | Latency for Cloud Firestore events batch write operations |
gcs_event_storage_downloads | counter | GCP GCS | Number of downloads from the GCS backend |
gcs_event_storage_downloads_seconds | histogram | Internal GoLang | Latency for GCS download operations |
gcs_event_storage_uploads | counter | Internal GoLang | Number of uploads to the GCS backend |
gcs_event_storage_uploads_seconds | histogram | Internal GoLang | Latency for GCS upload operations |
go_gc_duration_seconds | summary | Internal GoLang | A summary of the GC invocation durations. |
go_goroutines | gauge | Internal GoLang | Number of goroutines that currently exist. |
go_info | gauge | Internal GoLang | Information about the Go environment. |
go_memstats_alloc_bytes | gauge | Internal GoLang | Number of bytes allocated and still in use. |
go_memstats_alloc_bytes_total | counter | Internal GoLang | Total number of bytes allocated, even if freed. |
go_memstats_buck_hash_sys_bytes | gauge | Internal GoLang | Number of bytes used by the profiling bucket hash table. |
go_memstats_frees_total | counter | Internal GoLang | Total number of frees. |
go_memstats_gc_cpu_fraction | gauge | Internal GoLang | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes | gauge | Internal GoLang | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes | gauge | Internal GoLang | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes | gauge | Internal GoLang | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes | gauge | Internal GoLang | Number of heap bytes that are in use. |
go_memstats_heap_objects | gauge | Internal GoLang | Number of allocated objects. |
go_memstats_heap_released_bytes | gauge | Internal GoLang | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes | gauge | Internal GoLang | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds | gauge | Internal GoLang | Number of seconds since 1970 of last garbage collection. |
go_memstats_lookups_total | counter | Internal GoLang | Total number of pointer lookups. |
go_memstats_mallocs_total | counter | Internal GoLang | Total number of mallocs. |
go_memstats_mcache_inuse_bytes | gauge | Internal GoLang | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes | gauge | Internal GoLang | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes | gauge | Internal GoLang | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes | gauge | Internal GoLang | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes | gauge | Internal GoLang | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes | gauge | Internal GoLang | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes | gauge | Internal GoLang | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes | gauge | Internal GoLang | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes | gauge | Internal GoLang | Number of bytes obtained from system. |
go_threads | gauge | Internal GoLang | Number of OS threads created. |
process_cpu_seconds_total | counter | Internal GoLang | Total user and system CPU time spent in seconds. |
process_max_fds | gauge | Internal GoLang | Maximum number of open file descriptors. |
process_open_fds | gauge | Internal GoLang | Number of open file descriptors. |
process_resident_memory_bytes | gauge | Internal GoLang | Resident memory size in bytes. |
process_start_time_seconds | gauge | Internal GoLang | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes | gauge | Internal GoLang | Virtual memory size in bytes. |
process_virtual_memory_max_bytes | gauge | Internal GoLang | Maximum amount of virtual memory available in bytes. |
promhttp_metric_handler_requests_in_flight | gauge | prometheus | Current number of scrapes being served. |
promhttp_metric_handler_requests_total | counter | prometheus | Total number of scrapes by HTTP status code. |
reversetunnel_connected_proxies | gauge | Teleport | Number of known proxies being sought. |
rx | counter | Teleport | Number of bytes received. |
server_interactive_sessions_total | gauge | Teleport | Number of active sessions |
trusted_clusters | gauge | Teleport | Number of tunnels per state |
tx | counter | Teleport | Number of bytes transmitted. |
audit_failed_disk_monitoring | counter | Teleport Audit Log | Number of times disk monitoring failed. |
audit_failed_emit_events | counter | Teleport Audit Log | Number of times emitting audit event failed. |
audit_percentage_disk_space_used | gauge | Teleport Audit Log | Percentage disk space used. |
audit_failed_emit_events | counter | Teleport Audit Log | Number of times emitting audit event failed. |
audit_percentage_disk_space_used | gauge | Teleport Audit Log | Percentage disk space used. |
audit_server_open_files | gauge | Teleport Audit Log | Number of open audit files |
auth_generate_requests | gauge | Teleport Auth | Number of current generate requests |
auth_generate_requests_throttled_total | counter | Teleport Auth | Number of throttled requests to generate new server keys |
auth_generate_requests_total | counter | Teleport Auth | Number of requests to generate new server keys |
auth_generate_seconds | histogram | Teleport Auth | Latency for generate requests |