Fork me on GitHub
Teleport

Metrics

Teleport Prometheus endpoint

Teleport provides HTTP endpoints for monitoring purposes. They are disabled by default, but you can enable them using the --diag-addr flag to teleport start:

sudo teleport start --diag-addr=127.0.0.1:3000

Now you can see the monitoring information by visiting several endpoints:

  • http://127.0.0.1:3000/metrics is the list of internal metrics Teleport is tracking. It is compatible with Prometheus collectors.
  • http://127.0.0.1:3000/healthz returns "OK" if the process is healthy or 503 otherwise.
  • http://127.0.0.1:3000/readyz is similar to /healthz, but it returns "OK" only after the node successfully joined the cluster, i.e.it draws the difference between "healthy" and "ready".
  • http://127.0.0.1:3000/debug/pprof/ is Golang's standard profiler. It's only available when -d flag is given in addition to --diag-addr
NameTypeComponentDescription
backend_batch_read_requests_totalcountercacheNumber of read requests to the backend.
backend_batch_read_secondshistogramcacheLatency for batch read operations.
backend_batch_write_requests_totalcountercacheNumber of batch write requests to the backend.
backend_batch_write_secondshistogramcacheLatency for backend batch write operations.
backend_read_requests_totalcountercacheNumber of read requests to the backend.
backend_read_secondshistogramcacheLatency for read operations.
backend_write_requests_totalcountercacheNumber of write requests to the backend.
backend_write_secondshistogramcacheLatency for backend write operations.
etcd_backend_batch_read_requestscounteretcdNumber of read requests to the etcd database.
etcd_backend_batch_read_secondshistogrametcdLatency for etcd read operations.
etcd_backend_read_requestscounteretcdNumber of read requests to the etcd database.
etcd_backend_read_secondshistogrametcdLatency for etcd read operations.
etcd_backend_tx_requestscounteretcdNumber of transaction requests to the database.
etcd_backend_tx_secondshistogrametcdLatency for etcd transaction operations.
etcd_backend_write_requestscounteretcdNumber of write requests to the database.
etcd_backend_write_secondshistogrametcdLatency for etcd write operations.
firestore_events_backend_batch_read_requestscounterGCP Cloud FirestoreNumber of batch read requests to Cloud Firestore events.
firestore_events_backend_batch_read_secondshistogramGCP Cloud FirestoreLatency for Cloud Firestore events batch read operations.
firestore_events_backend_batch_write_requestscounterGCP Cloud FirestoreNumber of batch write requests to Cloud Firestore events.
firestore_events_backend_batch_write_secondshistogramGCP Cloud FirestoreLatency for Cloud Firestore events batch write operations.
gcs_event_storage_downloadscounterGCP GCSNumber of downloads from the GCS backend.
gcs_event_storage_downloads_secondshistogramInternal GoLangLatency for GCS download operations.
gcs_event_storage_uploadscounterInternal GoLangNumber of uploads to the GCS backend.
gcs_event_storage_uploads_secondshistogramInternal GoLangLatency for GCS upload operations.
go_gc_duration_secondssummaryInternal GoLangA summary of the GC invocation durations.
go_goroutinesgaugeInternal GoLangNumber of goroutines that currently exist.
go_infogaugeInternal GoLangInformation about the Go environment.
go_memstats_alloc_bytesgaugeInternal GoLangNumber of bytes allocated and still in use.
go_memstats_alloc_bytes_totalcounterInternal GoLangTotal number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytesgaugeInternal GoLangNumber of bytes used by the profiling bucket hash table.
go_memstats_frees_totalcounterInternal GoLangTotal number of frees.
go_memstats_gc_cpu_fractiongaugeInternal GoLangThe fraction of this program's available CPU time used by the GC since the program started.
go_memstats_gc_sys_bytesgaugeInternal GoLangNumber of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytesgaugeInternal GoLangNumber of heap bytes allocated and still in use.
go_memstats_heap_idle_bytesgaugeInternal GoLangNumber of heap bytes waiting to be used.
go_memstats_heap_inuse_bytesgaugeInternal GoLangNumber of heap bytes that are in use.
go_memstats_heap_objectsgaugeInternal GoLangNumber of allocated objects.
go_memstats_heap_released_bytesgaugeInternal GoLangNumber of heap bytes released to OS.
go_memstats_heap_sys_bytesgaugeInternal GoLangNumber of heap bytes obtained from system.
go_memstats_last_gc_time_secondsgaugeInternal GoLangNumber of seconds since 1970 of last garbage collection.
go_memstats_lookups_totalcounterInternal GoLangTotal number of pointer lookups.
go_memstats_mallocs_totalcounterInternal GoLangTotal number of mallocs.
go_memstats_mcache_inuse_bytesgaugeInternal GoLangNumber of bytes in use by mcache structures.
go_memstats_mcache_sys_bytesgaugeInternal GoLangNumber of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytesgaugeInternal GoLangNumber of bytes in use by mspan structures.
go_memstats_mspan_sys_bytesgaugeInternal GoLangNumber of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytesgaugeInternal GoLangNumber of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytesgaugeInternal GoLangNumber of bytes used for other system allocations.
go_memstats_stack_inuse_bytesgaugeInternal GoLangNumber of bytes in use by the stack allocator.
go_memstats_stack_sys_bytesgaugeInternal GoLangNumber of bytes obtained from system for stack allocator.
go_memstats_sys_bytesgaugeInternal GoLangNumber of bytes obtained from system.
go_threadsgaugeInternal GoLangNumber of OS threads created.
process_cpu_seconds_totalcounterInternal GoLangTotal user and system CPU time spent in seconds.
process_max_fdsgaugeInternal GoLangMaximum number of open file descriptors.
process_open_fdsgaugeInternal GoLangNumber of open file descriptors.
process_resident_memory_bytesgaugeInternal GoLangResident memory size in bytes.
process_start_time_secondsgaugeInternal GoLangStart time of the process since unix epoch in seconds.
process_virtual_memory_bytesgaugeInternal GoLangVirtual memory size in bytes.
process_virtual_memory_max_bytesgaugeInternal GoLangMaximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flightgaugeprometheusCurrent number of scrapes being served.
promhttp_metric_handler_requests_totalcounterprometheusTotal number of scrapes by HTTP status code.
reversetunnel_connected_proxiesgaugeTeleportNumber of known proxies being sought.
rxcounterTeleportNumber of bytes received.
server_interactive_sessions_totalgaugeTeleportNumber of active sessions.
trusted_clustersgaugeTeleportNumber of tunnels per state.
txcounterTeleportNumber of bytes transmitted.
audit_failed_disk_monitoringcounterTeleport Audit LogNumber of times disk monitoring failed.
audit_failed_emit_eventscounterTeleport Audit LogNumber of times emitting audit event failed.
audit_percentage_disk_space_usedgaugeTeleport Audit LogPercentage disk space used.
audit_server_open_filesgaugeTeleport Audit LogNumber of open audit files.
auth_generate_requestsgaugeTeleport AuthNumber of current generate requests.
auth_generate_requests_throttled_totalcounterTeleport AuthNumber of throttled requests to generate new server keys.
auth_generate_requests_totalcounterTeleport AuthNumber of requests to generate new server keys.
auth_generate_secondshistogramTeleport AuthLatency for generate requests.
cluster_name_not_found_totalcounterTeleport AuthNumber of times a cluster was not found.
heartbeat_connections_received_totalcounterTeleport AuthNumber of times auth received a heartbeat connection.
heartbeat_connections_missed_totalcounterTeleport AuthNumber of times auth did not receive a heartbeat from a node.
user_login_totalcounterTeleport AuthNumber of user logins.
failed_connect_to_node_attempts_totalcounterTeleport ProxyNumber of times a user failed connecting to a node
proxy_connection_limit_exceeded_totalcounterTeleport ProxyNumber of connections that exceeded the proxy connection limit.
certificate_mismatch_totalcounterTeleport ProxyNumber of times there was a certificate mismatch.
failed_login_attempts_totalcounterTeleport ProxyNumber of failed tsh login or tsh ssh logins.
user_max_concurrent_sessions_hit_totalcounterTeleport NodeNumber of times a user exceeded their concurrent session limit.
Have a suggestion or can’t find something?
IMPROVE THE DOCS