Fork me on GitHub

Teleport

Troubleshooting

Improve

In this guide, we will explain how to address issues or unexpected behavior in your Teleport cluster.

You can use these steps to get more visibility into the teleport process so you can troubleshoot the Auth Service, Proxy Service, and resource-specific services such as the Application Service and Database Service.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 10.3.1.

    tctl version

    Teleport v10.3.1 go1.18

    tsh version

    Teleport v10.3.1 go1.18

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 10.3.1, which you can download by visiting the customer portal.

    tctl version

    Teleport v10.3.1 go1.18

    tsh version

    Teleport v10.3.1 go1.18

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.2.2. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.2.2 go1.18

    tsh version

    Teleport v10.2.2 go1.18

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 10.3.1

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 10.2.2

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Step 1/3. Enable verbose logging

To diagnose problems, you can configure the teleport process to run with verbose logging enabled by passing it the -d flag. teleport will write logs to stderr.

Logs will resemble the following (these logs were printed while joining a Node to a cluster, then terminating the teleport process on the Node):

DEBU [NODE:PROX] Agent connected to proxy: [aee1241f-0f6f-460e-8149-23c38709e46d.tele.example.com aee1241f-0f6f-460e-8149-23c38709e46d teleport-proxy-us-west-2-6db8db844c-ftmg9.tele.example.com teleport-proxy-us-west-2-6db8db844c-ftmg9 localhost 127.0.0.1 ::1 tele.example.com 100.92.90.42 remote.kube.proxy.teleport.cluster.local]. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:414
DEBU [NODE:PROX] Changing state connecting -> connected. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:210
DEBU [NODE:PROX] Discovery request channel opened: teleport-discovery. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:526
DEBU [NODE:PROX] handleDiscovery requests channel. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:544
DEBU [NODE:PROX] Pool is closing agent. leaseID:2 target:tele.example.com:11106 reversetunnel/agentpool.go:238
DEBU [NODE:PROX] Pool is closing agent. leaseID:3 target:tele.example.com:11106 reversetunnel/agentpool.go:238

Debug logs include the file and line number of the code that emitted the log, so you can investigate (or report) what a teleport process was doing before it ran into problems.

It is not recommended to run Teleport in production with verbose logging as it generates a substantial amount of data.

Step 2/3. Generate a debug dump

The teleport binary is a Go program. Go programs assign work to CPU threads using an abstraction called a goroutine. You can get a goroutine dump of a running teleport process by sending it a USR1 signal.

This is especially useful for troubleshooting a teleport process that appears stuck, since you can see which a goroutine is blocked and and why. For example, goroutines often communicate using channels, and a goroutine dump indicates whether a goroutine is waiting to send or receive on a channel.

To generate a goroutine dump, send a USR1 signal to a teleport process:

kill -USR1 $(pidof teleport)

Teleport will print the debug information to stderr. Here what you will see in the logs:

INFO [PROC:1]    Got signal "user defined signal 1", logging diagnostic info to stderr. service/signals.go:99
Runtime stats
goroutines: 64
OS threads: 10
GOMAXPROCS: 2
num CPU: 2
...
goroutines: 84
...
Goroutines
goroutine 1 [running]:
runtime/pprof.writeGoroutineStacks(0x3c2ffc0, 0xc0001a8010, 0xc001011a38, 0x4bcfb3)
	/usr/local/go/src/runtime/pprof/pprof.go:693 +0x9f
...

You can print a goroutine dump without enabling verbose logging.

Step 3/3. Ask for help

Once you have collected verbose logs and a goroutine dump from your teleport binary, you can use this information to get help from the Teleport community and Support team.

Collect your Teleport version

Determine the version of the teleport process you are investigating.

teleport version

Teleport v8.3.7 git:v8.3.7-0-ga8d066935 go1.17.3

You can also collect the versions of the Teleport Auth Service, Proxy Service, and client tools to rule out version compatibility issues.

Get the versions of your client tools:

tctl version

Teleport v9.0.4 git: go1.18

tsh version

Teleport v9.0.4 git: go1.18

Pose your question

If you need help, please ask on our community forum. You can also open an issue on GitHub or create a ticket through the customer dashboard.

If you need help, please ask on our community forum. You can also open an issue on GitHub.

For more information about custom features, or to try our Enterprise edition of Teleport, please reach out to us at sales.

Further reading

This guide showed how to investigate issues with the teleport process. To see how you can monitor more general health and performance data from your Teleport cluster, read our Teleport Diagnostics guide.

For additional sources of Teleport support, please see the Teleport Support and Education Center.

Common Issues

teleport.cluster.local

It is common to see references to teleport.cluster.local within logs and errors in Teleport. This is a special value that is used within Teleport for two purposes and seeing it within your logs is not necessarily an indication that anything is incorrect.

Firstly, Teleport uses this value within certificates (as a DNS Subject Alternative Name) issued to the Auth and Proxy Service. Teleport clients can then use this value to validate the service's certificates during the TLS handshake regardless of the service address as long as the client already has a copy of the cluster's certificate authorities. This is important as there are often multiple different ways that a client can connect to the Auth Service and these are not always via the same address.

Secondly, this value is used by clients as part of the URL when making gRPC or HTTP requests to the Teleport API. This is because the Teleport API client uses special logic to open the connection to the Auth Service to make the request, rather than connecting to a single address as a typical client may do. This special logic is necessary for the client to be able to support connecting to a list of Auth Services or to be able to connect to the Auth Service through a tunnel via the Proxy Service. This means that teleport.cluster.local appears in log messages that show the URL of a request made to the Auth Service, and does not explicitly indicate that something is misconfigured.