Skip to main content
Version: 14.x

Machine ID Troubleshooting Guide

A bot failed to renew a certificate due to a "generation mismatch"​

Symptoms​

The bot will log an error like this:

ERROR: renewable cert generation mismatch: stored=3, presented=2

Subsequent connection attempts by the bot may see errors like the following:

ERROR: failed direct dial to auth server: auth API: access denied [00]
"\tauth API: access denied [00], failed dial to auth server through reverse tunnel: Get \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://example.com:3025/webapi/find\": x509: cannot validate certificate for example.com because it doesn't contain any IP SANs"
"\tGet \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://example.com:3025/webapi/find\": x509: cannot validate certificate for example.com because it doesn't contain any IP SANs"

In particular, note the message auth API: access denied.

In self-hosted Teleport deployments, the Teleport Auth Service will also provide some additional context:

[AUTH]      WARN lock targeting User:"bot-example" is in force: The bot user "bot-example" has been locked due to a certificate generation mismatch, possibly indicating a stolen certificate. auth/apiserver.go:224

Explanation​

Machine ID (with token-based joining) uses a certificate generation counter to detect potentially stolen renewable certificates. Each time a bot fetches a new renewable certificate, the Auth Service increments the counter, stores it on the backend, and embeds a copy of the counter in the certificate.

If the counter embedded in your bot certificate doesn't match the counter stored in Teleport's Auth Server, the renewal will fail and the bot user will be automatically locked.

Renewable certificates are exclusively stored in the bot's internal data directory, by default /var/lib/teleport/bot. It's possible to trigger this by accident if multiple bots are started using the same internal data directory, or if this internal data is otherwise being shared between multiple bot instances.

Additionally, if a bot fails to save its freshly renewed certificates (for example, due to a filesystem error) and crashes, it will attempt a renewal with old certificates and trigger a lock.

Resolution​

Before unlocking the bot, try to determine if either of the two scenarios described above apply. If the certificates were stolen, there may be underlying security concerns that need to be addressed.

Otherwise, first ensure only one bot instance is using the internal data directory. Multiple bots can be run on a single system, but separate data directories must be configured for each.

Additionally, ensure the internal data is not being shared with or copied to any other nodes, for example via a shared NFS volume. If you'd like to share certificates between nodes, only copy or share content from destination directories (usually /opt/machine-id) rather than the internal data directory (by default, /var/lib/teleport/bot).

Once you have addressed the underlying cause, follow these steps to reset a locked bot:

  1. Remove the lock on the bot's user
  2. Reset the bot's generation counter by deleting and re-creating the bot

To remove the lock, first find and remove the lock targeting the bot user:

$ tctl get locks
kind: lock
metadata:
  id: 1658359514703080513
  name: 5cee949f-5203-4f3b-9805-dac35d798a16
spec:
  message: The bot user "bot-example" has been locked due to a certificate generation
    mismatch, possibly indicating a stolen certificate.
  target:
    user: bot-example
version: v2

$ tctl rm lock/5cee949f-5203-4f3b-9805-dac35d798a16

Next, reset the generation counter by deleting and recreating the bot:

$ tctl bots rm example

$ tctl bots add example --roles=foo,bar

Finally, reconfigure the bot with the new token and restart it. It will detect the new token and automatically reset its internal data directory.

tbot shows a "bad certificate error" at startup​

Symptoms​

Restarting a tbot process outputs a log like the following:

INFO [TBOT]      Successfully loaded bot identity, valid: after=2022-07-21T21:49:26Z, before=2022-07-21T22:50:26Z, duration=1h1m0s | kind=tls, renewable=true, disallow-reissue=false, roles=[bot-test], principals=[-teleport-internal-join], generation=2 tbot/tbot.go:281
ERRO [TBOT]      Identity has expired. The renewal is likely to fail. (expires: 2022-07-21T22:50:26Z, current time: 2022-07-25T20:18:33Z) tbot/tbot.go:415
WARN [TBOT]      Note: onboarding config ignored as identity was loaded from persistent storage tbot/tbot.go:288
ERRO [TBOT]      Failed to resolve tunnel address Get "https://auth.example.com:3025/webapi/find": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs reversetunnel/transport.go:90
ERRO [TBOT]      Failed to resolve tunnel address Get "https://auth.example.com:3025/webapi/find": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs reversetunnel/transport.go:90
ERROR: failed direct dial to auth server: Get "https://teleport.cluster.local/v2/configuration/name": remote error: tls: bad certificate
"\tGet \"https://teleport.cluster.local/v2/configuration/name\": remote error: tls: bad certificate, failed dial to auth server through reverse tunnel: Get \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://auth.example.com:3025/webapi/find\": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs"
"\tGet \"https://teleport.cluster.local/v2/configuration/name\": Get \"https://auth.example.com:3025/webapi/find\": x509: cannot validate certificate for auth.example.com because it doesn't contain any IP SANs"

In particular, note the log line: "Identity has expired. The renewal is likely to fail."

Explanation​

Token-joined bots are unable to reauthenticate to the Teleport Auth Service once their certificates have expired. Tokens in token-based joining (as opposed to AWS IAM joining) can only be used once, so when the bot's internal certificates expire, it will not be able to connect.

When a bot's identity expires, certain parameters associated with the bot on the Auth Service must be reset and a new joining token must be issued. The simplest way to accomplish this is by removing and recreating the bot, which purges all server-side data and issues a new joining token.

Resolution​

Remove and recreate the bot, replacing the name and role list as desired:

$ tctl bots rm example
$ tctl bots add example --roles=access

Copy the resulting join token into the existing bot config—either the --token CLI flag or the onboarding.token parameter in tbot.yaml—and restart the bot. It will detect the new token and rejoin the cluster as normal.

SSH connections fail with ssh: handshake failed: ssh: unable to authenticate​

Symptoms​

When attempting to connect to a node via SSH, connections fail with an error like the following:

$ ssh -F /opt/machine-id/ssh_config [email protected]
ERROR: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

ERROR: unable to execute tsh
executing `tsh proxy`
exit status 1

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

In particular, note the ssh: unable to authenticate message.

Explanation​

This can occur when attempting to log into the node as a user not listed as a principal on the SSH certificate.

You can verify this by viewing the tbot logs and looking for the log message when impersonated certificates for the matching outputs were renewed.

In the following example, the only principal listed for the identity in /opt/machine-id is alice (via the access role):

INFO [TBOT]      Successfully renewed impersonated certificates for directory /opt/machine-id, valid: after=2022-07-21T21:49:26Z, before=2022-07-21T22:50:26Z, duration=1h1m0s | kind=tls, renewable=false, disallow-reissue=true, roles=[access], principals=[alice -teleport-internal-join], generation=0 tbot/renew.go:630

However, the SSH command attempted to log in as bob.

Resolution​

Ensure the bot identity is allowed to log in as the requested user by taking any of the following actions:

  • Changing the SSH command to log in as an allowed user
  • Modifying the access role to allow the alice principal
  • Adding a role granting login via the bob principal

Note that if roles are added or modified, the certificates will need to be renewed for the changes to take effect. The bot will renew certificates on its own after the renewal interval (by default, 20 minutes), but you can trigger a renewal immediately by either restarting the tbot process or sending it a reload signal:

## If using systemd, you can restart the process:
$ systemctl restart machine-id
## Alternatively, you can send `tbot` a reload signal directly:
$ pkill -sigusr1 tbot

Database requests fail with database "example" not found, but the database exists​

Symptoms​

When requesting Database Access certificates, the certificate request fails with an error like the following:

ERROR: Failed to generate impersonated certs for directory /opt/machine-id: database "example" not found
database "example" not found

However, the database exists and can be seen by regular users via tsh:

$ tsh db ls
Name       Description Allowed Users Labels  Connect
---------- ----------- ------------- ------- -------
example                [alice]       env=dev

Explanation​

Unlike regular Teleport users, Machine ID bot users are granted only minimal Teleport RBAC permissions and are not allowed to view or list databases by default unless granted permission via one or more roles.

Resolution​

Per the Machine ID Database Access Guide, ensure at least one role providing database permissions has been granted to the output listed in the error.

For example, note the rules section in the following example role:

kind: role
version: v5
metadata:
  name: machine-id-db
spec:
  allow:
    db_labels:
      '*': '*'
    db_names: [example]
    db_users: [alice]
    rules:
      - resources: [db_server, db]
        verbs: [read, list]

Ensure the bot has a role that grants it at least these RBAC rules. If desired you can examine bot roles with tctl to ensure the necessary rules have been granted:

$ tctl get role/machine-id-db

If the role is missing database permissions, it can be modified:

## save the role to a local file
$ tctl get role/machine-id-db > db-role.yaml
## edit the role as necessary
$ nano db-role.yaml
## replace the existing role with the modified copy
$ tctl create -f db-role.yaml
note

By default, outputs (like /opt/machine-id) are granted all roles provided to the bot via tctl bots add --roles=..., but it's possible to grant only a subset of these roles using the roles: ... parameter in tbot.yaml.

If permissions are unexpectedly missing, ensure tbot.yaml requests your database role, either by relying on default behavior or adding the role to the roles: ... list.

Once fixed, restart or reload the tbot clients for the updated role to take effect.

If the bot was not granted the role initially, the simplest solution is to delete and recreate the bot, being sure to include the role in the --roles=... flag:

$ tctl bots rm example
$ tctl bots add example --roles=foo,bar,machine-id-db

Kubernetes connections are failing with Unable to connect to the server: x509: certificate signed by unknown authority​

Symptoms​

A self-hosted Teleport cluster is connecting Machine ID to Kubernetes clusters with the following errors. This can happen for non-TLS configured Teleport clusters.

E0322 22:53:31.653051    1699 memcache.go:265] couldn't get current server API group list: Get "https://teleport.example.com:443/api?timeout=32s": x509: certificate signed by unknown authority

To confirm the TLS routing mode check the value of the proxy.tls_routing_enabled key with this command, substituting your proxy address:

curl https://teleport.example.com:443/webapi/ping | jq

If the value is false then this is a non-TLS routing configuration.

Explanation​

Proxies configured with non-TLS routing use specific ports for various types of traffic. That requires that a Kubernetes connection use its designated port. Currently Machine ID requires that the Kubernetes public address is set to use the correct port. Otherwise it will use the Proxy web port which can cause these type of errors.

Resolution​

The Kubernetes public address is via the kube_public_addr within the Teleport proxy_service configuration by administrators. The proxy will require a restart after the configuration is updated.

proxy_service:
  enabled: true
  kube_listen_addr: 0.0.0.0:3026
  kube_public_addr: teleport.example.com:3026

Retrieve the configuration listing from the proxy web address to confirm the Kubernetes public address is populated in proxy.kube.public_addr.

curl https://teleport.example.com:443/webapi/ping | jq

Destination kubernetes_secret: identity-output must be a directory in exec plugin mode​

By default, when outputting a Kubernetes identity, tbot outputs make use of a Kubernetes exec plugin to always provide the latest version of the credentials.

When outputting a Kubernetes identity to a Kubernetes secret, however, it is important to disable the use of the exec plugin by adding disable_exec_plugin: true to the output. This means that a static kubeconfig file with embedded short-lived credentials is written instead:

outputs:
  - type: kubernetes
    # Specify the name of the Kubernetes cluster you wish the credentials to
    # grant access to.
    kubernetes_cluster: example-k8s-cluster
    # Required when outputting a Kubernetes identity to a Kubernetes secret.
    disable_exec_plugin: true
    destination:
      type: kubernetes_secret
      # For this guide, identity-output is used as the secret name.
      # You may wish to customize this. Multiple outputs cannot share the same
      # destination.
      name: identity-output

Failure to add the disable_exec_plugin flag will result in a warning being displayed: Destination kubernetes_secret: identity-output must be a directory in exec plugin mode.