Bound Keypair Joining Admin Guide

Report an IssueView as Markdown

Is this page helpful?

How can we improve this page?

This guide discusses various tasks users administering Bound Keypair clients may need to perform over the lifespan of the bot or agent.

Allowing additional recovery attempts

When using the standard recovery mode, only a configured number of recovery attempts can be made. If the limit is reached, no further recovery attempts can be made until the limit is increased.

To increase this limit and allow an expired client to join again, edit the token using tctl edit:

tctl edit token/example-token

Find the spec.bound_keypair.recovery.limit field and increment the limit by the desired amount. You are free to select any desired threshold. For example, consider these use cases:

• If human intervention is desired for each join attempt you can increase this value by 1. This single recovery attempt will be immediately consumed, so future recoveries will again require human intervention, and may result in downtime.

  While this approach makes downtime likely, it does ensure a human verifies the state of the client on each recovery.

• If you want human intervention for each recovery, but want to avoid downtime, you can increase this value by 2. The first attempt will be consumed immediately, but the client will have one recovery attempt for automatic future use.

  A human user can periodically audit the recovery count and client to ensure a recovery attempt is always available and the host is behaving as expected.

• Any larger value will increase the amount of time required between human intervention. You can select your tolerance for automatic client recoveries as desired.

Alternatively, if you wish to allow an unlimited number of automatic recovery attempts, refer to the entry below on the relaxed recovery mode.

Note that the recovery limit is always relative to the recovery counter (in the status.bound_keypair.recovery_count field in the token resource). It is valid to decrease the limit or set it to zero, however doing so may prevent future recovery attempts until the limit is increased again.

Additionally, note that join state verification is still required, and will prevent multiple concurrent uses of the same keypair and token. In other words, increasing the recovery limit will not allow multiple clients to join.

Allowing unlimited recovery attempts

To allow unlimited recovery attempts, the spec.bound_keypair.recovery.mode field should be set to relaxed. To do this, use tctl edit to edit the token:

tctl edit token/example-token

Find or create the spec.bound_keypair.recovery.mode field and set the value to relaxed. Save the file and quit your editor to update the token.

When the recovery mode is set to relaxed, the limit field is ignored and the status.bound_keypair.recovery_count field may increase beyond the written limit. If the mode is later changed back to standard, be aware that future recovery attempts will fail unless the limit is increased to accommodate the current value of recovery_count.

Note that when relaxed mode is in use, join state verification is still required and will prevent multiple concurrent uses of the same keypair and token. If your use case requires this, you can disable join state verification, but doing so does impact the security of the token.

Requesting a keypair rotation

To request a keypair rotation, set the .spec.bound_keypair.rotate_after field to contain a timestamp. On the next authentication attempt after that timestamp has elapsed, the bot will automatically rotate its keypair.

To simplify this process, you can use the tctl bound-keypair rotate helper:

tctl bound-keypair rotate token-name

This sets the timestamp to the current time. Note that by default bots only reauthenticate every 20 minutes, so it may take some time for the request to be acknowledged. You can monitor the rotation status by watching the token's .status.bound_keypair.last_rotated_at field.

If you want to force an early rotation and have access to the bot host, you can restart the tbot process, or send it a signal with pkill -usr1 tbot to request an early rotation.

Note that the previous 10 keypairs are retained on the client for use in case of a cluster rollback; refer to the cluster rollback section for additional information.

Agent note

Note that non-bot agents manage their certificates differently from bots: they generally do not reauthenticate to Teleport on a regular basis and only do so under certain conditions, like if role changes are requested (adding a new service role) or cluster CAs are rotated. Bound Keypair rotation can only happen during the bound keypair challenge process, which for agents only happens during these rare, manual events.

Locking a bound_keypair client

The simplest way to lock out a client - either bot or agent - that joined using the bound_keypair join method is to use a join token lock target:

tctl lock --join-token=token-name

As a bound keypair token is linked to a single client, this will effectively lock it. It will not be able to reauthenticate, recover, interact with the Teleport API, or otherwise use its credentials until the lock is removed.

Bot-specific locking notes

Note that if a bot is locked for long enough - bots have a 1 hour certificate TTL by default - its certificates will expire. If you intend to remove this lock and reinstate the bot, you may also need to increase the recovery limit (.spec.bound_keypair.recovery.limit) to accommodate the additional recovery attempt.

Other lock targets can also be used, but are not preferred:

• Bot instance (tctl lock --bot-instance-id ...): will lock only a single instance of the bot. Note that if the recovery limit allows for it, the automatic recovery process will attempt to rejoin and, if successful, will generate a new bot instance ID.
• Bot name (tctl lock --user bot-<name>): will lock all bots using the same bot / user. This may be overly broad and lock other instances running under this bot user.

Locking agents by their unique ID

Similar to bots, agents have a unique ID. Find the unique ID by running:

tctl inventory ls

Find the desired host in the list and copy its ID from the "Server ID" column. Then, create a lock for that ID:

tctl lock --server-id=1234abcd-1234-5678-abcd-1234abcd5678

Recovering a locked bound_keypair client

Bots or agents joined with the bound_keypair join method can become automatically locked under various conditions, including:

• Failing to correctly complete join state verification
• Connecting with certificates that have an invalid generation counter (bots only)
• Locked manually by a cluster admin

To recover a client that has become locked, first ensure the client's internal storage has not been compromised, like the bot's storage directory (usually /var/lib/teleport/bot), or the agent's data directory (/var/lib/teleport). These locking conditions are designed to trigger if more than one client tries to join using a copy of the same certificates and private key. This can occur due to a misconfiguration or due to an attacker copying a bot's credentials, so ideally the latter should be ruled out before unlocking the client.

Next, determine the name (UUID) of the lock or locks targeting the client:

tctl get lock
kind: lockmetadata:  name: 372af058-76d1-4e64-93da-3b04d7d03ac2spec:  target:    user: bot-exampleversion: v2---kind: lockmetadata:  name: 791d0b1d-01b4-4752-8a99-9b2908aebfaespec:  target:    bot_instance_id: e7d494ae-a0ff-4d12-b935-de5e2025f667version: v2---kind: lockmetadata:  name: a69fdbb2-8e53-406a-b453-48b2cda6991dspec:  target:    join_token: example-token-nameversion: v2---kind: lockmetadata:  name: 7a892628-97a6-4038-be36-d4739bf78109spec:  target:    server_id: 3d8b0237-f2e0-4add-aec1-7f293278b35eversion: v2

Note the different locks and lock targets shown above. Bots can be targeted by any of these lock types:

• their Teleport user name (bot-example)
• the bot instance ID (a UUID)
• the join token name.

For agents, a different set of locks can be generated:

• the join token name
• the server ID (only manually)

Locks created automatically for clients using Bound Keypair Joining will typically use a join_token target, but a lock targeting any of these values could be created manually.

Note that locks may have a message field containing details about why the lock was created.

Once the lock name(s) have been determined, remove each using tctl rm:

tctl rm lock/372af058-76d1-4e64-93da-3b04d7d03ac2

Next, join state should be reset. Use tctl edit to set the token's recovery mode to insecure, but make a note of the current value (standard or relaxed):

tctl edit token/example-token

Change the .spec.bound_keypair.recovery.mode field to insecure, save, and quit the editor.

The client is now allowed to rejoin. Given sufficient time it will retry on its own, but if you have access to the host, run the following to restart it:

• For bots: systemctl restart tbot
• For agents: systemctl restart teleport

The client should now be able to join successfully. You can monitor progress by watching for new audit events in Teleport's web UI, or by waiting for the recovery counter to increase:

tctl get token/example-token --format=json | jq '.[].status.bound_keypair.recovery_count'

Once the client has joined successfully, reset the recovery mode to its previous value using tctl edit:

tctl edit token/example-token

If you do suspect the client's credentials may have been compromised, you may also want to request a keypair rotation in addition to taking other steps to ensure the host is properly secured.

Disabling join state verification

It is occasionally useful to intentionally disable join state verification. For example, this can enable use with:

• CI/CD providers without an explicit delegated join method.
• Nodes with immutable storage that cannot store an updated join state document after each join.

Before continuing, be aware that disabling join state verification will prevent Teleport from detecting if multiple clients are joining using the same bound keypair token. In other words, if the private key is copied by an attacker, they will be able to join indefinitely. Take care to protect the keypair, and for bots, make certain to limit access from the bot identity using Teleport's RBAC system.

When ready, use tctl edit to modify the Bound Keypair token:

tctl edit token/example-token

Find or add the spec.bound_keypair.recovery.mode field and set it to insecure. Save and quit your editor to update the token.

With the mode set to insecure, the recovery.limit is ignored, allowing unlimited reuse of the token, and join state verification is disabled, allowing concurrent or stateless reuse.

Recovery after a cluster rollback

If your Teleport cluster is rolled back for any reason, joining clients may fail join state verification as their local join state document may not match the values currently (or previously) known to Teleport.

The simplest workaround is to temporarily set all bound keypair tokens to insecure recovery mode for the first join attempt following a cluster restore. Once they've joined once, they will once again have a valid join state, so the recovery mode can be restored to its previous value.

To change the recovery mode, use tctl edit to modify the token resource:

tctl edit token/example-token

Find the spec.bound_keypair.recovery.mode field, and set the value to "insecure". Repeat this for each bound keypair token. Wait for all bound keypair clients to reauthenticate, and repeat this process to restore the recovery mode to its previous value.

If keypairs were rotated between the snapshot and restore of the Teleport cluster, note that clients only keep a record of the previous 10 keypairs. This means server-side recovery may be impossible if the keypair expected by the restored Teleport cluster has been rotated out of the client-side history, or if the client-side history has been lost or deleted.

Manually rotating static keys

Static keys prevent automatic key rotation as clients cannot update keys in an arbitrary - and potentially remote - keystore. However, it may still be possible to automate rotation if your environment or secret store allows you to update secrets through an API.

The specific steps needed to automate these will vary based on your environment, but the general steps are:

1. Generate a new keypair on any node using the tbot client:

   tbot keypair create --proxy-server example.teleport.sh:443 --static --format json

2. Parse the .public_key and .private_key values using your tool of choice, like jq or any other JSON parser.

3. Replace the token in Teleport to trust the new public key, using the value in the .public_key field:

   cat my-token.yaml
   version: v2kind: tokenmetadata:  name: my-tokenspec:  bot_name: example-bot  bound_keypair:    onboarding:      initial_public_key: <insert .public_key value here>    recovery:      mode: insecure  join_method: bound_keypair  roles:  - Bot
   tctl create -f my-token.yaml

4. Insert the new private key into your keystore. This will vary depending on which keystore or provider you are using.

   • If passing the private key via an environment variable, copy the value directly
   • If passing the private key via file, decode the base64-encoded private key first:

     tbot keypair create --proxy-server example.teleport.sh:443 --static --format json | jq -r .private_key | base64 -d

     ...and store the result as needed.

5. Future jobs should now use the new keypair.

Frequently rotating static keys can help to mitigate the security tradeoffs of insecure recovery. See the concepts page for more information about static keys.

Limitations of Bound Keypair joining for Teleport Agents

As of Teleport v18.8, the bound_keypair join method supports standard agent types as well as Machine & Workload Identity bots.

Note that the benefits of Bound Keypair joining for agents are more limited, so most users will still probably prefer to use the traditional token join method:

• Agents are ultimately still issued long-lived certificates and do not generally renew these identities outside of limited circumstances, namely during cluster CA rotation or when new system roles are added to an existing agent.
• As most Agents only join a single time, Bound Keypair's Join State Verification is effectively unused.
• As keypair rotation can only take place at join time, agents will only honor a keypair rotation request under certain conditions, like a role change.
• Agent identities don't have access privileges for other Teleport resources, so they don't benefit from or take advantage of the additional protections provided by Bound Keypair joining.
• The Bound Keypair join method is more complex than token joining, and does introduce potential new failure modes.
• Agents can only join via registration secrets (Bound Keypair's default mode) or static key files. Non-static preregistered keys are not currently supported.

That said, you might consider Bound Keypair for Agents in certain circumstances:

• You want an alternative to standard token-type tokens that is strictly single-use.

  Bound Keypair tokens (and their registration secrets) are strictly paired to a single host (be it a bot or agent), so providing a bound keypair token and registration secret to a user would only allow them to join a single node.

• You do not want any shared secrets in your joining process. Bound Keypair's static keys allow you to generate a keypair on the target node itself. Only the public key would ever leave the system, and no secrets or private key material would ever have to leave the host.

If you'd like to join an agent using Bound Keypair joining, refer to these guides:

• Joining an agent using Bound Keypair
• Joining an agent using Bound Keypair Static Keys

Was this page helpful?

How can we improve this page?