Navigating Access Challenges in Kubernetes-Based Infrastructure
Sep 19
Virtual
Register Today
Teleport logoTry For Free
Fork me on GitHub

Teleport

Agent Update Management

While many Teleport resources support agentless mode, agent deployments are sometimes simpler and more convenient. However, large Teleport deployments can create an additional burden: updating all agents.

Teleport supports automatic agent updates for systemd-based Linux distributions using apt, yum or zypper package managers, and Kubernetes clusters.

Update logic and failure modes

An updater is a piece of software deployed next to an agent which is responsible for updating it. Updating multiple agents requires multiple updaters.

We designed the updater to be as decoupled from Teleport as possible. The updater can update agents even when they cannot join the Teleport cluster. Pushing a broken version can happen, but a rollback/roll-forward must always be possible without manually connecting to the resource and fixing the agent.

The updater recurrently fetches the target version from a version server and updates the agent to the target version. Because restarting the agent can disrupt currently open sessions, it will only update the agent in two cases: during a maintenance window or when the agent is unhealthy.

When enrolled in a cluster with automatic updates, the agent will retrieve its maintenance schedule from the Teleport cluster and save it. When a maintenance schedule is available, the updater will honour it. However, if the updater cannot find the maintenance schedule, it will consider the agent unhealthy and perform updates as soon as possible. Similarly, if the updater detects the agent is unhealthy, it immediately applies any pending update to try to recover from a degraded state.

We implemented an additional failsafe: the critical maintenance toggle. The version server can specify that an update is critical. Critical updates are applied even if the updater is outside its regular maintenance window.

Security

When updating the agent, the updater will ensure the new version's authenticity before deploying it. On Linux distributions using apt, yum or zypper, it relies on the existing package signature system. On Kubernetes-based environments, it validates the OCI image signature (using cosign's signature ).

Version server and source of truth

The agent version is subject to the following constraints:

  • the agent must never exceed the Proxy or Auth Service version,
  • the agent must always be no more than one major version below the Proxy or Auth Service version.

The best practice is to always align the agent version with the Proxy and Auth ones. To upgrade Auth and Proxy, follow the Teleport Cluster upgrade guide .

For this reason, all updaters must subscribe to a release channel targeting versions that are compatible with their Teleport cluster. Teleport Cloud users must use the Teleport Cloud version server with the stable/cloud release channel. Self-hosted Teleport users must host their own version server and either update their release channel each time they update their Auth and Proxy instances, or use the stable/rolling channel.

Teleport Cloud

If you are using Teleport Cloud you can use Teleport Cloud's version server only if an instance is enrolled in automatic updates. This version server will always target the best version from a feature, compatibility, security and stability point of view.

If your Teleport Cloud control plane is not automatically updated you must not use automatic agent updates. This is because your Teleport instance version might differ from the other Teleport Cloud instances and might not yet support the latest agent version.

Self-hosted Teleport

If you are self-hosting Teleport Enterprise you can set up automatic agent updates. The updater uses a version channel to retrieve the targeted version for an agent. The Teleport Proxy Service provides a default version channel that matches to its current version. You can configure the default version and set additional channel versions.

You are responsible for ensuring the version channels are compatible with the current Teleport Proxy Service and Teleport Auth Service versions. You must also monitor the agent's health and rollout status to ensure every agent is healthy and running the correct version.

Next steps

Self-hosted users must first set up self-hosted automatic agent upgrades .

After that, you can set enroll agents in automatic updates as part of the upgrading procedure.