Skip to main content

Managed Updates for Teleport Agents and Bots

warning

This document describes Managed Updates for Agents and Bots (v2), which replaces Managed Updates for Agents (v1).

For Managed Updates v1 instructions, see Managed Updates for Agents (v1).

For Managed Updates, a binary called teleport-update is distributed in all Teleport packages, alongside the teleport, tbot, and other binaries. Admins configure updates by managing the autoupdate_version and autoupdate_config dynamic resources.

This document covers how to use teleport-update and the autoupdate_* resources to manage automated agent and bot updates from Teleport. It describes:

teleport-update supports:

  • Teleport Enterprise and Teleport Community Edition
  • Both cloud and self-hosted Teleport Enterprise deployments
  • Regular and FIPS variants of Teleport
  • amd64, arm64, and other supported CPU architectures
  • systemd-based operating systems, regardless of the package manager used
Compatibility between Managed Updates v1 and v2

The Managed Updates v2 teleport-update binary is backwards-compatible with the cluster_maintenance_config resource. The Managed Updates v1 teleport-upgrade script is forwards-compatible with the autoupdate_config and autoupdate_version resources. Agents connected to the same cluster will all update to the same version.

If the autoupdate_config resource is configured, it takes precedence over cluster_maintenance_config. This allows for a safe, non-breaking, incremental migration between Managed Updates v1 and v2. If autoupdate_config is not present and autoupdate_version is present, the autoupdate_config settings are implicitly derived from cluster_maintenance_config.

Regardless of how the cluster is configured, teleport-update is capable of managing both Teleport Agent and tbot installations, while teleport-upgrade is only capable of managing Teleport Agents.

Users of cloud-hosted Teleport Enterprise have been migrated to Managed Updates v2 and should migrate their agents to teleport-update as soon as possible.

How it works

Managed Updates for Agents and Bots are designed to manage long-running, unattended Teleport clients, such as Teleport Agents and tbot. This is different from Managed Updates for Client Tools, which are designed to manage interactive Teleport clients, such as tsh and tctl.

When Managed Updates are enabled, a Teleport updater is installed alongside each new Teleport Agent or tbot. The updater communicates with the Teleport Proxy Service to determine when an update is available and if it should perform the update now.

Each installation belongs to an update group. The update schedule specifies when each group is updated. The schedule is stored in the autoupdate_config resource and can be edited via tctl. The tctl autoupdate agents subcommands are used to interact with the rollout for both Teleport Agents and long-running tbot installations.

For Linux server-based installations, teleport-update command configures Managed Updates for Teleport Agents and tbot locally on the server.

For Kubernetes-based installations, the teleport-kube-agent Helm chart deploys a controller that automatically updates the main Teleport container.

Agents and bots that were installed before Managed Updates was enabled on the cluster usually need to be manually enrolled into Managed Updates.

Prerequisites

  • Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components in your cluster.
  • Teleport Agent or tbot installations that are not yet enrolled in Managed Updates.
    • A running Teleport cluster. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

    • The tctl and tsh clients.

      Installing tctl and tsh clients
      1. Determine the version of your Teleport cluster. The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/find and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:

        TELEPORT_DOMAIN=teleport.example.com:443
        TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"
      2. Follow the instructions for your platform to install tctl and tsh clients:

        Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

        curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg

        In Finder double-click the pkg file to begin installation.

        danger

        Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and [email protected] to your Teleport username:
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status

    Cluster teleport.example.com

    Version 18.4.1

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Quick setup for existing Linux Agent and Bot installations

Users can enable Managed Updates v2 on Linux servers that are already running a Teleport Agent by running the following command on every server:

sudo teleport-update enable
note

If this command is not available, update the teleport package to the latest version that is supported by your cluster.

The teleport-update enable command will disable (but not remove) the v1 updater if present. No other action is necessary.

If everything is working, the v1 updater package can be removed:

sudo apt remove teleport-ent-updater

If the v2 updater does not work, your installation can be reverted back to manual updates or the v1 updater (if it has not been removed):

sudo teleport-update uninstall

If Teleport was installed via the apt or yum package, teleport-update uninstall will revert the running version of Teleport back to the version provided by the package.

Migrating Bots

Existing tbot installations require additional steps to be converted to Managed Updates.

When teleport-update enable is run, a disabled systemd service is created at /etc/systemd/system/tbot.service if a service does not already exist at that location.

If a custom tbot systemd service is already installed at /etc/systemd/system/tbot.service, a warning will be displayed when teleport-update enable is run. To overwrite that custom service and replace it with an updater-managed service, run the following command:

sudo teleport-update enable --overwrite

If a custom tbot systemd service is installed with a different name (e.g., /etc/systemd/system/machineid.service), it must be stopped if it shares the same configuration and data directories as the updater-managed service:

sudo systemctl disable machineid --now

After teleport-update enable has successfully created the service, its output will recommend that you run the following command to enable tbot.service:

sudo systemctl enable tbot --now

Note that you must have a valid /etc/tbot.yaml file to use tbot. See Deploying tbot on Linux for more information.

Quick setup for new Linux Agents and Bot installations

The Web UI onboarding and Install Script are the fastest ways to onboard new Linux servers. However, you may also use teleport-update by itself to set up a Teleport Agent and/or tbot manually. Note that web-based agent enrollment does not automatically configure tbot. See the end of this section for information on how to configure and enable tbot.

Users can create a new installation of Teleport using any version of the teleport-update binary. First, download copy of the teleport-update tarball from the Agent Installer & Updater section of the downloads page. Next, invoke teleport-update to install the correct version for your cluster.

tar xf teleport-update-[version].tgz
cd teleport-update-[version]
sudo ./teleport-update enable --proxy example.teleport.sh

After Teleport is installed, you can create /etc/teleport.yaml, either manually or using teleport configure. After, the Teleport Agent can be enabled and started via the systemctl command:

sudo systemctl enable teleport --now

Similarly, you can create an /etc/tbot.yaml file, either manually or using tbot configure. See Deploying tbot on Linux for more information.

After, tbot can be enabled and started via the systemctl command:

sudo systemctl enable tbot --now

Configuring managed agent and tbot updates

Managed agent and bot updates are configured via two Teleport resources:

  • autoupdate_config controls the update schedule
  • autoupdate_version controls the desired version

Self-hosted Teleport users must configure both autoupdate_config and autoupdate_version.

Cloud-hosted Teleport Enterprise users can configure the autoupdate_config, while the autoupdate_version is managed by Teleport Cloud. Updates will roll out automatically during the first chosen maintenance window that is at least 36 hours after the cluster version is updated.

To configure Managed Updates in your cluster, you must have access to the autoupdate_config and autoupdate_version resources. By default, the editor role can modify both resources.

Configuring the schedule

For both cloud-hosted and self-hosted editions of Teleport, an update schedule may be set with the autoupdate_config resource. The default resource looks like this:

kind: autoupdate_config
metadata:
  name: autoupdate-config
spec:
  agents:
    mode: enabled
    strategy: halt-on-error
    schedules:
      regular:
        - name: default
          days: [ "Mon", "Tue", "Wed", "Thu" ]
          # start_hour is in UTC
          start_hour: 16

This example configures a single group named "default" for all agents. All agents will be placed in this group, as agents with missing or unknown groups are always placed in the last listed group. Currently, only the "regular" schedule is user-configurable.

For example, a Teleport user with staging and production environments might create a custom schedule that looks like this:

kind: autoupdate_config
metadata:
  name: autoupdate-config
spec:
  agents:
    mode: enabled
    strategy: halt-on-error
    schedules:
      regular:
        - name: staging
          days: [ "Mon", "Tue", "Wed", "Thu" ]
          start_hour: 4
        - name: production
          days: [ "Mon", "Tue", "Wed", "Thu" ]
          start_hour: 5
          wait_hours: 24

This schedule would update agents and bots in the staging group at 4 UTC, and then update the production group at 5 UTC the next day. The production group will not execute update until the staging group has updated. The wait_hours field sets a minimum duration between groups, ensuring that production happens the day after staging, and not one hour after.

Two update rollout strategies are available:

  • The halt-on-error strategy provides predictable, sequential updates across environments. It's ideal for traditional development pipelines where you want to ensure that development environments are successfully updated before proceeding to staging and production.
  • The time-based strategy is designed for environments where update groups are independent of each other, such as geographical regions or different teams. It allows updates to occur whenever the specified maintenance window is active for a group, regardless of the status of other groups. This strategy does not provide ordering guarantees across groups.

With the halt-on-error strategy, the canary_count field can be set on each group to specify a number of randomly selected agents (fewer than five) to update and verify before proceeding to the rest of the agents in the group. This can be used to reduce the impact of a failed update that might not be caught by earlier groups due to environment differences.

You can find more information in the Managed Updates v2 resource reference

Except for autoupdate_config.agents.mode, changes to autoupdate_config fields take effect during the next version rollout. A new rollout happens when autoupdate_version is changed and targets a new version. Version is automatically updated for Cloud-hosted Teleport clusters; for self-hosted ones you have to update the version manually, see the dedicated guide section.

Setting the version (self-hosted only)

For cloud-hosted Teleport Enterprise, Managed Updates are enabled by default. The autoupdate_version resource is managed for you and cannot be edited. This ensures your agents are always up-to-date and running the best version for your Teleport cluster.

important

Self-hosted Teleport users must specify which version their agents and bots should update to via the autoupdate_version resource. If the resource does not exist, agents and bots will not update.

Create a file called autoupdate_version.yaml containing:

kind: autoupdate_version
metadata:
  name: autoupdate-version
spec:
  agents:
    start_version: 17.2.0
    target_version: 17.2.1
    schedule: regular
    mode: enabled

This resource is used to deploy new versions of Teleport to your agents and bots. The cluster will update agents and bots to target_version according to the update schedule specified in the autoupdate_config.

The start_version is only used to determine the version used for newly connected agents and bots when their update window has not occurred yet. This is useful to prevent version drift within groups, but some users may prefer to set both version fields to the same version.

Run the following command to create or update the resource:

tctl create -f autoupdate_version.yaml

Changes to autoupdate_version can take up to a minute to create a new rollout. You can observe the current rollout state with the command:

tctl autoupdate agents status
Agent autoupdate mode: enabledRollout creation date: 2025-03-10 15:01:45Start version: 1.2.3Target version: 1.2.4Rollout state: ActiveStrategy: halt-on-error
Group Name State Start Time State Reason---------- --------- ------------------- ------------------------dev Active 2025-03-11 12:00:10 can_startstage Unstarted previous_groups_not_doneprod Unstarted previous_groups_not_done

Monitoring tbot updates

Unlike the Teleport Agent, tbot does not have a persistent connection to the cluster and cannot be monitored directly during upgrades.

However, tbot installation failures are still tracked if tbot is installed alongside a running Teleport Agent.

If a tbot upgrade fails and tbot is installed alongside an agent, both tbot and the agent will be rolled back to the previous version, and the update group may be marked as failed. If tbot is installed without an agent, tbot will still be rolled back to the previous version, but the upgrade may still progress to further groups.

Similarly, tbot installations are only considered candidates for canary installations if they are deployed alongside a running Teleport Agent.

Managing Rollouts

Managed Update groups are automatically progressed as configured by the autoupdate_config resource. However, it is possible to manually trigger or rollback updates for a selection of groups using tctl autoupdate agents commands:

Commands: autoupdate agents status Prints agents auto update status. autoupdate agents report Aggregates the agent autoupdate reports and displays agent count per version and per update group. autoupdate agents start-update Starts updating one or many groups. autoupdate agents mark-done Marks one or many groups as done updating. autoupdate agents rollback Rolls back one or many groups.

For example, if an earlier group cannot be started, the other groups can be triggered manually:

tctl autoupdate agents start-update stage prod
Started updating agents groups: [stage prod].New agent rollout status:
Group Name State Start Time State Reason---------- --------- ------------------- --------------dev Unstarted cannot_startstage Active 2025-03-10 15:04:16 manual_triggerprod Active 2025-03-10 15:04:16 manual_trigger

While individual agents will automatically and immediately rollback if they fail health checks during the update, regressions or breaking changes in new versions could make it desirable to rollback an agent update to an earlier version. The tctl autoupdate agents rollback command can be used to rollback one or more groups to the cluster's start_version. Rollbacks are immediate and do not wait for canaries to complete.

Migrating agents on Linux servers to Managed Updates

Finding unmanaged agents

Use the tctl inventory ls command to list connected agents along with their current version. Use the --upgrader=none flag to list agents that are not enrolled in managed updates.

tctl inventory ls --upgrader=none
Server ID Hostname Services Version Upgrader------------------------------------ ------------- -------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node v14.4.5 none...

Use the --upgrader=unit flag to list agents that are using Managed Updates v1 and should be updated to Managed Updates v2:

tctl inventory ls --upgrader=unit
Server ID Hostname Services Version Upgrader------------------------------------ ------------- -------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-131 Node v14.4.5 unit...

Agents enrolled into Managed Updates v2 can be queried with the --upgrader=binary flag.

Note that it may take several minutes for newly upgraded agents to be reflected in the inventory output.

Enrolling unmanaged agents

  1. For each agent ID returned by the tctl inventory ls command, copy the ID and run the following tctl command to access the host via tsh:

    HOST=00000000-0000-0000-0000-000000000000
    USER=root
    tsh ssh "${USER?}@${HOST?}"
  2. Run teleport-update enable on each agent you would like to enroll into Managed Updates v2:

    sudo teleport-update enable
  3. Confirm that the version of the teleport binary is the one you expect:

    teleport version
  4. Remove the Managed Updates v1 updater if present:

    sudo apt remove teleport-ent-updater
Running the agent as a non-root user

If you changed the agent user to run as non-root, create /etc/teleport-upgrade.d/schedule and grant ownership to your Teleport user:

sudo mkdir -p /etc/teleport-upgrade.d/
sudo touch /etc/teleport-upgrade.d/schedule
sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule

While teleport-update does not read this file, teleport will warn if it cannot disable the Managed Update v1 updater using this file.

Enroll Kubernetes agents in Managed Updates

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

  1. Add the following chart values to the values file for the teleport-kube-agent chart:

    updater:
      enabled: true
    
  2. Update the Teleport Helm repository to include any new versions of the teleport-kube-agent chart:

    helm repo update teleport
  3. Update the Helm chart release with the new values:

    helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version="18.2.8"
  4. You can validate the updater is running properly by checking if its pod is ready:

    kubectl -n teleport-agent get pods
    NAME READY STATUS RESTARTS AGE<your-agent-release>-0 1/1 Running 0 14m<your-agent-release>-1 1/1 Running 0 14m<your-agent-release>-2 1/1 Running 0 14m<your-agent-release>-updater-d9f97f5dd-v57g9 1/1 Running 0 16m
  5. Check for any deployment issues by checking the updater logs:

    kubectl -n teleport logs deployment/teleport-agent-updater
    2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

GitOps tools

Managed updates for Kubernetes agents requires workarounds when used with GitOps tools for continuous deployment. The teleport-kube-agent Helm chart owns the version of the teleport-agent resource, so when the teleport-agent-updater modifies the image version of the teleport-agent resource, the GitOps tool will detect a drift or a diff in the teleport-agent resource.

The sections below describe workarounds for various GitOps tools.

ArgoCD deployments

After a managed update, ArgoCD reports the teleport-agent resource as OutOfSync. As a workaround to this problem use a Diff Customization to ignore the difference in image version. Here is an example deployment using the name teleport-agent and namespace teleport.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: teleport-agent
  namespace: teleport
spec:
  ignoreDifferences:
    - group: apps
      kind: StatefulSet
      name: teleport-agent
      namespace: teleport
      jqPathExpressions:
        - .spec.template.spec.containers[] | select(.name == "teleport").image
...

FluxCD deployments

After a managed update, FluxCD reports a DriftDetected event. As a workaround to this problem modify the drift detection configuration to ignore the difference in image version. Here is an example deployment using the name teleport-agent and namespace teleport.

apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: teleport-agent
  namespace: teleport
spec:
  driftDetection:
    mode: enabled
    ignore:
      - paths: [ "/spec/template/spec/containers/0/image" ]
        target:
          kind: StatefulSet
          name: teleport-agent
          namespace: teleport
...

Troubleshooting

You can inspect the current autoupdate status by running:

tctl autoupdate agents status

Agent autoupdate mode: enabledRollout creation date: 2025-02-24 16:01:44Start version: 17.2.0Target version: 17.2.1Rollout state: UnstartedStrategy: time-based
Group Name State Start Time State Reason---------- --------- ---------- --------------default Unstarted outside_window

This rollout state is computed by each Auth Service instance every minute. An autoupdate_config or autoupdate_version change might take up to a minute to be reflected and applied.

Teleport Agents are not updated immediately when a new version of Teleport is released, and agent updates can lag behind the cluster by a few days.

If the Teleport Agent has not been automatically updating for several weeks, you can consult the updater logs as described above to help troubleshoot the problem.

The teleport-update status command provides the best UX for determining how an agent is being instructed to update. However, if teleport-update is not available, this information can also be queried directly from the Teleport cluster using curl`:

curl -s https://teleport.example.com/webapi/find | jq .auto_update
{ ... "agent_version": "18.2.7", # version that the agent should update to (also used for new agents) "agent_auto_update": false, # true if in window and the agent should update now "agent_update_jitter_seconds": 60 # jitter to reduce load on the cluster and CDN}

This may be tuned for a specific agent using the group and update_id params:

curl -s https://teleport.example.com/webapi/find?group=staging&update_id=$(</tmp/teleport-update.id) | jq .auto_update
{ ... "agent_version": "18.2.8", "agent_auto_update": true, "agent_update_jitter_seconds": 60}

Troubleshooting managed agent upgrades on Kubernetes

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

  1. Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:

    kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'
  2. To suspend Managed Updates for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Troubleshooting managed agent upgrades on Linux

  1. You can query the updater status by running:

    teleport-update status
    proxy: teleport.example.com:443path: /usr/local/binbase_url: https://cdn.teleport.devenabled: truepinned: falseactive: version: 17.2.0 flags: [Enterprise]target: version: 17.2.1 flags: [Enterprise]in_window: falsejitter: 1m0s

    Here, the local active version is 17.2.0. The cluster's target version is 17.2.1, but we are not in an update window, so the agent is not immediately updated.

  2. If an agent is not automatically updated, you can invoke the updater manually and look at its logs:

    sudo teleport-update update --now

Using a different CDN URL

If your agents cannot reach the default Teleport CDN URL (cdn.teleport.dev), they will be unable to download updates.

Here are a couple of potential solutions to this issue:

Use an HTTP CONNECT proxy

If you configure the HTTPS_PROXY variable in the teleport-update process's environment, it will use this proxy to pull updates.

The easiest way to configure a proxy with a default install is to add this variable to /etc/systemd/system/teleport-update.service.d/override.conf:

$ sudo mkdir -p /etc/systemd/system/teleport-update.service.d
$ sudo tee -a /etc/systemd/system/teleport-update.service.d/override.conf > /dev/null <<'EOF'
[Service]
Environment=HTTPS_PROXY=http://proxy-url:3128
EOF

You can view the teleport-update process logs with sudo journalctl -u teleport-update.service.

Mirror the Teleport tarball packages and change the base-url

If you can mirror the Teleport tarball installers somewhere that your agents are able to access, you can change the base-url used by Teleport updaters so they can pull them directly.

To change the base-url, you should add the -b or --base-url flag to the teleport-update enable command:

$ sudo teleport-update enable --base-url https://teleport.artifactory.company.local

It is safe to re-run sudo teleport-update enable to modify the base URL. Existing updater settings will be preserved if not explicitly overridden by flags.

More information about flags that can be used with teleport-update enable can be found here