Scaling Privileged Access for Modern Infrastructure: Real-World Insights
Apr 25
Virtual
Register Today
Teleport logoTry For Free
Fork me on GitHub

Teleport

Set up Automatic Agent Upgrades

Teleport supports automatic agent upgrades for systemd-based Linux distributions using apt, yum, and zypper package managers, as well as Kubernetes clusters.

Teleport agents run an upgrader that queries a version server to determine whether they are out of date. This guide describes how to set up your infrastructure to support automatic updates. If you are a Teleport Cloud user or run a version server already, return to the Upgrading menu for the appropriate next steps to upgrade Teleport.

The Automatic Update Architecture guide explains how automatic agent upgrades work in more detail.

Warning

Systemd agents enrolled into automatic updates can only install versions present in their package repositories. As Teleport 14 won't be published to stable/v13, those agents will require manual intervention to be upgraded to the next major version (adding a new APT/YUM/zypper repo for stable/v14). Alternatively, you can use the stable/rolling channel, which contains Teleport v13.3.2 forward, including future major releases.

Prerequisites

  • Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components of your cluster.
  • Self-hosted Teleport cluster running at least v14.3.7 or v15.1.3.
  • The tctl and tsh client tools version >= 15.2.2. Read Installation for how to install these.
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status

    Cluster teleport.example.com

    Version 15.2.2

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/4. Check the version advertised by the Teleport Proxy Service

The Teleport Proxy service exposes by default a version channel called default that serves the same version the Proxy Service is running.

If your proxy public address is teleport.example.com:443, you can query the version server with the following command:

curl "https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version"
15.2.2

By default, the version server has a single default channel, serving the version of the Teleport Proxy Service. If you want to override the default version or add other channels you can use the automatic_upgrades_channels field in the Proxy Service configuration file:

proxy_service:
  enabled: "yes"
  automatic_upgrades_channels:
    # Override the default version channel reachable at
    # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version
    default:
      static_version: v14.2.1
    # Define a new version channel with a static version reachable at
    # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/m-static-channel/version
    my-static-channel:
      static_version: v14.2.0
    # Define a new version channel forwarding requests to an upstream version server
    my-remote-channel:
      forward_url: https://updates.releases.teleport.dev/v1/stable/cloud

You must ensure all proxies share the same automatic_upgrades_channels configuration. If Proxy Service instances are configured differently, you will experience agents flickering between versions as the version served is not consistent across instances.

Step 2/4. Configure the maintenance schedule

At this point, the updaters can be configured to pull the version from the release channel and upgrade the agents. In this step you'll configure a maintenance schedule for the Teleport cluster that agents will use to determine when to check for upgrades.

Create a Teleport role

Create a Teleport role that can manage cluster maintenance configurations through the cluster_maintenance_config dynamic resource. No preset Teleport roles provide this ability, so you will need to create one.

  1. Create a file called cmc-editor.yaml with the following content:

    kind: role
    version: v7
    metadata:
      name: cmc-editor
    spec:
      allow:
        rules:
        - resources: ['cluster_maintenance_config']
          verbs: ['create', 'read', 'update', 'delete']
    
  2. Create the role resource:

    tctl create cmd-editor.yaml
  3. Add the role to your Teleport user:

Assign the myrole role to your Teleport user by running the appropriate commands for your authentication provider:

  1. Retrieve your local user's roles as a comma-separated list:

    ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')
  2. Edit your local user to add the new role:

    tctl users update $(tsh status -f json | jq -r '.active.username') \ --set-roles "${ROLES?},myrole"
  3. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your github authentication connector:

    tctl get github/github --with-secrets > github.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the github.yaml file. Because this key contains a sensitive value, you should remove the github.yaml file immediately after updating the resource.

  2. Edit github.yaml, adding myrole to the teams_to_roles section.

    The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.

    Here is an example:

      teams_to_roles:
        - organization: octocats
          team: admins
          roles:
            - access
    +       - myrole
    
  3. Apply your changes:

    tctl create -f github.yaml
  4. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your saml configuration resource:

    tctl get --with-secrets saml/mysaml > saml.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the saml.yaml file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource.

  2. Edit saml.yaml, adding myrole to the attributes_to_roles section.

    The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

    Here is an example:

      attributes_to_roles:
        - name: "groups"
          value: "my-group"
          roles:
            - access
    +       - myrole
    
  3. Apply your changes:

    tctl create -f saml.yaml
  4. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your oidc configuration resource:

    tctl get oidc/myoidc --with-secrets > oidc.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the oidc.yaml file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource.

  2. Edit oidc.yaml, adding myrole to the claims_to_roles section.

    The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

    Here is an example:

      claims_to_roles:
        - name: "groups"
          value: "my-group"
          roles:
            - access
    +       - myrole
    
  3. Apply your changes:

    tctl create -f oidc.yaml
  4. Sign out of the Teleport cluster and sign in again to assume the new role.

Create a cluster maintenance configuration

Create the following cmc.yaml manifest allowing maintenance on Monday, Wednesday and Friday between 02:00 and 03:00 UTC.

kind: cluster_maintenance_config
spec:
  agent_upgrades:
    # Maintenance window start hour in UTC.
    # The maintenance window lasts 1 hour.
    utc_start_hour: 2
    # Week days when maintenance is allowed
    # Possible values are:
    # - Short names: Sun, Mon, Tue, Wed, Thu, Fri, Sat
    # - Long names: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday
    weekdays:
      - Mon
      - Wed
      - Fri

Finally, apply the manifest using tctl:

tctl create cmc.yaml
maintenance window has been updated

Step 3/4. Enroll Kubernetes agents in automatic updates

Now that you have deployed a version server, you can enroll agents in automatic upgrades. This guide begins with agents deployed on Kubernetes. If all of your agents run on Linux servers, you can skip to Step 4.

Install the agent upgrader Helm chart

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

  1. Add the following chart values to the values file for the teleport-kube-agent chart:

    updater:
      enabled: true
      # No need to set the `versionServer` value anymore since v13.4.15, v14.3.1 and v15.
      releaseChannel: default
    
  2. Update the Helm chart release with the new values:

    helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version=15.2.2

Verify that the upgrader is working properly

  1. You can validate the updater is running properly by checking if its pod is ready:

    kubectl -n teleport-agent get pods
    NAME READY STATUS RESTARTS AGE<your-agent-release>-0 1/1 Running 0 14m<your-agent-release>-1 1/1 Running 0 14m<your-agent-release>-2 1/1 Running 0 14m<your-agent-release>-updater-d9f97f5dd-v57g9 1/1 Running 0 16m
  2. Check for any deployment issues by checking the updater logs:

    kubectl -n teleport logs deployment/teleport-agent-updater
    2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

Troubleshooting automatic agent upgrades on Kubernetes

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

  1. Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:

    kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'
  2. To suspend automatic updates for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Step 4/4. Enroll Linux agents in automatic updates

This section shows you how to enroll Teleport agents running on Linux virtual or bare-metal machines into automatic updates.

Follow these instructions on each of your Teleport agents.

Install the agent upgrader

  1. Ensure the Teleport repository is added and Teleport Enterprise is installed.

    To verify if the Teleport repository was added to the system, check if either of the follow files exist:

    ls /etc/apt/sources.list.d/teleport.list

    or

    ls /etc/yum.repos.d/teleport.repo

    The upgrader checks the repository for available releases, so make sure that it is up to date.

  2. If the repository was added, make sure the Teleport binary installed on the agent can run the automatic updater:

    which teleport-upgrade || echo "Install the upgrader"
    Install the upgrader
  3. If the Teleport repository is not found, or the Teleport binary you installed does not include the upgrader, add the appropriate repository and reinstall Teleport.

    Download Teleport's PGP public key

    sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc

    Source variables about OS version

    source /etc/os-release

    Add the Teleport APT repository for v15. You'll need to update this

    file for each major release of Teleport.

    echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v15" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

    sudo apt-get update
    sudo apt-get install teleport-ent-updater

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo apt-get install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport YUM repository for v15. You'll need to update this

    file for each major release of Teleport.

    First, get the major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
    sudo yum install -y yum-utils
    sudo yum-config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v15/teleport.repo")"
    sudo yum install teleport-ent-updater

    Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

    echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo yum install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport Zypper repository for v15. You'll need to update this

    file for each major release of Teleport.

    First, get the OS major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

    Use zypper to add the teleport RPM repo

    sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/cloud/teleport-zypper.repo")
    sudo yum install teleport-ent-updater

    Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

    echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo yum install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport YUM repository for v15. You'll need to update this

    file for each major release of Teleport.

    First, get the major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

    Use the dnf config manager plugin to add the teleport RPM repo

    sudo dnf config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v15/teleport.repo")"

    Install teleport

    sudo dnf install teleport-ent-updater

    Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

    echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo dnf install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport Zypper repository.

    First, get the OS major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

    Use Zypper to add the teleport RPM repo

    sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v15/teleport-zypper.repo")

    Install teleport

    sudo zypper install teleport-ent-updater

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo zypper install teleport-ent-fips

    In the example commands below, update $SYSTEM_ARCH with the appropriate value (amd64, arm64, or arm). All example commands using this variable will update after one is filled out.

    curl https://cdn.teleport.dev/teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-bin.tar.gz.sha256

    <checksum> <filename>

    curl -O https://cdn.teleport.dev/teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-bin.tar.gz
    shasum -a 256 teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-bin.tar.gz

    Verify that the checksums match

    tar -xvf teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-bin.tar.gz
    cd teleport-ent
    sudo ./install

    For FedRAMP/FIPS-compliant installations of Teleport Enterprise, package URLs will be slightly different:

    curl https://cdn.teleport.dev/teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-fips-bin.tar.gz.sha256

    <checksum> <filename>

    curl -O https://cdn.teleport.dev/teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-fips-bin.tar.gz
    shasum -a 256 teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-fips-bin.tar.gz

    Verify that the checksums match

    tar -xvf teleport-ent-v15.2.2-linux-$SYSTEM_ARCH-fips-bin.tar.gz
    cd teleport-ent
    sudo ./install

Configure the upgrader

  1. Create the upgrade configuration directory:

    sudo mkdir -p /etc/teleport-upgrade.d/
  2. If you changed the agent user to run as non-root, create /etc/teleport-upgrade.d/schedule and grant ownership to your Teleport user. Otherwise, you can skip this step:

    sudo touch /etc/teleport-upgrade.d/schedule
    sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule
  3. Configure the upgrader to connect to your custom version server and subscribe to the right release channel:

    echo "teleport.example.com:443/v1/webapi/automaticupgrades/channel/default" | sudo tee /etc/teleport-upgrade.d/endpoint

    Make sure not to include https:// as a prefix to the server address, nor suffix the endpoint with /version.

Verify that the upgrader is working properly

  1. Verify that the upgrader can see your version endpoint by checking for upgrades:

    sudo teleport-upgrade dry-run
  2. You should see one of the following messages, depending on the target version you are currently serving:

    no upgrades available (1.2.3 == 1.2.3)
    an upgrade is available (1.2.3 -> 2.3.4)
    

    teleport-upgrade may display warnings about not having a valid upgrade schedule. This is expected immediately after install as the maintenance schedule might not be exported yet.

Troubleshooting automatic agent upgrades on Linux

  1. If an agent is not automatically upgraded, you can invoke the upgrader manually and look at its logs:

    sudo teleport-upgrade run
  2. To suspend automatic updates, disable the systemd timer:

    sudo systemctl disable --now teleport-upgrade.timer
  3. To enable and start the systemd timer after suspending:

    sudo systemctl enable --now teleport-upgrade.timer

Next steps

While this guide showed you how to apply a cluster_maintenance_config resource using tctl, we recommend using infrastructure as code to maintain your Teleport resources. See the teleport_cluster_maintenance_config for how to declare a cluster maintenance configuration with Terraform.