Meet us at KubeCon + CloudNativeCon: Paris, France - March 19
Book Demo
Teleport logoTry For Free
Fork me on GitHub

Teleport

Set up Automatic Agent Upgrades

  • Available for:
  • Enterprise

Teleport supports automatic agent upgrades for systemd-based Linux distributions using apt, yum, and zypper package managers, as well as Kubernetes clusters.

Teleport agents run an upgrader that queries a version server to determine whether they are out of date. This guide describes how to set up your infrastructure to support automatic updates. If you are a Teleport Cloud user or run a version server already, return to the Upgrading menu for the appropriate next steps to upgrade Teleport.

The Automatic Update Architecture guide explains how automatic agent upgrades work in more detail.

Warning

Systemd agents enrolled into automatic updates can only install versions present in their package repositories. As Teleport 14 won't be published to stable/v13, those agents will require manual intervention to be upgraded to the next major version (adding a new APT/YUM/zypper repo for stable/v14). Alternatively, you can use the stable/rolling channel, which contains Teleport v13.3.2 forward, including future major releases.

Prerequisites

  • Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components of your cluster.
  • Self-hosted Teleport cluster v13.0 or higher.
  • The tctl and tsh client tools version >= 15.1.1.
    tctl version

    Teleport v15.1.1 go1.21


    tsh version

    Teleport v15.1.1 go1.21

  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status

    Cluster teleport.example.com

    Version 15.1.1

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

The version server must be hosted on a webserver with trusted TLS certificates and reachable by all agents. You must have either:

  • A public Amazon S3 or Google Cloud Storage bucket.
  • A web server accessible from all agents with valid TLS certificates.

If you do not have an S3 or GCS bucket for your version server, this guide provides an example of how to create one using Terraform. You can install Terraform to spin up a demo version server along with this guide so you can get started with automatic agent upgrades. Terraform is not required to set up a version server.

  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status

    Cluster teleport.example.com

    Version 15.1.1

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/5. Create release channel files

A release channel contains two pieces of information: the targeted version and if the upgrade is critical. Updaters subscribe to a release channel and will upgrade to the provided version during a maintenance window (which we will configure later in this guide). If the upgrade is critical, updaters will ignore the maintenance schedule and upgrade as soon as possible.

The version server is a static file server that responds to the following queries:

curl https://<hosting-domain-and-path>/current/version
15.1.1
curl https://<hosting-domain-and-path>/current/critical
no
  1. Create a project directory for the files we create in this guide:

    mkdir version-server
    cd version-server
  2. Create a directory for the new release channel current.

    mkdir current/
  3. Make the current release channel target the version 15.1.1:

    echo -n "15.1.1" > current/version
  4. And mark the upgrade as not critical:

    echo -n "no" > current/critical

Step 2/5. Create a Terraform configuration

  1. In the project directory you created in the previous section, create a file called main.tf and populate it with the following content, depending on whether you will use Amazon S3 or Google Cloud Storage:

    In the configuration below, replace REGION with the name of the AWS region where you will deploy the version server:

    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5.0"
        }
      }
    }
    
    provider "aws" {
      region = "REGION"
    }
    
    resource "aws_s3_bucket" "version_server" {
      // Replace the bucket name to ensure it is unique
      bucket = "version-server"
    }
    
    resource "aws_s3_object" "version" {
      bucket = aws_s3_bucket.version_server.id
      key = "current/version"
      source = "${path.root}/current/version"
    }
    
    resource "aws_s3_object" "critical" {
      bucket = aws_s3_bucket.version_server.id
      key = "current/critical"
      source = "${path.root}/current/critical"
    }
    
    resource "aws_s3_bucket_policy" "version_server" {
      depends_on = [aws_s3_bucket_public_access_block.version_server]
      bucket = aws_s3_bucket.version_server.id
      policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Sid       = "GrantAnonymousReadPermissions"
            Principal = "*"
            Action = [
              "s3:GetObject",
            ]
            Effect   = "Allow"
            Resource = ["${aws_s3_bucket.version_server.arn}/*"]
          },
        ]
      })
    }
    
    resource "aws_s3_bucket_public_access_block" "version_server" {
      bucket = aws_s3_bucket.version_server.id
    
      block_public_acls       = false
      block_public_policy     = false
      ignore_public_acls      = false
      restrict_public_buckets = false
    }
    

    In the configuration below, replace PROJECT and REGION with the Google Cloud project and region where you will deploy the version server:

    terraform {
      required_providers {
        google = {
          source  = "hashicorp/google"
          version = "~> 5.0"
        }
      }
    }
    
    provider "google" {
      project     = "PROJECT"
      region      = "REGION"
    }
    
    resource "google_storage_bucket" "version_server" {
      // Change the location as needed. See:
      // https://cloud.google.com/storage/docs/locations
      location = "US-EAST1"
      // Replace this to ensure it is unique
      name = "version-server"
    }
    
    resource "google_storage_bucket_object" "version" {
      name   = "current/version"
      source = "${path.root}/current/version"
      bucket = google_storage_bucket.version_server.name
    }
    
    resource "google_storage_bucket_object" "critical" {
      name   = "current/critical"
      source = "${path.root}/current/critical"
      bucket = google_storage_bucket.version_server.name
    }
    
    data "google_iam_policy" "viewer" {
      binding {
        role = "roles/storage.objectViewer"
        members = [
            "allUsers",
        ] 
      }
    }
    
    resource "google_storage_bucket_iam_policy" "version_server" {
      bucket = google_storage_bucket.version_server.name
      policy_data = data.google_iam_policy.viewer.policy_data
    }
    
  2. Make your cloud provider credentials available to Terraform. The method to use depends on your organization, e.g., pasting environment variables into your terminal.

  3. Apply the configuration:

terraform init
terraform apply

Step 3/5. Configure the maintenance schedule

At this point the updaters can be configured to pull the version from the release channel and upgrade the agents. In this step you'll configure a maintenance schedule for the Teleport cluster that agents will use to determine when to check for upgrades.

Create a Teleport role

Create a Teleport role that can manage cluster maintenance configurations through the cluster_maintenance_config dynamic resource. No preset Teleport roles provide this ability, so you will need to create one.

  1. Create a file called cmc-editor.yaml with the following content:

    kind: role
    version: v7
    metadata:
      name: cmc-editor
    spec:
      allow:
        rules:
        - resources: ['cluster_maintenance_config']
          verbs: ['create', 'read', 'update', 'delete']
    
  2. Create the role resource:

    tctl create cmd-editor.yaml
  3. Add the role to your Teleport user:

Assign the myrole role to your Teleport user by running the appropriate commands for your authentication provider:

  1. Retrieve your local user's roles as a comma-separated list:

    ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')
  2. Edit your local user to add the new role:

    tctl users update $(tsh status -f json | jq -r '.active.username') \ --set-roles "${ROLES?},myrole"
  3. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your github authentication connector:

    tctl get github/github --with-secrets > github.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the github.yaml file. Because this key contains a sensitive value, you should remove the github.yaml file immediately after updating the resource.

  2. Edit github.yaml, adding myrole to the teams_to_roles section.

    The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.

    Here is an example:

      teams_to_roles:
        - organization: octocats
          team: admins
          roles:
            - access
    +       - myrole
    
  3. Apply your changes:

    tctl create -f github.yaml
  4. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your saml configuration resource:

    tctl get --with-secrets saml/mysaml > saml.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the saml.yaml file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource.

  2. Edit saml.yaml, adding myrole to the attributes_to_roles section.

    The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

    Here is an example:

      attributes_to_roles:
        - name: "groups"
          value: "my-group"
          roles:
            - access
    +       - myrole
    
  3. Apply your changes:

    tctl create -f saml.yaml
  4. Sign out of the Teleport cluster and sign in again to assume the new role.

  1. Retrieve your oidc configuration resource:

    tctl get oidc/myoidc --with-secrets > oidc.yaml

    Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the oidc.yaml file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource.

  2. Edit oidc.yaml, adding myrole to the claims_to_roles section.

    The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

    Here is an example:

      claims_to_roles:
        - name: "groups"
          value: "my-group"
          roles:
            - access
    +       - myrole
    
  3. Apply your changes:

    tctl create -f oidc.yaml
  4. Sign out of the Teleport cluster and sign in again to assume the new role.

Create a cluster maintenance configuration

Create the following cmc.yaml manifest allowing maintenance on Monday, Wednesday and Friday between 02:00 and 03:00 UTC.

kind: cluster_maintenance_config
spec:
  agent_upgrades:
    # Maintenance window start hour in UTC.
    # The maintenance window lasts 1 hour.
    utc_start_hour: 2
    # Week days when maintenance is allowed
    # Possible values are:
    # - Short names: Sun, Mon, Tue, Wed, Thu, Fri, Sat
    # - Long names: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday
    weekdays:
      - Mon
      - Wed
      - Fri

Finally, apply the manifest using tctl:

tctl create cmc.yaml
maintenance window has been updated

Step 4/5. Enroll Kubernetes agents in automatic updates

Now that you have deployed a version server, you can enroll agents in automatic upgrades. This guide begins with agents deployed on Kubernetes. If all of your agents run on Linux servers, you can skip to Step 5.

Install the agent upgrader Helm chart

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

  1. Confirm you are using the Teleport Enterprise edition of the teleport-kube-agent chart. You should see the following when you query your teleport-kube-agent release:

    helm -n teleport get values -n teleport-agent -o json | jq '.enterprise'
    true
  2. Add the following chart values to the values file for the teleport-kube-agent chart:

    updater:
      enabled: true
      versionServer: https://version.example.com
      releaseChannel: release-channel
    

    Change release-channel to the name of the release channel you would like the agent to query for upgrades. For the current version, choose current.

  3. Update the Helm chart release with the new values:

    helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version=15.1.1

Verify that the upgrader is working properly

  1. You can validate the updater is running properly by checking if its pod is ready:

    kubectl -n teleport-agent get pods
    NAME READY STATUS RESTARTS AGE<your-agent-release>-0 1/1 Running 0 14m<your-agent-release>-1 1/1 Running 0 14m<your-agent-release>-2 1/1 Running 0 14m<your-agent-release>-updater-d9f97f5dd-v57g9 1/1 Running 0 16m
  2. Check for any deployment issues by checking the updater logs:

    kubectl -n teleport logs deployment/teleport-agent-updater
    2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

Troubleshooting automatic agent upgrades on Kubernetes

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

  1. Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:

    kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'
  2. To suspend automatic updates for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Step 5/5. Enroll Linux agents in automatic updates

This section shows you how to enroll Teleport agents running on Linux virtual or bare-metal machines into automatic updates.

Follow these instructions on each of your Teleport agents.

Install the agent upgrader

  1. Ensure the Teleport repository is added and Teleport Enterprise is installed.

    To verify if the Teleport repository was added to the system, check if either of the follow files exist:

    ls /etc/apt/sources.list.d/teleport.list

    or

    ls /etc/yum.repos.d/teleport.repo

    The upgrader checks the repository for available releases, so make sure that it is up to date.

  2. If the repository was added, make sure the Teleport binary installed on the agent can run the automatic updater:

    which teleport-upgrade || echo "Install the upgrader"
    Install the upgrader
  3. If the Teleport repository is not found, or the Teleport binary you installed does not include the upgrader, add the appropriate repository and reinstall Teleport.

    Download Teleport's PGP public key

    sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc

    Source variables about OS version

    source /etc/os-release

    Add the Teleport APT repository for v15. You'll need to update this

    file for each major release of Teleport.

    echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v15" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

    sudo apt-get update
    sudo apt-get install teleport-ent-updater

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo apt-get install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport YUM repository for v15. You'll need to update this

    file for each major release of Teleport.

    First, get the major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")
    sudo yum install -y yum-utils
    sudo yum-config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v15/teleport.repo")"
    sudo yum install teleport-ent-updater

    Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

    echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo yum install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport Zypper repository for v15. You'll need to update this

    file for each major release of Teleport.

    First, get the OS major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

    Use zypper to add the teleport RPM repo

    sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/cloud/teleport-zypper.repo")
    sudo yum install teleport-ent-updater

    Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

    echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo yum install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport YUM repository for v15. You'll need to update this

    file for each major release of Teleport.

    First, get the major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

    Use the dnf config manager plugin to add the teleport RPM repo

    sudo dnf config-manager --add-repo "$(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v15/teleport.repo")"

    Install teleport

    sudo dnf install teleport-ent-updater

    Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

    echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo dnf install teleport-ent-fips

    Source variables about OS version

    source /etc/os-release

    Add the Teleport Zypper repository.

    First, get the OS major version from $VERSION_ID so this fetches the correct

    package version.

    VERSION_ID=$(echo $VERSION_ID | grep -Eo "^[0-9]+")

    Use Zypper to add the teleport RPM repo

    sudo zypper addrepo --refresh --repo $(rpm --eval "https://zypper.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v15/teleport-zypper.repo")

    Install teleport

    sudo zypper install teleport-ent-updater

    For FedRAMP/FIPS-compliant installations, install the teleport-ent-fips package instead:

    sudo zypper install teleport-ent-fips

    In the example commands below, update $SYSTEM_ARCH with the appropriate value (amd64, arm64, or arm). All example commands using this variable will update after one is filled out.

    curl https://cdn.teleport.dev/teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-bin.tar.gz.sha256

    <checksum> <filename>

    curl -O https://cdn.teleport.dev/teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-bin.tar.gz
    shasum -a 256 teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-bin.tar.gz

    Verify that the checksums match

    tar -xvf teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-bin.tar.gz
    cd teleport-ent
    sudo ./install

    For FedRAMP/FIPS-compliant installations of Teleport Enterprise, package URLs will be slightly different:

    curl https://cdn.teleport.dev/teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-fips-bin.tar.gz.sha256

    <checksum> <filename>

    curl -O https://cdn.teleport.dev/teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-fips-bin.tar.gz
    shasum -a 256 teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-fips-bin.tar.gz

    Verify that the checksums match

    tar -xvf teleport-ent-v15.1.1-linux-$SYSTEM_ARCH-fips-bin.tar.gz
    cd teleport-ent
    sudo ./install

Configure the upgrader

  1. Create the upgrade configuration directory:

    sudo mkdir -p /etc/teleport-upgrade.d/
  2. If you changed the agent user to run as non-root, create /etc/teleport-upgrade.d/schedule and grant ownership to your Teleport user. Otherwise, you can skip this step:

    sudo touch /etc/teleport-upgrade.d/schedule
    sudo chown your-teleport-user /etc/teleport-upgrade.d/schedule
  3. Configure the upgrader to connect to your custom version server and subscribe to the right release channel:

    echo version-server-url/path/release-channel | sudo tee /etc/teleport-upgrade.d/endpoint

    Make sure not to include https:// as a prefix to the server address.

Verify that the upgrader is working properly

  1. Verify that the upgrader can see your version endpoint by checking for upgrades:

    sudo teleport-upgrade dry-run
  2. You should see one of the following messages, depending on the target version you are currently serving:

    no upgrades available (1.2.3 == 1.2.3)
    an upgrade is available (1.2.3 -> 2.3.4)
    

    teleport-upgrade may display warnings about not having a valid upgrade schedule. This is expected immediately after install as the maintenance schedule might not be exported yet.

Troubleshooting automatic agent upgrades on Linux

  1. If an agent is not automatically upgraded, you can invoke the upgrader manually and look at its logs:

    sudo teleport-upgrade run
  2. To suspend automatic updates, disable the systemd timer:

    sudo systemctl disable --now teleport-upgrade.timer
  3. To enable and start the systemd timer after suspending:

    sudo systemctl enable --now teleport-upgrade.timer

Next steps

While this guide showed you how to apply a cluster_maintenance_config resource using tctl, we recommend using infrastructure as code to maintain your Teleport resources. See the teleport_cluster_maintenance_config for how to declare a cluster maintenance configuration with Terraform.