Teleport
Upgrading Reference
- Version 17.x
- Version 16.x
- Version 15.x
- Version 14.x
- Older Versions
This guide explains how to upgrade components of a Teleport cluster in non-standard situations.
To ensure that your Teleport cluster remains up to date with the lowest amount of manual overhead, we recommend signing up for a cloud-hosted Teleport Enterprise account and following Enroll Agents in Automatic Upgrades.
If your infrastructure does not support automatic agent updates, follow this guide to determine the best approach for keeping your Teleport cluster up to date.
Before reading this guide, become familiar with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components in your cluster.
Working with the automatic updater
This section explains how to manage the automatic updater.
On Kubernetes deployments, the updater is a controller that periodically
reconciles expected Kubernetes resources with those in the cluster. On a Linux
server, the updater is an executable script called teleport-upgrade
.
teleport-upgrade
commands
The teleport-upgrade
tool provides some basic commands to verify and perform an
update of the Teleport agent.
teleport-upgrade helpUSAGE: /usr/sbin/teleport-upgrade <command>
Tool for automatic upgrades of Teleport agents.
Commands: run check for and potentially apply a teleport upgrade. dry-run check for new teleport version but do not upgrade. force performs an upgrade if an upgrade is available. version print the current version of /usr/sbin/teleport-upgrade. help show this help text.
The dry-run
command can be used to check for an available update without performing
an update.
Example output when teleport is already on the latest compatible version.
teleport-upgrade dry-run[i] no upgrades available (14.3.14 == 14.3.14) [ 582 ]Example output when an update is available.
teleport-upgrade dry-run[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] within maintenance window, upgrade will be attempted. [ 596 ]
The run
command performs an update if available.
Successful teleport update from 13.4.14 to 14.3.14.
teleport-upgrade run[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] within maintenance window, upgrade will be attempted. [ 596 ][i] attempting apt install teleport-ent=14.3.14... [ 480 ][...][i] gracefully restarting Teleport (if already running) [ 449 ]Teleport updates are not attempted when outside the maintenance window.
teleport-upgrade run[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ]
The force
command performs an update immediately even when outside the maintenance
window.
teleport-upgrade force[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ][i] attempting apt install teleport-ent=14.3.14... [ 480 ][...][i] gracefully restarting Teleport (if already running) [ 449 ]
Configuring the teleport-upgrade
tool
-
Create the upgrade configuration directory:
sudo mkdir -p /etc/teleport-upgrade.d/ -
If you changed the agent user to run as non-root, create
/etc/teleport-upgrade.d/schedule
and grant ownership to your Teleport user. Otherwise, you can skip this step:sudo touch /etc/teleport-upgrade.d/schedulesudo chown your-teleport-user /etc/teleport-upgrade.d/schedule -
Configure the upgrader to connect to your version server and subscribe to the right release channel:
echo "teleport.example.com:443/v1/webapi/automaticupgrades/channel/default" | sudo tee /etc/teleport-upgrade.d/endpointMake sure not to include
https://
as a prefix to the server address, nor suffix the endpoint with/version
.
Choosing a release channel
When configuring the updater, you can select a release channel.
The following channels are available for APT, YUM, and Zypper repos:
Channel name | Description |
---|---|
stable/<major> | Receives releases for the specified major release line, i.e. v16 |
stable/cloud | Rolling channel that receives releases compatible with current Cloud version |
stable/rolling | Rolling channel that receives all published Teleport releases |
Updating the updater
The updater is designed to be minimal and relatively stable, but the updater will
receive updates on occasion. Currently, the updater does not have the capability
to update itself. Updates to the teleport-ent-updater
package must be done manually.
The teleport-ent-updater
will be backwards compatible with older versions of Teleport,
so you can always update the teleport-ent-updater
package to the latest available
version.
Version locking
As of Teleport 15.1.10
, a version locking mechanism is enabled by the updater.
This mechanism locks the version of Teleport and prevents manual updates of the Teleport
package. This prevents unintentional updates during routine system maintenance, or
an accidental update by a user. The updater still has the capability to update the
Teleport package, and all updates of Teleport are expected to be performed by the
updater.
The version locking mechanism is implemented using the features of the package managers. In case a user would like to manually update the Teleport version, this can be done with the following commands.
With the apt
package manager CLI, the --allow-change-held-packages
flag must be provided
to bypass the version lock.
apt-get install --allow-change-held-packages "teleport-ent=<target-version>"
With the yum
package manager CLI, the --disableexcludes="teleport"
flag must be provided
to bypass the version lock.
yum install --disablerepo="*" --enablerepo="teleport" --disableexcludes="teleport" "teleport-ent-<target-version>"
With the zypper
package manager CLI, the lock must be disabled and then re-enabled
after the update.
zypper removelock "teleport-ent"zypper install --repo="teleport" "teleport-ent-<target-version>"zypper addlock "teleport-ent"
Automatic update limitations
Automatic updates are not available in all Teleport editions and installation methods. If you cannot use automatic updates, read Manual updates for possible steps.
Automatic updates with Teleport Community Edition
Automatic updates is not currently supported with the community editions of Teleport.
Ensure that you are using the Teleport Enterprise edition of the teleport-kube-agent
chart. You should see the following when you query your teleport-kube-agent
release:
helm -n teleport get values teleport-agent -o json | jq '.enterprise'true
Automatic updates with direct binary installation
Automatic updates is not currently supported with the direct binary installation method.
Automatic updates is only supported with installations via the apt
, yum
, and
zypper
package managers.
Automatic updates with GitOps tools
Automatic updates are incompatible with some GitOps tools used for continuous
deployment. The teleport-kube-agent
Helm chart owns the version of the
teleport-agent
resource, so when the teleport-agent-updater
modifies the
image version of the teleport-agent
resource, the GitOps tool will detect a
drift or a diff in the teleport-agent
resource.
ArgoCD deployments
After an automatic update, ArgoCD reports the teleport-agent
resource as OutOfSync
.
As a workaround to this problem use a Diff Customization
to ignore the difference in image version. Here is an example deployment using the
name teleport-agent
and namespace teleport
.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: teleport-agent
namespace: teleport
spec
ignoreDifferences:
- group: apps
kind: StatefulSet
name: teleport-agent
namespace: teleport
jqPathExpressions:
- .spec.template.spec.containers[] | select(.name == "teleport").image
...
FluxCD deployments
After an automatic update, FluxCD reports a DriftDetected
event. As a workaround
to this problem modify the drift detection
configuration to ignore the difference in image version. Here is an example deployment
using the name teleport-agent
and namespace teleport
.
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
name: teleport-agent
namespace: teleport
spec
driftDetection:
mode: enabled
ignore:
- paths: ["/spec/template/spec/containers/0/image"]
target:
kind: StatefulSet
name: teleport-agent
namespace: teleport
...
Manual updates
This section shows you how to upgrade Teleport manually. You can perform manual upgrades on Teleport Auth Service and Proxy Service instances running in self-hosted clusters, as well as all Teleport agents.
Teleport agents
-
Identify the latest compatible Teleport agent version by querying the
webapi
endpoint of the Teleport Proxy Service:curl https://teleport.example.com/webapi/automaticupgrades/channel/stable/cloud/versionv15.2.1 -
Use the
tctl inventory ls
command to list connected agents along with their current version. Use the--older-than
flag to list agents that are upgradable:tctl inventory ls --older-than=v15.2.1Server ID Hostname Services Version Upgrader------------------------------------ -------------- -------------- ------- --------00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node v14.4.5 none00000000-0000-0000-0000-000000000001 teleport-proxy Proxy v15.2.0 none00000000-0000-0000-0000-000000000002 teleport-auth Auth,Discovery v15.2.0 none... -
For each agent ID returned by the
tctl inventory ls
command, copy the ID and run the followingtctl
command to access the host viatsh
:HOST=00000000-0000-0000-0000-000000000000USER=roottsh ssh "${USER?}@${HOST?}" -
On each Linux server, follow the instructions in the next section to install the new version of the
teleport
binary. -
If you have deployed any agents on Kubernetes using the
teleport-kube-agent
Helm chart, follow the instructions to upgrade the Helm release.
Single Teleport binaries on Linux servers
You can upgrade a single Teleport binary running on a Linux host by running the one-line installation script with a higher version than the current one.
Before upgrading Teleport across a self-hosted cluster, read the Compatibility Overview to ensure you are upgrading components in the correct order.
Complete the following steps on all servers that run the Auth Service and Proxy Service, then on each of your agents:
-
Get the current version:
teleport version -
Assign edition to one of the following, depending on your Teleport edition:
Edition Value Teleport Enterprise (Cloud) cloud
Teleport Enterprise (Self-Hosted) enterprise
Teleport Community Edition oss
-
Assign version to the version you want to install.
-
Install the new Teleport version on your Linux server:
curl https://cdn.teleport.dev/install-v16.4.2.sh | bash -s version editionThe installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the installation guide.
-
Confirm that the version of the
teleport
binary is the one you expect:teleport version -
Now that you have installed a more recent
teleport
binary on your Auth Service and Proxy Service instances, restart Teleport on these servers to run the new version.Configure your Teleport instance to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed your Teleport instance.
On the host where you will run your Teleport instance, enable and start Teleport:
sudo systemctl enable teleportsudo systemctl start teleportOn the host where you will run your Teleport instance, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:
sudo teleport install systemd -o /etc/systemd/system/teleport.servicesudo systemctl enable teleportsudo systemctl start teleportYou can check the status of your Teleport instance with
systemctl status teleport
and view its logs withjournalctl -fu teleport
.
Self-hosted Teleport clusters on Kubernetes
The instructions in this section assume that you have configured the
teleport-cluster
Helm chart with a values file called values.yaml
, and that
your teleport-cluster
release is called teleport-cluster
.
-
Shrink the Auth Service pool. You must reduce the number of Auth Service instances to one in order to ensure a consistent cluster state during the upgrade.
Ensure that your
teleport-cluster
values file includes the following configuration:auth: highAvailability: replicaCount: 1
-
Update the Teleport Helm chart repository so you can install the latest version of the
teleport-cluster
chart:Set up the Teleport Helm repository.
Allow Helm to install charts that are hosted in the Teleport Helm repository:
helm repo add teleport https://charts.releases.teleport.devUpdate the cache of charts from the remote repository so you can upgrade to all available releases:
helm repo update -
Upgrade the
teleport-cluster
Helm release:helm upgrade teleport-cluster teleport/teleport-cluster \ --version=16.4.2 \ --values=values.yamlThe
teleport-cluster
Helm chart automatically waits for the previous version of the Proxy Service to stop responding to requests before running a new version of the Auth Service. -
Once you have completed this guide and upgraded the cluster, you can configure your cluster for high availability again.
Teleport agents running on Kubernetes
The instructions in this section assume that you have configured the
teleport-kube-agent
Helm chart with a values file called values.yaml
, and
that your teleport-kube-agent
release is called teleport-agent
.
-
Update the Teleport Helm chart repository so you can install the latest version of the
teleport-kube-agent
chart:Set up the Teleport Helm repository.
Allow Helm to install charts that are hosted in the Teleport Helm repository:
helm repo add teleport https://charts.releases.teleport.devUpdate the cache of charts from the remote repository so you can upgrade to all available releases:
helm repo update -
Upgrade the Helm release:
helm -n teleport upgrade teleport-agent teleport/teleport-kube-agent \ --values=values.yaml \ --version=16.4.2