Fork me on GitHub
Teleport

Migrating to Teleport Kubernetes Access 5.0

In release 5.0, Teleport has changed the Kubernetes integration to improve configuration and user experience. These changes are backward compatible and there is no required migration. However, to get the most out of Teleport, you should consider migrating as described below.

The main changes in 5.0 are:

  • New kubernetes_service configuration section in teleport.yaml, decoupled from proxy_service
  • Support for multiple Kubernetes clusters per Teleport cluster (replaces the need to use Trusted Clusters to achieve this)
  • Role-Based Access Control (RBAC) support for Kubernetes clusters

Important note: proxy_service with an open port for Kubernetes requests is still required. A proxy is always the public-facing gateway into the Teleport cluster, acting as an authentication point and a connection router.

Example scenarios

We'll describe the migration for several common Teleport configurations.

Standalone proxy for a single cluster

In this scenario, we're running a Teleport proxy outside of the Kubernetes cluster and connecting to it using a kubeconfig_file.

Before 5.0:

# teleport.yaml
proxy_service:
  kubernetes:
    enabled: yes
    listen_addr: 0.0.0.0:3026
    kubeconfig_file: /path/to/kubeconfig

After 5.0:

# teleport.yaml
proxy_service:
  kube_listen_addr: 0.0.0.0:3026

kubernetes_service:
  enabled: yes
  listen_addr: 0.0.0.0:3027
  kubeconfig_file: /path/to/kubeconfig

The listen_port for kubernetes_service only needs to be exposed to the Proxies within your cluster. This port will not be directly used by the clients.

Proxy in a Kubernetes pod of a single cluster

In this scenario, we're running a Teleport proxy inside of the Kubernetes cluster and connecting to it using the pod service account credentials.

Before 5.0:

# teleport.yaml
proxy_service:
  kubernetes:
    enabled: yes
    listen_addr: 0.0.0.0:3026

After 5.0:

# teleport.yaml
proxy_service:
  kube_listen_addr: 0.0.0.0:3026

kubernetes_service:
  enabled: yes
  listen_addr: 0.0.0.0:3027

Multiple Kubernetes clusters

In this scenario, we're running multiple Trusted Teleport clusters, one for each Kubernetes cluster.

Recommend Option 1: Teleport Kubernetes agents in each Kubernetes cluster

This is the recommended approach for handling multiple Kubernetes clusters.

Instead of running multiple Teleport clusters (auth and proxy for each Kubernetes cluster), we will connect multiple kubernetes_service agents to the same root Teleport cluster.

Each kubernetes_service will create its own reverse tunnel back to a Teleport proxy to join the cluster.

Before 5.0:

# teleport-root.yaml
auth_service:
  enabled: yes

proxy_service:
  kubernetes:
    enabled: yes
    public_addr: [root.example.com:3026]
    listen_addr: 0.0.0.0:3026

---
# teleport-leaf-1.yaml
auth_service:
  enabled: yes

proxy_service:
  kubernetes:
    enabled: yes
    public_addr: [leaf-1.example.com:3026]
    listen_addr: 0.0.0.0:3026
    # Not specifying kubeconfig_file, we'll assume that this process runs in a
    # Kubernetes pod

---
# teleport-leaf-2.yaml
auth_service:
  enabled: yes

proxy_service:
  kubernetes:
    enabled: yes
    public_addr: [leaf-2.example.com:3026]
    listen_addr: 0.0.0.0:3026
    # Not specifying kubeconfig_file, we'll assume that this process runs in a
    # Kubernetes pod

After 5.0:

# teleport-root.yaml
auth_service:
  enabled: yes
  tokens:
  # This is a static token, you can also use dynamic join tokens
  - kube:secret_static_join_token

proxy_service:
  kube_public_addr: [root.example.com]
  kube_listen_addr: 0.0.0.0:3026

---
# teleport-kube-1.yaml
teleport:
  # Note: auth_servers points at the default proxy listening port.
  auth_servers: ["https://root.example.com:3080"]
  auth_token: secret_static_join_token

auth_service:
  enabled: no

proxy_service:
  enabled: no

kubernetes_service:
  enabled: yes
  kube_cluster_name: kube-1
  # Not specifying kubeconfig_file, we'll assume that this process runs in a
  # Kubernetes pod

---
# teleport-kube-2.yaml
teleport:
  # Note: auth_servers points at the default proxy listening port.
  auth_servers: ["https://root.example.com:3080"]
  auth_token: secret_static_join_token

auth_service:
  enabled: no

proxy_service:
  enabled: no

kubernetes_service:
  enabled: yes
  kube_cluster_name: kube-2
  # Not specifying kubeconfig_file, we'll assume that this process runs in a
  # Kubernetes pod

After all the Teleport agents are running, you can view the list of Kubernetes clusters using tsh kube ls and switch between them using tsh kube login <kube_cluster_name>:

tsh kube ls

Kube Cluster Name Selected

----------------- --------

kube-1 *

kube-2

tsh kube login kube-2

Logged into kubernetes cluster "kube-2"

Option 2: One Teleport agent connected to multiple clusters

This approach requires you to manually craft a kubeconfig file that contains the exact set of Kubernetes cluster entries you wish to connect. Updates to the set of clusters require a restart.

Instead of running multiple Teleport clusters (auth and proxy for each Kubernetes cluster), we will connect multiple Kubernetes clusters to a single kubernetes_service using kubeconfig_file.

Before 5.0:

# teleport-root.yaml
auth_service:
  enabled: yes

proxy_service:
  kubernetes:
    enabled: yes
    public_addr: [root.example.com:3026]
    listen_addr: 0.0.0.0:3026

---
# teleport-leaf-1.yaml
auth_service:
  enabled: yes

proxy_service:
  kubernetes:
    enabled: yes
    public_addr: [leaf-1.example.com:3026]
    listen_addr: 0.0.0.0:3026
    kubeconfig_file: kubeconfig-leaf-1.yaml

---
# teleport-leaf-2.yaml
auth_service:
  enabled: yes

proxy_service:
  kubernetes:
    enabled: yes
    public_addr: [leaf-2.example.com:3026]
    listen_addr: 0.0.0.0:3026
    kubeconfig_file: kubeconfig-leaf-2.yaml

After 5.0:

# teleport.yaml
proxy_service:
  kube_public_addr: [root.example.com]
  kube_listen_addr: 0.0.0.0:3026

kubernetes_service:
  enabled: yes
  listen_addr: 0.0.0.0:3027
  kubeconfig_file: kubeconfig-all.yaml

---
# kubeconfig-all.yaml
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://10.0.0.1
  name: kube-1-server
- cluster:
    certificate-authority-data: ...
    server: https://10.0.0.2
  name: kube-2-server
contexts:
- context:
    cluster: kube-1-server
    user: kube-1-creds
  name: kube-1
- context:
    cluster: kube-2-server
    user: kube-2-creds
  name: kube-2
current-context: kube-1
users:
- name: kube-1-creds
  user:
    ...cluster-specific credentials...
- name: kube-2-creds
  user:
    ...cluster-specific credentials...

You can view the list of Kubernetes clusters using tsh kube ls and switch between them using tsh kube login <kube_cluster_name>:

tsh kube ls

Kube Cluster Name Selected

----------------- --------

kube-1 *

kube-2

tsh kube login kube-2

Option 3: Keep using Trusted Clusters

You may keep using the Trusted Clusters as before and migrate them internally to kubernetes_service, as described in the first two scenarios.

Note that tsh kube ls will not show all Kubernetes clusters across all Trusted Clusters. It only shows the Kubernetes clusters from the Teleport cluster you are currently logged into.

RBAC

Note
Kubernetes RBAC integration only works with kubernetes_service. Make sure you've migrated all the clusters as described above before proceeding.

By default, all users have access to all Kubernetes clusters (as a limited set of Kubernetes user/group identities). In 5.0, you can restrict user access to a subset of Kubernetes clusters using Kubernetes cluster labels and Teleport roles.

This RBAC integration works exactly like the integration for SSH nodes.

First, set the set of Kubernetes labels a user can access on their roles:

# role-admin.yaml
kind: role
version: v4
metadata:
  name: admin
spec:
  allow:
    logins: ['{{internal.logins}}']
    # Define the Kubernetes users and groups mapped to users with this role.
    kubernetes_users: ["{{internal.kubernetes_users}}"]
    kubernetes_groups: ["some-group", "{{internal.kubernetes_groups}}"]

    # List of kubernetes labels a user will be allowed to connect to:
    kubernetes_labels:
      # A user can only connect to clusters marked with 'test' label:
      'environment': 'test'
      # The wildcard ('*') means "any cluster"
      '*': '*'
      # Labels can be specified as a list:
      'environment': ['test', 'staging']
      # Regular expressions are also supported, for example the equivalent
      # of the list example above can be expressed as:
      'environment': '^test|staging$'
  # You can also strictly restrict access using deny kubernetes_labels.
  deny: {}

Next, define Kubernetes labels on the cluster side:

# teleport.yaml
kubernetes_service:
  enabled: yes
  listen_addr: 0.0.0.0:3027
  kube_cluster_name: staging
  # Note: if you use kubeconfig_file, the below labels apply to all Kubernetes
  # clusters in that kubeconfig. To get different sets of labels, you need to
  # run separate kubernetes_service instances.
  labels:
    environment: staging
  commands:
  - name: project
    command: [fetch-project-name.sh]
    period: 1h

Users with a role that matches a specific Kubernetes cluster label will be able to see and connect to that cluster. Other users will neither see nor be able to connect to it.

Have a suggestion or can’t find something?
IMPROVE THE DOCS