Skip to main content
Version: 19.x (unreleased)

Deploying tbot on Kubernetes

Report an Issue

This guide shows you how to deploy the Machine & Workload Identity agent, tbot, on a Kubernetes cluster and use a static reference to the cluster's Service Account issuer keys for authentication.

How it works​

In the setup we demonstrate in this guide, tbot runs as a Kubernetes deployment. It writes output credentials to a Kubernetes secret, which can then be mounted in the pods that need to use the credentials. While tbot can also run as a sidecar within the same pod as the service that needs to use the credentials it generates, we recommend running tbot as a standalone deployment due to the limited support Kubernetes has for sidecars.

In this guide, we demonstrate the kubernetes join method, in which tbot proves its identity to the Teleport Auth Service by presenting a JSON web token (JWT) signed by the Kubernetes API server. This JWT contains the service account, the pod and the namespace in which tbot is running. The Teleport Auth Service checks the signature of the JWT against the Kubernetes cluster's public signing key.

OIDC Joining

Certain cloud providers like Amazon EKS regularly rotate their OIDC signing keys, which will cause the static_jwks configuration you create in this guide to become invalid after a short period of time.

On Kubernetes providers with OIDC support, like Amazon's Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS), consider using Kubernetes OIDC joining instead.

Using another join method

When deploying tbot to a Teleport cluster, it is generally recommended to use the kubernetes join method. This will work with most Kubernetes clusters. The guide that follows will demonstrate configuring this join method.

However, when using certain cloud Kubernetes services, it is possible to use the join method associated with that platform rather than the kubernetes join method. This may be beneficial if you wish to manage the joining of tbot within the Kubernetes clusters and on standard VMs on the same platform with a single join token. These services are:

Prerequisites​

  • A running Teleport cluster accessible at a hostname with a valid TLS certificate. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl and tsh clients.

    Installing tctl and tsh clients

    1. Determine the version of your Teleport cluster. The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/find and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:

      TELEPORT_DOMAIN=teleport.example.com:443
      TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"

    2. Follow the instructions for your platform to install tctl and tsh clients:

      Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

      curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg

      In Finder double-click the pkg file to begin installation.

      danger

      Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

    Connecting with TLS routing disabled

    This guide's commands assume your Teleport cluster uses TLS routing (proxy_listener_mode: multiplex), where the tctl and tsh clients reach every Teleport service through the Proxy Service's web address on port 443. If you're not sure whether this applies to your cluster, check with whoever manages it.

    If your cluster uses separate listener ports instead, adjust ports as follows:

    • tsh commands (e.g., tsh login --proxy=...): continue using the Proxy Service web address on port 3080 (or 443 if behind a load balancer). Do not change these to port 3025.

    • Direct tctl or Auth Service API commands: use port 3025 for the Auth Service gRPC listener:

      tctl status --auth-server=teleport.example.com:3025
  • Check that you can connect to your Teleport cluster and verify that you can run tctl and tsh commands using your current credentials.

    1. Assign teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and [email protected] to your Teleport username.

    2. Authenticate to your Teleport cluster. This depends on whether your shell is interactive or not.

      In an interactive shell: Run the following command. By default, this triggers a multi-factor authentication prompt:

      tsh login --proxy=teleport.example.com --user=[email protected]
      tctl status
      Cluster  teleport.example.com
      Version  19.0.0-dev
      CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

      On non-interactive environments: If you are running tsh and tctl as an AI agent, in a CI/CD environment, or similar, make sure the TELEPORT_IDENTITY_FILE environment variable is assigned to a valid file path with credentials for your cluster. tsh and tctl read the file path from the environment variable and do not require a separate authentication step. If there is no identity file available, we recommend that you set up Machine ID to provision one automatically.

      When executing tctl commands with an identity file, you must pass the --auth-server flag to provide the Teleport Auth Service address, which is not included in the identity file. If you provide the Proxy Service address, tctl connects to the Proxy Service, which forwards traffic to and from the Teleport Auth Service. Update 443 to 3025 if you are contacting the Auth Service directly with tctl:

      tctl status --auth-server=teleport.example.com:443

      For tsh commands that read an identity file, you must pass the --proxy flag, which points tsh to the address of the Teleport Proxy Service:

      tsh status --proxy=teleport.example.com

      Ensure client commands can access your identity file. Replace path/to/identity/file with the path to your identity file:

      export TELEPORT_IDENTITY_FILE="${TELEPORT_IDENTITY_FILE:-path/to/identity/file}"

      Add the --auth-server or --proxy flags to all subsequent tctl and tsh commands.

    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
  • A Kubernetes cluster with support for Token Request Projection (which graduated to a generally available feature in Kubernetes 1.20).
  • kubectl authenticated with the ability to create resources in the cluster you wish to deploy tbot into.
  • The helm CLI tool installed.

The examples in this guide will install a tbot deployment in the default Namespace of the Kubernetes cluster. Adjust references to default to the Namespace you wish to use.

Step 1/3. Create a Bot​

Next, you need to create a Bot. A Bot is a Teleport identity for a machine or group of machines. Like users, bots have a set of roles and traits which define what they can access.

Create bot.yaml:

kind: bot
version: v1
metadata:
  # name is a unique identifier for the Bot in the cluster.
  name: example
spec:
  # roles is a list of roles to grant to the Bot. Don't worry if you don't know
  # what roles you need to specify here, the Access Guides will walk you through
  # creating and assigning roles to the already created Bot.
  roles: []

Make sure you replace example with a unique, descriptive name for your Bot.

Use tctl to apply this file:

tctl create bot.yaml

Step 2/3. Create a join token​

Next, a join token needs to be configured. This will be used by tbot to join the cluster. As the kubernetes join method will be used, the public key of the Kubernetes cluster must first be determined. The public key used to sign JWTs is exposed on the "JWKS" endpoint of the Kubernetes API server. This public key can then be used by the Teleport Auth Service to verify that the Service Account JWT presented by tbot is signed legitimately by the Kubernetes cluster.

Run the following commands to determine the JWKS formatted public key:

kubectl get --raw /openid/v1/jwks
{"keys":[--snip--]}%

Create bot-token.yaml, ensuring you insert the value from the JWKS endpoint in spec.kubernetes.static_jwks.jwks:

kind: token
version: v2
metadata:
  # name will be specified in the `tbot` to use this token
  name: example-bot
spec:
  roles: [Bot]
  # bot_name should match the name of the bot created earlier in this guide.
  bot_name: example
  join_method: kubernetes
  kubernetes:
    # static_jwks configures the Auth Service to validate the JWT presented by
    # `tbot` using the public key from a statically configured JWKS.
    type: static_jwks
    static_jwks:
      jwks: |
        # Place the data returned by the curl command here
        {"keys":[--snip--]}
    # allow specifies the rules by which the Auth Service determines if `tbot`
    # should be allowed to join.
    allow:
    - service_account: "default:tbot" # service_account

Use tctl to apply this file:

tctl create -f bot-token.yaml

Step 3/3. Create a tbot deployment​

Now, you'll deploy tbot to your Kubernetes cluster using the Teleport tbot Helm chart. This will be configured using values provided to the Helm CLI tool.

First, create a file called tbot-values.yaml to hold the configuration values for the Helm chart:

# Replace the cluster name with the name of your Teleport cluster.
# This is not necessarily the public address of your Teleport Proxy Service.
clusterName: "example.teleport.sh"
# Replace this with the address of your Teleport Proxy Service.
teleportProxyAddress: "example.teleport.sh:443"
# Ensure this matches the name of the join token you created earlier.
token: "example-bot"
FIPS Compliance

The default tbot-distroless image does not contain the FIPS-compliant binaries. If you operate in an environment where FIPS compliance is required, additionally set the image: public.ecr.aws/gravitational/tbot-fips-distroless.

Before you can deploy the Helm chart, if you have not previously deployed a Teleport Helm chart, you'll need to add the Teleport chart repository to your CLI:

helm repo add teleport https://charts.releases.teleport.dev
helm repo update

You can now deploy the tbot Helm chart using the configuration you created earlier, ensuring you specify the namespace you wish to deploy tbot into:

helm install tbot teleport/tbot \  --namespace default \  --values tbot-values.yaml

Use kubectl to verify that the deployment is healthy:

kubectl describe deployment/tbot
kubectl logs deployment/tbot

With this complete, tbot is now successfully deployed to your cluster.

Next step: configure infrastructure access​

By default, the tbot Helm chart is configured to write an identity file to a Kubernetes Secret called tbot-out in the namespace where tbot has been deployed.

This identity file can be mounted into other pods and used with tsh or tctl to access and configure your Teleport cluster. For example:

apiVersion: v1
kind: Pod
metadata:
  name: tsh
  namespace: default
spec:
  containers:
    - name: tsh
      image: public.ecr.aws/gravitational/teleport-distroless:19.0.0-dev
      command:
        - tsh
      args:
       - -i
       - /identity-output/identity
       - --proxy
       - example.teleport.sh:443
       - ls
      volumeMounts:
        - name: identity-output
          mountPath: /identity-output
  volumes:
    - name: identity-output
      secret:
        secretName: tbot-out

If you wish to use tbot for a different kind of access, you can override the type of output using the services value of the Helm chart and setting defaultOutput.enabled to false.

Follow one of the access guides to find out more about how to configure tbot for your use case.

Requirements for deployments without Helm​

The Helm chart handles several requirements for tbot automatically. If you deploy tbot yourself — with your own Deployment, ServiceAccount, and tbot configuration — you must configure the following in addition to the standard tbot setup.

Token audience projection​

The Kubernetes join method requires that the service account JWT presented to Teleport has an audience matching the Teleport cluster name. The service account token mounted in a Pod by default is not issued with the Teleport cluster name as its audience.

You must project a service account token volume with the correct audience and tell tbot where to find it via the KUBERNETES_TOKEN_PATH environment variable:

spec:
  containers:
  - name: tbot
    env:
    - name: KUBERNETES_TOKEN_PATH
      value: "/var/run/secrets/teleport/token"
    volumeMounts:
    - name: teleport-token
      mountPath: /var/run/secrets/teleport
  volumes:
  - name: teleport-token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          audience: example.teleport.sh
          expirationSeconds: 600

Replace example.teleport.sh with the name of your Teleport cluster (the value of clusterName, not the proxy address).

Writing credentials to a Kubernetes secret​

When using destination.type: kubernetes_secret in the tbot configuration, the following additional setup is required:

  • Set the POD_NAMESPACE environment variable so tbot knows which namespace to write the secret to. Use the Kubernetes downward API:

    env:
- name: POD_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace

  • Grant the tbot service account permission to manage secrets in its namespace. The create verb is required because tbot creates the destination secret automatically if it does not already exist:

    apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tbot-secret-writer
  namespace: default
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tbot-secret-writer
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tbot-secret-writer
subjects:
- kind: ServiceAccount
  name: tbot
  namespace: default

    Replace default with the namespace tbot runs in.

Further reading​