Deploying Machine ID on Kubernetes

This guide shows you how to deploy the Machine ID daemon tbot, on a Kubernetes cluster.

In the setup we demonstrate in this guide, tbot runs as a Kubernetes deployment. It writes output credentials to a Kubernetes secret, which can then be mounted in the pods that need to use the credentials. While tbot can also run as a sidecar within the same pod as the service that needs to use the credentials it generates, we recommend running tbot as a standalone deployment due the limited support Kubernetes has for sidecars.

In this guide, we demonstrate the kubernetes join method, in which tbot proves its identity to the Teleport Auth Service by presenting a JSON web token (JWT) signed by the Kubernetes API server. This JWT contains identifies the service account, the pod and the namespace in which tbot is running. The Teleport Auth Service checks the signature of the JWT against the Kubernetes cluster's public signing key.

Prerequisites

• A running Teleport cluster version 15.2.4 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

• The tctl admin tool and tsh client tool.

  On Teleport Enterprise, you must use the Enterprise version of tctl, which you can download from your Teleport account workspace. Otherwise, visit Installation for instructions on downloading tctl and tsh for Teleport Community Edition.

• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:

  tsh login --proxy=teleport.example.com --user=[email protected]
  tctl status
  Cluster  teleport.example.com
  Version  15.2.4
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
• A Kubernetes cluster with support for Token Request Projection (which graduated to a generally available feature in Kubernetes 1.20).
• kubectl authenticated with the ability to create resources in the cluster you wish to deploy tbot into.

The examples in this guide will install a tbot deployment in the default Namespace of the Kubernetes cluster. Adjust references to default to the Namespace you wish to use.

Step 1/4. Prepare Kubernetes RBAC

In order to prepare the Kubernetes cluster for Machine ID, several Kubernetes RBAC resources must be created.

A ServiceAccount will be created and later assigned to the Pod that will run tbot. This creates a static identity that we can allow access to join the Teleport Cluster and also provides an identity to which we can assign Kubernetes privileges.

A Role granting the ability to read and write to secrets in the Namespace will be created and then assigned to the ServiceAccount using a RoleBinding. This will allow the tbot Pod to read and write credentials to a Secret.

Create a file called k8s-rbac.yaml:

# This ServiceAccount will be used to give the `tbot` pods a discrete identity
# which can be validated by the Teleport Auth Server.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tbot
  namespace: default
---
# This role grants the ability to manage secrets within the namespace - this is
# necessary for the `kubernetes_secret` destination to work correctly.
#
# You may wish to add the `resourceNames` field to the role to further restrict
# this access in sensitive environments.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secrets-admin
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["*"]
---
# Bind the role to the service account created for tbot.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tbot-secrets-admin
  namespace: default
subjects:
  - kind: ServiceAccount
    name: tbot
roleRef:
  kind: Role
  name:  secrets-admin
  apiGroup: rbac.authorization.k8s.io

Apply this file to your Kubernetes cluster:

kubectl apply -f ./k8s-rbac.yaml

Step 2/4. Create a join token and bot user

Next, a join token needs to be configured. This will be used by tbot to join the cluster. As the kubernetes join method will be used, the public key of the Kubernetes cluster must first be determined. The public key used to sign JWTs is exposed on the "JWKS" endpoint of the Kubernetes API server. This public key can then be used by the Teleport Auth to verify that the Service Account JWT presented by tbot is signed legitimately by the Kubernetes cluster.

Run the following commands to determine the JWKS formatted public key:

kubectl proxy -p 8080
curl http://localhost:8080/openid/v1/jwks
{"keys":[--snip--]}%

Create bot-token.yaml, ensuring you insert the value from the JWKS endpoint in spec.kubernetes.static_jwks.jwks:

kind: token
version: v2
metadata:
  # name will be specified in the `tbot` to use this token
  name: example-bot
spec:
  roles: [Bot]
  # bot_name will match the name of the bot created later in this guide.
  bot_name: example
  join_method: kubernetes
  kubernetes:
    # static_jwks configures the Auth Server to validate the JWT presented by
    # `tbot` using the public key from a statically configured JWKS.
    type: static_jwks
    static_jwks:
      jwks: |
        # Place the data returned by the curl command here
        {"keys":[--snip--]}
    # allow specifies the rules by which the Auth Server determines if `tbot`
    # should be allowed to join.
    allow:
    - service_account: "default:tbot" # service_account

Use tctl to apply this file:

tctl create -f bot-token.yaml

Create the bot, specifying the token that you have created:

tctl bots add example --token example-bot

Step 3/4. Create a tbot deployment

First, a ConfigMap will be created to contain the configuration file for tbot. This will then be mounted into the Pod.

Create k8s-deployment-config.yaml, replacing the value of token with the name of the token you created earlier and the value of proxy_server with the address of your Teleport Proxy Service:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tbot-config
  namespace: default
data:
  tbot.yaml: |
    version: v2
    onboarding:
      join_method: kubernetes
      # ensure token is set to the name of the join token you created earlier
      token: bot-kubernetes
    storage:
      # a memory destination is used for the bots own state since the kubernetes
      # join method does not require persistence.
      type: memory
    # ensure this is configured to the address of your Teleport Proxy Service.
    proxy_server: example.teleport.sh:443
    # outputs will be filled in during the completion of an access guide.
    outputs: []

Apply this file to your Kubernetes cluster:

kubectl apply -f k8s-deployment-config.yaml

With the ConfigMap created, you can now create the tbot deployment itself.

Create k8s-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tbot
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: tbot
  template:
    metadata:
      labels:
        app.kubernetes.io/name: tbot
    spec:
      containers:
        - name: tbot
          image: public.ecr.aws/gravitational/tbot-distroless:15.2.4
          args:
            - start
            - -c
            - /config/tbot.yaml
          env:
            # POD_NAMESPACE is required for the kubernetes_secret` destination
            # type to work correctly.
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            # KUBERNETES_TOKEN_PATH specifies the path to the service account
            # JWT to use for joining.
            # This path is based on the configuration of the volume and
            # volumeMount.
            - name: KUBERNETES_TOKEN_PATH
              value: /var/run/secrets/tokens/join-sa-token
            # TELEPORT_ANONYMOUS_TELEMETRY enables the submission of anonymous
            # usage telemetry.  This helps us shape the future development of
            # `tbot`. You can disable this by omitting this.
            - name: TELEPORT_ANONYMOUS_TELEMETRY
              value: "1"
          volumeMounts:
            - mountPath: /config
              name: config
            - mountPath: /var/run/secrets/tokens
              name: join-sa-token
      serviceAccountName: tbot
      volumes:
        - name: config
          configMap:
            name: tbot-config
        - name: join-sa-token
          projected:
            sources:
              - serviceAccountToken:
                  path: join-sa-token
                  # 600 seconds is the minimum that Kubernetes supports. We
                  # recommend this value is used.
                  expirationSeconds: 600
                  # `example.teleport.sh` must be replaced with the name of
                  # your Teleport cluster.
                  audience: example.teleport.sh

Replace example.teleport.sh with the name of your Teleport cluster - this is not necessarily the public address (the port should not be included).

This is an example manifest, consider modifying it to fit within the conventions of deployments to your clusters (e.g customizing labels).

Apply this file to your Kubernetes cluster:

kubectl apply -f ./k8s-deployment.yaml

Use kubectl to verify that the deployment is healthy:

kubectl describe deployment/tbot
kubectl logs deployment/tbot

With this complete, tbot is now successfully deployed to your cluster. However, it is not yet producing any useful output.

Step 4/4. Configure outputs

Follow one of the access guides to configure an output that meets your access needs.

In order to adjust the access guides to work well with Kubernetes, use the Kubernetes Secret destination type. This will write the generated artifacts to a specified Kubernetes Secret, for example:

outputs:
  - type: identity
    destination:
      type: kubernetes_secret
      name: identity-output

The output can then be consumed by mounting this secret within another pod:

apiVersion: v1
kind: Pod
metadata:
  name: tsh
  namespace: default
spec:
  containers:
    - name: tsh
      image: public.ecr.aws/gravitational/teleport-distroless:15.2.4
      command:
        - tsh
      args:
       - -i
       - /identity-output/identity
       - --proxy
       - example.teleport.sh:443
       - ls
      volumeMounts:
        - name: identity-output
          mountPath: /identity-output
  volumes:
    - name: identity-output
      secret:
        secretName: identity-output

Next steps

• Follow the access guides to finish configuring tbot for your environment.
• Read the configuration reference to explore all the available configuration options.
• More information about TELEPORT_ANONYMOUS_TELEMETRY.