Machine ID with Kubernetes Access

In this guide, we will demonstrate how to configure an automated service to use Machine ID to access a Kubernetes cluster protected by Teleport.

With Machine ID, Teleport issues short-lived certificates, tied to a machine identity, that can be rotated, audited, and managed with the same access controls that Teleport provides for human users.

Prerequisites

• If you have not already connected your Kubernetes cluster to Teleport, follow the Kubernetes Access Getting Started Guide.

curl https://teleport.example.com:443/webapi/ping | jq

proxy_service:
  enabled: true
  kube_listen_addr: 0.0.0.0:3026
  kube_public_addr: teleport.example.com:3026

• If you're not already familiar with Machine ID, follow the Getting Started Guide to familiarize yourself with Machine ID. You'll also need tctl access to initially configure the bot.
• Make sure you can connect to Teleport. Log in to your cluster using tsh, then use tctl remotely:

  tsh login --proxy=teleport.example.com [email protected]
  tctl status
  Cluster  teleport.example.com
  Version  13.0.3
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  You can run subsequent tctl commands in this guide on your local machine.

  For full privileges, you can also run tctl commands on your Auth Service host.

• Ensure the tbot binary is installed on your Machine ID client system. The client system is any system from which you want to access your Teleport cluster and the resources it protects from an automated service. Refer to our Installation guide for instructions on installing Teleport, which includes the necessary tbot binary.
• Finally, to interact with the connected Kubernetes cluster, your client system will need to have kubectl installed. See the Kubernetes documentation for installation instructions.

Step 1/3. Create a Machine ID bot and assign permissions

In this example, you'll create a bot named "example" and assign it a role granting access to a Kubernetes cluster named "example".

First, create a file named role.yaml with the following content:

kind: role
metadata:
  name: machine-id-kube-role
version: v6
spec:
  allow:
    kubernetes_labels:
      '*': '*'
    kubernetes_resources:
      - kind: pod
        namespace: "*"
        name: "*"
    kubernetes_groups:
    - example
    kubernetes_users:
    - alice

Be sure to configure the allow fields to match your environment:

• kubernetes_labels must be specified to allow access to a matching Kubernetes cluster in your Teleport environment.

• Each item in kubernetes_resources must match the namespace and name of a Kubernetes resource that you would like users with machine-id-kube-role to access. Currently, Teleport only supports the pod kind. In the configuration above, this role can access all pods in all namespaces.

• One (or both) of kubernetes_groups and kubernetes_users must be specified to so that the bot user may be mapped to a Kubernetes user and/or group.

  If only kubernetes_groups is set, kubernetes_users will be set to the bot's username by default.

  The user and group values entered here are relayed directly to the Kubernetes cluster as incoming user and group names. Kubernetes users and groups do not need to be created in advance, but do need to match one or more RoleBindings for the bot user to have any permissions on the target Kubernetes cluster.

Once finished, create the role in Teleport:

tctl create -f role.yaml

kubectl get clusterrole

# This creates a role allowing `get` and `list` on pods and deployments.
kubectl create role example-role \  --namespace=example-namespace \  --verb=get,list \  --resource=pods,deployments

# This binds the new role to the group `example`
kubectl create rolebinding example-rolebinding-group \  --namespace=demo \  --role=example-role \  --group=example
# Alternatively, this binds the new role to the user `alice`
kubectl create rolebinding example-rolebinding-user \  --namespace=demo \  --role=example-role \  --user=alice
# If binding to a ClusterRole like `view`, instead run the following:
kubectl create rolebinding example-rolebinding-user \  --namespace=demo \  --clusterrole=view \  --user=alice

Step 2/3. Configure and start Machine ID

On a node with tctl access, such as your local machine, create a new Machine ID bot using the Teleport role created in the previous step:

tctl bots add example --roles=machine-id-kube-role

The creates a bot named example with the necessary Kubernetes role. Be sure to note the bot join token and CA PIN.

Next, on the bot node, create a Machine ID configuration file at /etc/tbot.yaml:

auth_server: "teleport.example.com:443"
onboarding:
  join_method: "token"
  token: "abcd123-insecure-do-not-use-this"
  ca_pins:
  - "sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678"
storage:
  directory: /var/lib/teleport/bot
destinations:
  - directory: /opt/machine-id
    kubernetes_cluster: example-k8s-cluster

Be sure to configure the token and ca_pins fields to match the output from tctl bots add ..., and set kubernetes_cluster to match the cluster name as shown in tsh kube ls. For this example, we'll be connecting to the example-k8s-cluster cluster.

Machine ID also allows you to use Linux ACLs to control access to certificates on disk. You will use this to ensure only your application has access to the short-lived certificates Machine ID uses.

We'll work with the assumption you will be running Machine ID as the Linux user teleport and your automated service as the Linux user app. Create and initialize the destination directory by running this tbot init command either as root or as the teleport user:

tbot init \    -c /etc/tbot.yaml \    --init-dir=/opt/machine-id \    --bot-user=teleport \    --owner=teleport:teleport \    --reader-user=app

Create the bot data directory and grant permissions to access it to the Linux user (in our example, teleport) which tbot will run as.

Make the bot directory and assign ownership to teleport user
sudo mkdir -p /var/lib/teleport/bot
sudo chown teleport:teleport /var/lib/teleport/bot
Allow teleport user to open directory
sudo chmod +x /var/lib/teleport /var/lib/teleport/bot

Next, you will use systemd to run Machine ID in the background on your bot node. Create a systemd.unit file at /etc/systemd/system/machine-id.service:

[Unit]
Description=Teleport Machine ID Service
After=network.target

[Service]
Type=simple
User=teleport
Group=teleport
Restart=on-failure
Environment="TELEPORT_ANONYMOUS_TELEMETRY=1"
ExecStart=/usr/local/bin/tbot start -c /etc/tbot.yaml
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/machine-id.pid
LimitNOFILE=524288

[Install]
WantedBy=multi-user.target

Finally, run the following commands to start Machine ID:

sudo systemctl enable machine-id
sudo systemctl start machine-id
sudo systemctl status machine-id

Step 3/3. Connect to your Kubernetes cluster with the Machine ID identity

With Machine ID up and running, you should now have Kubernetes certificates written to /opt/machine-id. This can be verified in a few ways:

# A `kubeconfig.yaml` should exist in the destination directory:
ls /opt/machine-id/kubeconfig.yaml
/opt/machine-id/kubeconfig.yaml
# Additionally, the log should mention the Kubernetes certificates:
journalctl -u machine-id | grep -i kubernetes
Jul 13 20:46:42 example tbot[29177]: INFO [TBOT]      Generated identity for Kubernetes cluster {"example-k8s-cluster"} tbot/renew.go:406

You can now use the generated kubeconfig.yaml to connect to the cluster with kubectl:

kubectl --kubeconfig /opt/machine-id/kubeconfig.yaml get pods -n demo

This kubeconfig.yaml can also be passed to any other Kubernetes API clients that support credential provider plugins, including those built with kubernetes/client-go and most other language libraries.

Next steps

More information about TELEPORT_ANONYMOUS_TELEMETRY.