Fork me on GitHub

Teleport

Machine ID with Kubernetes Access

Improve

In this guide, we will demonstrate how to configure an automated service to use Machine ID to access a Kubernetes cluster protected by Teleport.

With Machine ID, Teleport issues short-lived certificates, tied to a machine identity, that can be rotated, audited, and managed with the same access controls that Teleport provides for human users.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 13.0.3.

    tctl version

    Teleport v13.0.3 go1.20

    tsh version

    Teleport v13.0.3 go1.20

    See Installation for details.

  • A running Teleport Enterprise cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The Enterprise tctl admin tool and tsh client tool version >= 13.0.3, which you can download by visiting your Teleport account.

    tctl version

    Teleport Enterprise v13.0.3 go1.20

    tsh version

    Teleport v13.0.3 go1.20

Cloud is not available for Teleport v.
Please use the latest version of Teleport Enterprise documentation.
Kubernetes Public Address Required for non-TLS Routing

For self-hosted Teleport Clusters that have non-TLS routing the Kubernetes public address must be set for Machine ID Kubernetes connections.

To confirm the TLS routing mode check the proxy.tls_routing_enabled from this command with your proxy address:

curl https://teleport.example.com:443/webapi/ping | jq

The optional tool jq is used here to help display the JSON output. If the value proxy.tls_routing_enabled is false then non-TLS routing is set and a Kubernetes public address is required so Machine ID will connect to the right port. You can confirm the Kubernetes public address is set if proxy.kube.public_addr is populated.

The kube_public_addr is set within the proxy_service by Teleport administrators:

proxy_service:
  enabled: true
  kube_listen_addr: 0.0.0.0:3026
  kube_public_addr: teleport.example.com:3026
  • If you're not already familiar with Machine ID, follow the Getting Started Guide to familiarize yourself with Machine ID. You'll also need tctl access to initially configure the bot.
  • Make sure you can connect to Teleport. Log in to your cluster using tsh, then use tctl remotely:
    tsh login --proxy=teleport.example.com [email protected]
    tctl status

    Cluster teleport.example.com

    Version 13.0.3

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    You can run subsequent tctl commands in this guide on your local machine.

    For full privileges, you can also run tctl commands on your Auth Service host.

  • Ensure the tbot binary is installed on your Machine ID client system. The client system is any system from which you want to access your Teleport cluster and the resources it protects from an automated service. Refer to our Installation guide for instructions on installing Teleport, which includes the necessary tbot binary.
  • Finally, to interact with the connected Kubernetes cluster, your client system will need to have kubectl installed. See the Kubernetes documentation for installation instructions.

Step 1/3. Create a Machine ID bot and assign permissions

In this example, you'll create a bot named "example" and assign it a role granting access to a Kubernetes cluster named "example".

First, create a file named role.yaml with the following content:

kind: role
metadata:
  name: machine-id-kube-role
version: v6
spec:
  allow:
    kubernetes_labels:
      '*': '*'
    kubernetes_resources:
      - kind: pod
        namespace: "*"
        name: "*"
    kubernetes_groups:
    - example
    kubernetes_users:
    - alice

Be sure to configure the allow fields to match your environment:

  • kubernetes_labels must be specified to allow access to a matching Kubernetes cluster in your Teleport environment.

  • Each item in kubernetes_resources must match the namespace and name of a Kubernetes resource that you would like users with machine-id-kube-role to access. Currently, Teleport only supports the pod kind. In the configuration above, this role can access all pods in all namespaces.

  • One (or both) of kubernetes_groups and kubernetes_users must be specified to so that the bot user may be mapped to a Kubernetes user and/or group.

    If only kubernetes_groups is set, kubernetes_users will be set to the bot's username by default.

    The user and group values entered here are relayed directly to the Kubernetes cluster as incoming user and group names. Kubernetes users and groups do not need to be created in advance, but do need to match one or more RoleBindings for the bot user to have any permissions on the target Kubernetes cluster.

Once finished, create the role in Teleport:

tctl create -f role.yaml

If you don't have an existing Kubernetes Role or RoleBinding to grant your Machine ID bot, create one as follows.

Kubernetes clusters often come with pre-made roles which may meet the needs of your application, such as view and edit. You may view these preexisting rules by running the following:

kubectl get clusterrole

If you'd prefer to create a role of your own, the following command can be used to create a rule with narrow access to certain resources types in the example-namespace namespace:

# This creates a role allowing `get` and `list` on pods and deployments.

kubectl create role example-role \ --namespace=example-namespace \ --verb=get,list \ --resource=pods,deployments

Next, create a role binding to connect the bot to your role. You may bind either the bot's mapped User or mapped Group to a role, however it must have been granted the matching User or Group via a Teleport role above:

# This binds the new role to the group `example`

kubectl create rolebinding example-rolebinding-group \ --namespace=demo \ --role=example-role \ --group=example

# Alternatively, this binds the new role to the user `alice`

kubectl create rolebinding example-rolebinding-user \ --namespace=demo \ --role=example-role \ --user=alice

# If binding to a ClusterRole like `view`, instead run the following:

kubectl create rolebinding example-rolebinding-user \ --namespace=demo \ --clusterrole=view \ --user=alice

You can grant cluster-wide access to your bot user by instead creating ClusterRole and ClusterRoleBinding resources, which are not namespaced and use an otherwise identical syntax.

Step 2/3. Configure and start Machine ID

On a node with tctl access, such as your local machine, create a new Machine ID bot using the Teleport role created in the previous step:

tctl bots add example --roles=machine-id-kube-role

The creates a bot named example with the necessary Kubernetes role. Be sure to note the bot join token and CA PIN.

Next, on the bot node, create a Machine ID configuration file at /etc/tbot.yaml:

auth_server: "teleport.example.com:443"
onboarding:
  join_method: "token"
  token: "abcd123-insecure-do-not-use-this"
  ca_pins:
  - "sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678"
storage:
  directory: /var/lib/teleport/bot
destinations:
  - directory: /opt/machine-id
    kubernetes_cluster: example-k8s-cluster

Be sure to configure the token and ca_pins fields to match the output from tctl bots add ..., and set kubernetes_cluster to match the cluster name as shown in tsh kube ls. For this example, we'll be connecting to the example-k8s-cluster cluster.

Machine ID also allows you to use Linux ACLs to control access to certificates on disk. You will use this to ensure only your application has access to the short-lived certificates Machine ID uses.

We'll work with the assumption you will be running Machine ID as the Linux user teleport and your automated service as the Linux user app. Create and initialize the destination directory by running this tbot init command either as root or as the teleport user:

tbot init \ -c /etc/tbot.yaml \ --init-dir=/opt/machine-id \ --bot-user=teleport \ --owner=teleport:teleport \ --reader-user=app

Create the bot data directory and grant permissions to access it to the Linux user (in our example, teleport) which tbot will run as.

Make the bot directory and assign ownership to teleport user

sudo mkdir -p /var/lib/teleport/bot
sudo chown teleport:teleport /var/lib/teleport/bot

Allow teleport user to open directory

sudo chmod +x /var/lib/teleport /var/lib/teleport/bot

Next, you will use systemd to run Machine ID in the background on your bot node. Create a systemd.unit file at /etc/systemd/system/machine-id.service:

[Unit]
Description=Teleport Machine ID Service
After=network.target

[Service]
Type=simple
User=teleport
Group=teleport
Restart=on-failure
Environment="TELEPORT_ANONYMOUS_TELEMETRY=1"
ExecStart=/usr/local/bin/tbot start -c /etc/tbot.yaml
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/machine-id.pid
LimitNOFILE=524288

[Install]
WantedBy=multi-user.target

Finally, run the following commands to start Machine ID:

sudo systemctl enable machine-id
sudo systemctl start machine-id
sudo systemctl status machine-id

Step 3/3. Connect to your Kubernetes cluster with the Machine ID identity

With Machine ID up and running, you should now have Kubernetes certificates written to /opt/machine-id. This can be verified in a few ways:

# A `kubeconfig.yaml` should exist in the destination directory:

ls /opt/machine-id/kubeconfig.yaml

/opt/machine-id/kubeconfig.yaml

# Additionally, the log should mention the Kubernetes certificates:

journalctl -u machine-id | grep -i kubernetes

Jul 13 20:46:42 example tbot[29177]: INFO [TBOT] Generated identity for Kubernetes cluster {"example-k8s-cluster"} tbot/renew.go:406

You can now use the generated kubeconfig.yaml to connect to the cluster with kubectl:

kubectl --kubeconfig /opt/machine-id/kubeconfig.yaml get pods -n demo

This kubeconfig.yaml can also be passed to any other Kubernetes API clients that support credential provider plugins, including those built with kubernetes/client-go and most other language libraries.

Next steps

More information about TELEPORT_ANONYMOUS_TELEMETRY.