Skip to main content
Version: 19.x (unreleased)

Machine ID with Argo CD

Report an issue with this page

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It runs in Kubernetes and can deploy applications to the same cluster or to other "external" clusters.

In this guide, you will configure the Machine ID agent, tbot, to manage Argo CD's cluster credentials, enabling it to securely deploy applications using Teleport.

How it works​

Argo CD supports declaratively managing clusters using Kubernetes secrets (see the Argo CD documentation. The Argo CD application controller reads these secrets in order to retrieve cluster credentials. When configured to use the Argo CD output, tbot writes Teleport-signed Kubernetes cluster credentials into secrets that Argo CD can then use to access Teleport-protected Kubernetes clusters.

Prerequisite​

  • A running Teleport cluster. If you do not have one, read Getting Started.

  • The tctl and tsh clients.

    Installing tctl and tsh clients

    1. Determine the version of your Teleport cluster. The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/find and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:

      TELEPORT_DOMAIN=teleport.example.com:443
      TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"

    2. Follow the instructions for your platform to install tctl and tsh clients:

      Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

      curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg

      In Finder double-click the pkg file to begin installation.

      danger

      Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

  • Argo CD installed in a Kubernetes cluster. This cluster will be referred to as the source cluster throughout this guide, it does not need to be enrolled into Teleport.
  • The Kubernetes cluster to which you'd like Argo CD to deploy applications. This cluster will be referred to as the target cluster throughout this guide, it must be enrolled into Teleport, if you have not already done this, follow the Enroll a Kubernetes Cluster guide.
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and [email protected] to your Teleport username: 
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status
    Cluster  teleport.example.com
    Version  19.0.0-dev
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
  • To configure Kubernetes and deploy tbot you will need kubectl and helm installed.
  • To check clusters have been registered with Argo CD, you will need the argocd CLI installed.

Step 1/4. Configure Teleport and Kubernetes RBAC​

First, we need to configure the RBAC for both Teleport and Kubernetes in order to grant the Machine ID agent the correct level of access.

When forwarding requests to the Kubernetes API on behalf of a bot, the Teleport Proxy attaches the groups configured (using kubernetes_groups) in the bot's Teleport roles to the request. These groups are then used to configure a RoleBinding or ClusterRoleBinding in Kubernetes to grant specific permissions within the Kubernetes cluster to the bot.

For the purpose of this guide, we will bind the editor group to the default edit ClusterRole that is preconfigured in most Kubernetes clusters to give the bot read and write access to resources in all the cluster namespaces.

When configuring this in a production environment, you should consider:

  • If RoleBinding should be used instead of ClusterRoleBinding to limit the bot's access to a specific namespace.
  • If a Role should be created that grants the bot the least privileges necessary rather than using a pre-existing general Role such as edit.

To bind the editor group to the edit Cluster Role, run the following command against both the source and target clusters:

kubectl create clusterrolebinding teleport-editor-edit \  --clusterrole=edit \  --group=editor

With the appropriate RoleBinding configured in Kubernetes to grant access to a specific group, you now need to add this group to the role that the bot will impersonate when producing credentials. You also need to grant the bot access through Teleport to the cluster itself. This is done by creating a role that grants the necessary permissions and then assigning this role to the bot.

Create a file called role.yaml with the following content:

kind: role
version: v7
metadata:
  name: example-role
spec:
  allow:
    kubernetes_labels:
      '*': '*'
    kubernetes_groups:
    - editor
    kubernetes_resources:
    - kind: "*"
      namespace: "*"
      name: "*"
      verbs: ["*"]

Replace example-role with a descriptive name related to your use case.

Adjust the allow field for your environment:

  • kubernetes_labels should be adjusted to grant access to only the clusters that the bot will need to access. The value shown, '*': '*' will grant access to all Kubernetes clusters.
  • editor must match the name of the group you specified in the RoleBinding or ClusterRoleBinding.
  • kubernetes_resources can be used to apply additional restrictions to what the bot can access within the Kubernetes cluster. These restrictions are layered upon the RBAC configured within the Kubernetes role itself.

Use tctl create -f ./role.yaml to create the role.

tip

You can also create and edit roles using the Web UI. Go to Access -> Roles and click Create New Role or pick an existing role to edit.

Step 2/4. Create a bot​

Next, we need to create the bot. A bot is a Teleport identity for a machine or group of machines.

Create a file called bot.yaml with the following content:

kind: bot
version: v1
metadata:
  # name uniquely identifies the bot within Teleport
  name: example-bot
spec:
  # roles that will be granted to the bot.
  roles: [example-role]

Make sure you replace example-bot with a unique, descriptive name for your Bot, and replace example-role with the name of the role you created in the previous step.

Use tctl create -f ./bot.yaml to create the bot.

Step 3/4. Create a join token​

In order for tbot to be able to authenticate and join the Teleport cluster, we need to configure a join token. There are a number of different available methods, but for this guide we will use the kubernetes method with a static JWKS. Please refer to the Deploying tbot on Kubernetes guide for more in-depth information.

OIDC Joining

Certain cloud providers like Amazon EKS regularly rotate their OIDC signing keys, which will cause the static_jwks configuration you create in this guide to become invalid after a short period of time.

On Kubernetes providers with OIDC support, like Amazon's Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS), consider using Kubernetes OIDC joining instead.

First, run the following command against the source cluster, to determine the JWKS-formatted public key:

kubectl get --raw /openid/v1/jwks
{"keys":[--snip--]}%

Next, create a file called join-token.yaml with the following content, ensuring you insert the output from the curl command in spec.kubernetes.static_jwks.jwks and specify the namespace in which Argo CD is running in spec.kubernetes.allow[0].service_account:

kind: token
version: v2
metadata:
  # name will be specified in the tbot Helm chart values later
  name: example-join-token
spec:
  roles: [Bot]
  # bot_name must match the name of the bot created earlier in this guide.
  bot_name: example-bot
  join_method: kubernetes
  kubernetes:
    # static_jwks configures the Auth Service to validate the JWT presented by
    # `tbot` using the public key from a statically configured JWKS.
    type: static_jwks
    static_jwks:
      jwks: |
        # Place the data returned by the curl command here
        {"keys":[--snip--]}
    # allow specifies the rules by which the Auth Service determines if `tbot`
    # should be allowed to join.
    allow:
    - service_account: "argocd:tbot" # namespace:service_account

Use tctl create -f ./join-token.yaml to create the join.

Step 4/4. Deploy tbot​

Finally, we'll use the official Helm chart to deploy tbot.

Find your cluster name by running tctl status.

Create a file called tbot-values.yaml with the following content, ensuring you replace the placeholder values with your cluster name, proxy address, and using clusterSelectors to identify which Kubernetes clusters you'd like to expose to Argo CD:

clusterName: "test.teleport.sh"
teleportProxyAddress: "test.teleport.sh:443"
token: "example-join-token"
defaultOutput:
  enabled: false
argocd:
  enabled: true
  clusterSelectors:
    - labels:
        environment: production

Please refer to the Helm chart reference for all of the supported configuration options.

Install the Helm chart by running the following commands against the source cluster, ensuring you specify the namespace in which Argo CD is running:

helm repo add teleport https://charts.releases.teleport.dev
helm repo update
helm install tbot teleport/tbot \  --namespace argocd \  --values tbot-values.yaml

Once the Helm chart installation is complete (usually after a few minutes), you should see your Kubernetes clusters in Argo CD. You can check this by running the following command:

argocd cluster list
SERVER                                                                    NAMEhttps://test.teleport.sh:443/v1/teleport/dHAxLmZsb3BweS5jbw/Ym94b2ZyYWQ1  test.teleport.sh-prod-eu-1https://kubernetes.default.svc                                            in-cluster

You should also be able to see the tbot-managed cluster secrets in Kubernetes when running the following command:

kubectl get secrets -n argocd | grep ^teleport.argocd-cluster.
teleport.argocd-cluster.e815df7b7588be17   Opaque               3      7d3h

You can now configure Argo CD to deploy applications to your Teleport-enrolled clusters!

If new matching clusters are added in Teleport, tbot will register them with Argo CD on the bot's next certificate renewal. If needed, the tbot process can be restarted by deleting the pod or signalled (pkill -USR1 tbot) to trigger an immediate reload.

Please note that, for safety, if a cluster is deleted in Teleport or no longer matches your clusterSelectors, it will not be automatically removed from Argo CD - but tbot will stop refreshing its credentials.

Argo CD Impersonation​

Argo CD supports using Kubernetes' user impersonation feature to give the application sync process more restrictive privileges than the Argo control plane has generally. This can be used to create a permission boundary where applications deployed to the same cluster, but in different projects, cannot read or modify each other's resources.

In order to use Argo CD impersonation with Teleport, your bot must have a role where kubernetes_users contains a wildcard. For example:

kind: role
version: v7
metadata:
  name: kube-wildcard-access
spec:
  allow:
    kubernetes_users:
    - '*'

If you simply enumerate the users that Argo CD is allowed to impersonate in kubernetes_users, you will encounter the following error:

please select a user to impersonate, refusing to select a user due to several kubernetes_users set up for this user

This happens because Argo CD only sets impersonation headers on requests that operate on application-scoped resources. Because your role has permission to impersonate many users, it's ambiguous which one Teleport should use by default.

When kubernetes_users contains a wildcard, and no specific user is being impersonated, the Teleport Kubernetes Service will fall back to sending your bot's Teleport username by default. You should therefore create a RoleBinding or ClusterRoleBinding, in the target cluster, granting your bot user the broader permissions needed by the Argo CD control plane.

kubectl create clusterrolebinding example-bot-edit \  --clusterrole=edit \  --user=bot-example-bot
warning

When using impersonation to create a permission boundary between Argo CD projects, remember that the Teleport Kubernetes Service will automatically impersonate any groups listed in kubernetes_groups by default.

You should ensure that your bot user does not have any roles which contain kubernetes_groups, otherwise the privilege isolation provided by impersonating different users is compromised, because the application sync process will have the combination of the impersonated user's privileges and those granted to the groups.

Next steps​