Fork me on GitHub

Teleport

Teleport AKS Auto-Discovery (Preview)

Improve

AKS Auto-Discovery can automatically discover any AKS cluster and enroll it in Teleport if its tags match the configured labels.

The discovery process requires two separate Teleport components. The first, the Discovery Service, is responsible for watching the cloud provider and checking if there are any new clusters or if there have been any modifications to previously discovered clusters. The second, the Kubernetes Service, monitors the clusters created by the Discovery Service and is responsible for authenticating and forwarding requests into them.

This guide presents the Discovery and Kubernetes services running in the same process, however both can run independently and on different machines. Running the processes independently is beneficial if access to different clusters requires Kubernetes Service to be on the same network as the cluster.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 11.0.3.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 11.0.3, which you can download by visiting the customer portal.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.3.8. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.3.8 go1.19

    tsh version

    Teleport v10.3.8 go1.19

  • An Azure identity with permissions to create and attach AD Groups.
  • One or more AKS clusters running.
  • Access to AKS clusters.
  • A host to run the Teleport Discovery and Kubernetes services.

Step 1/2. Set up Azure Identity with the required permissions

Depending on each cluster's authentication and authorization settings, Azure uses a different way to configure the necessary permissions for Teleport to forward requests to the server.

Check the authentication modes used on your clusters and choose one or more permissions scenarios. In some configurations, Teleport has the ability to automatically configure the access to the cluster if you include the necessary permissions to do so.

In this scenario, the Teleport's authentication happens through Active Directory, and the permissions required to access the Kubernetes cluster are associated with the roles assigned to its identity.

This mode allows you to grant permissions to multiple Kubernetes clusters without requiring specific settings for each one of them.

To grant access to the AKS clusters running with this setting, create an AD role with the following content and assign it to the identity that the Teleport process will use.

  {
      "Name": "AKS Teleport Discovery",
      "Description": "Required permissions for Teleport auto-discovery.",
      "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action"
      ],
      "NotActions": [],
      "DataActions": [
        "Microsoft.ContainerService/managedClusters/groups/impersonate/action",
        "Microsoft.ContainerService/managedClusters/users/impersonate/action",
        "Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
        "Microsoft.ContainerService/managedClusters/pods/read",
        "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
        "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
      ],
      "NotDataActions": [],
      "assignableScopes": [
          "/subscriptions/{subscription_id}"
      ]
  }

Replace the {subscription_id} with the desired Subscription ID or a wildcard if you want to guarantee permissions on all subscriptions.

When using Azure AD authentication with Kubernetes RBAC mode, Azure is responsible for user authentication using AD credentials, but permissions management is Kubernetes' RBAC responsibility.

Therefore, for Teleport to work correctly, you must create the Kubernetes ClusterRole and ClusterRoleBinding resources on each discovered cluster. The ClusterRoleBinding must bind the ClusterRole to one of the AD groups configured in the Teleport identity.

Teleport can automatically create the ClusterRole and ClusterRoleBinding resources in the following cases:

  • Teleport's AD identity has permissions that allow access to the static cluster administrator credentials (local accounts).
  • Teleport's AD identity belongs to the cluster's administrator group.
  • Teleport's AD identity has permissions to create ClusterRole and ClusterRoleBinding on the cluster and permissions to execute remote commands.

In either of the specified cases, Teleport will be able to create the ClusterRole and bind it to the first AD group it belongs. To make this possible, associate the following permissions with Teleport's identity.

  {
      "Name": "AKS Teleport Discovery",
      "Description": "Required permissions for Teleport auto-discovery.",
      "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
        "Microsoft.ContainerService/managedClusters/runcommand/action",
        "Microsoft.ContainerService/managedclusters/commandResults/read"
      ],
      "NotActions": [],
      "DataActions": [],
      "NotDataActions": [],
      "assignableScopes": [
          "/subscriptions/{subscription_id}"
      ]
  }

In all other cases, you must manually set up the required access as described in the guide below.

Connect to every cluster to enroll with admin permissions and create the following resources using kubectl.

ClusterRole

Create the ClusterRole RBAC definition with the required permissions for Teleport Kubernetes Service to forward requests to the cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: teleport
rules:
- apiGroups:
  - ""
  resources:
  - users
  - groups
  - serviceaccounts
  verbs:
  - impersonate
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - "authorization.k8s.io"
  resources:
  - selfsubjectaccessreviews
  - selfsubjectrulesreviews
  verbs:
  - create

ClusterRoleBinding

Link the previously created ClusterRole into the Teleport AD group id.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: teleport
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teleport
subjects:
- kind: Group
  name: {azure_ad_group_id}
  apiGroup: rbac.authorization.k8s.io

Replace {azure_ad_group_id} with the appropriate Azure AD group id associated with Teleport identity.

In this case, Teleport will use the user credentials generated during the provisioning phase of the cluster.

  {
      "Name": "AKS Teleport Discovery",
      "Description": "Required permissions for Teleport auto-discovery.",
      "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
      ],
      "NotActions": [],
      "DataActions": [],
      "NotDataActions": [],
      "assignableScopes": [
          "/subscriptions/{subscription_id}"
      ]
  }

Step 2/2. Configure Teleport to discover AKS clusters

Teleport AKS Auto-Discovery requires a valid auth token for the Discovery and Kubernetes services to join the cluster. Generate one by running the following command against your Teleport Auth Service and save it in /tmp/token on the machine that will run Kubernetes Auto-Discovery:

tctl tokens add --type=discovery,kube

Enabling AKS Auto-Discovery requires that the discovery_service.azure section include at least one entry and that discovery_service.azure.types include aks. It also requires configuring the kubernetes_service.resources.tags to use the same labels configured at discovery_service.azure.tags or a subset of them to make the Kubernetes Service listen to the dynamic resources created by the Discovery Service.

version: v2
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  auth_servers:
  - "teleport.example.com:3080"
auth_service:
  enabled: off
proxy_service:
  enabled: off
ssh_service:
  enabled: off
discovery_service:
  enabled: "yes"
  azure:
  - types: ["aks"]
    regions: ["*"]
    subscriptions: ["*"]
    resource_groups: ["*"]
    tags:
      "env": "prod"
kubernetes_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod" # Match Kubernetes Cluster labels specified earlier

Once you have added this configuration, start Teleport. AKS clusters matching the tags and regions specified in the Azure section will be added to the Teleport cluster automatically.

Troubleshooting

Discovery Service

To check if the Discovery Service is working correctly, you can check if any Kubernetes clusters have been discovered. To do this, you can use the tctl get kube_cluster command and inspect if the expected clusters have already been imported into Teleport.

If some clusters do not appear in the list, check if the filtering labels match the missing cluster tags or look into the service logs for permission errors.

Kubernetes Service

If the tctl get kube_cluster command returns the discovered clusters, but the tsh kube ls command does not include them, check that you have set the kubernetes_service.resources section correctly.

kubernetes_service:
  enabled: `yes`
  resources:
  - tags:
      "env": "prod"

If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport.