Teleport AKS Auto-Discovery

AKS Auto-Discovery can automatically discover any AKS cluster and enroll it in Teleport if its tags match the configured labels.

Teleport cluster auto-discovery involves two components:

1. The Teleport Discovery Service that watches for new clusters or changes to previously discovered clusters. It dynamically registers each discovered cluster as a kube_cluster resource in your Teleport cluster. It does not need connectivity to the clusters it discovers.
2. The Teleport Kubernetes Service that monitors the dynamic kube_cluster resources registered by the Discovery Service. It proxies communications between users and the cluster.

Prerequisites

• A running Teleport cluster version 16.4.8 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

• The tctl admin tool and tsh client tool.

  Visit Installation for instructions on downloading tctl and tsh.

• An Azure identity with permissions to create and attach AD Groups.
• One or more AKS clusters running.
• Access to AKS clusters.
• A host to run the Teleport Discovery and Kubernetes services.

Step 1/2. Set up Azure Identity with the required permissions

Depending on each cluster's authentication and authorization settings, Azure uses a different way to configure the necessary permissions for Teleport to forward requests to the server.

Check the authentication modes used on your clusters and choose one or more permissions scenarios. In some configurations, Teleport has the ability to automatically configure the access to the cluster if you include the necessary permissions to do so.

  {
      "Name": "AKS Teleport Discovery",
      "Description": "Required permissions for Teleport auto-discovery.",
      "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action"
      ],
      "NotActions": [],
      "DataActions": [
        "Microsoft.ContainerService/managedClusters/groups/impersonate/action",
        "Microsoft.ContainerService/managedClusters/users/impersonate/action",
        "Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
        "Microsoft.ContainerService/managedClusters/pods/read",
        "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
        "Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
      ],
      "NotDataActions": [],
      "assignableScopes": [
          "/subscriptions/{subscription_id}"
      ]
  }

  {
      "Name": "AKS Teleport Discovery",
      "Description": "Required permissions for Teleport auto-discovery.",
      "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
        "Microsoft.ContainerService/managedClusters/runcommand/action",
        "Microsoft.ContainerService/managedclusters/commandResults/read"
      ],
      "NotActions": [],
      "DataActions": [],
      "NotDataActions": [],
      "assignableScopes": [
          "/subscriptions/{subscription_id}"
      ]
  }

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: teleport
rules:
- apiGroups:
  - ""
  resources:
  - users
  - groups
  - serviceaccounts
  verbs:
  - impersonate
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - "authorization.k8s.io"
  resources:
  - selfsubjectaccessreviews
  - selfsubjectrulesreviews
  verbs:
  - create

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: teleport
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teleport
subjects:
- kind: Group
  name: {azure_ad_group_id}
  apiGroup: rbac.authorization.k8s.io

  {
      "Name": "AKS Teleport Discovery",
      "Description": "Required permissions for Teleport auto-discovery.",
      "Actions": [
        "Microsoft.ContainerService/managedClusters/read",
        "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
      ],
      "NotActions": [],
      "DataActions": [],
      "NotDataActions": [],
      "assignableScopes": [
          "/subscriptions/{subscription_id}"
      ]
  }

Step 2/2. Configure Teleport to discover AKS clusters

Teleport AKS Auto-Discovery requires a valid auth token for the Discovery and Kubernetes services to join the cluster. Generate one by running the following command against your Teleport Auth Service and save it in /tmp/token on the machine that will run Kubernetes Discovery:

tctl tokens add --type=discovery,kube

Enabling AKS Auto-Discovery requires that the discovery_service.azure section include at least one entry and that discovery_service.azure.types include aks. It also requires configuring the kubernetes_service.resources.tags to use the same labels configured at discovery_service.azure.tags or a subset of them to make the Kubernetes Service listen to the dynamic resources created by the Discovery Service.

version: v3
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  proxy_server: teleport.example.com:443    
auth_service:
  enabled: off
proxy_service:
  enabled: off
ssh_service:
  enabled: off
discovery_service:
  enabled: "yes"
  discovery_group: "aks-prod"
  azure:
  - types: ["aks"]
    regions: ["*"]
    subscriptions: ["*"]
    resource_groups: ["*"]
    tags:
      "env": "prod"
kubernetes_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod" # Match Kubernetes Cluster labels specified earlier

Once you have added this configuration, start Teleport. AKS clusters matching the tags and regions specified in the Azure section will be added to the Teleport cluster automatically.

Troubleshooting

Discovery Service troubleshooting

First, check if any Kubernetes clusters have been discovered. To do this, you can use the tctl get kube_cluster command and check if the expected Kubernetes clusters have already been registered with your Teleport cluster.

If some Kubernetes clusters do not appear in the list, check if the Discovery Service selector labels match the missing Kubernetes cluster tags or look into the Discovery Service logs for permission errors.

Check that the Discovery Service is running with credentials for the correct AWS account. It can discover resources in another AWS account, but it must be configured to assume a role in the other AWS account if that's the case.

Check if there is more than one Discovery Services running:

tctl inventory status --connected

If you are running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud Kubernetes clusters or a different value if they are watching different cloud Kubernetes clusters. If this is not configured correctly, a typical symptom is kube_cluster resources being intermittently deleted from your Teleport cluster's registry.

Kubernetes Service troubleshooting

If the tctl get kube_cluster command returns the discovered clusters, but the tctl kube ls command does not include them, check that you have set the kubernetes_service.resources section correctly.

kubernetes_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod"

If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport.