Teleport
Teleport AKS Auto-Discovery
- Edge version
- Version 17.x
- Version 16.x
- Version 15.x
- Older Versions
AKS Auto-Discovery can automatically discover any AKS cluster and enroll it in Teleport if its tags match the configured labels.
Teleport cluster auto-discovery involves two components:
- The Teleport Discovery Service that watches for new clusters or
changes to previously discovered clusters.
It dynamically registers each discovered cluster as a
kube_cluster
resource in your Teleport cluster. It does not need connectivity to the clusters it discovers. - The Teleport Kubernetes Service that monitors the dynamic
kube_cluster
resources registered by the Discovery Service. It proxies communications between users and the cluster.
This guide presents the Discovery Service and Kubernetes Service running in the same process, however both can run independently and on different machines.
For example, you can run an instance of the Kubernetes Service in the same private network as the clusters you want to register with your Teleport cluster, and an instance of the Discovery Service in any network you wish.
Prerequisites
-
A running Teleport cluster version 16.4.10 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.
-
The
tctl
admin tool andtsh
client tool.Visit Installation for instructions on downloading
tctl
andtsh
.
- An Azure identity with permissions to create and attach AD Groups.
- One or more AKS clusters running.
- Access to AKS clusters.
- A host to run the Teleport Discovery and Kubernetes services.
Step 1/2. Set up Azure Identity with the required permissions
Depending on each cluster's authentication and authorization settings, Azure uses a different way to configure the necessary permissions for Teleport to forward requests to the server.
Check the authentication modes used on your clusters and choose one or more permissions scenarios. In some configurations, Teleport has the ability to automatically configure the access to the cluster if you include the necessary permissions to do so.
In this scenario, the Teleport's authentication happens through Active Directory, and the permissions required to access the Kubernetes cluster are associated with the roles assigned to its identity.
This mode allows you to grant permissions to multiple Kubernetes clusters without requiring specific settings for each one of them.
To grant access to the AKS clusters running with this setting, create an AD role with the following content and assign it to the identity that the Teleport process will use.
{
"Name": "AKS Teleport Discovery",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [
"Microsoft.ContainerService/managedClusters/read",
"Microsoft.ContainerService/managedClusters/listClusterUserCredential/action"
],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
Replace the {subscription_id}
with the desired Subscription ID or a wildcard if
you want to guarantee permissions on all subscriptions.
When using Azure AD authentication with Kubernetes RBAC mode, Azure is responsible for user authentication using AD credentials, but permissions management is Kubernetes' RBAC responsibility.
Therefore, for Teleport to work correctly, you must create the Kubernetes ClusterRole
and ClusterRoleBinding
resources on each discovered cluster.
The ClusterRoleBinding
must bind the ClusterRole
to one of the AD groups
configured in the Teleport identity.
Teleport can automatically create the ClusterRole
and ClusterRoleBinding
resources
in the following cases:
- Teleport's AD identity has permissions that allow access to the static cluster administrator credentials (local accounts).
- Teleport's AD identity belongs to the cluster's administrator group.
- Teleport's AD identity has permissions to create
ClusterRole
andClusterRoleBinding
on the cluster and permissions to execute remote commands.
In either of the specified cases, Teleport will be able to create the ClusterRole
and bind
it to the first AD group it belongs. To make this possible, associate the
following permissions with Teleport's identity.
{
"Name": "AKS Teleport Discovery",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [
"Microsoft.ContainerService/managedClusters/read",
"Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action",
"Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
"Microsoft.ContainerService/managedClusters/runcommand/action",
"Microsoft.ContainerService/managedclusters/commandResults/read"
],
"NotActions": [],
"DataActions": [],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
In all other cases, you must manually set up the required access as described in the guide below.
Connect to every cluster to enroll with admin permissions and create the following resources
using kubectl
.
ClusterRole
Create the ClusterRole
RBAC definition with the required permissions for Teleport
Kubernetes Service to forward requests to the cluster.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
ClusterRoleBinding
Link the previously created ClusterRole
into the Teleport AD group id.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport
subjects:
- kind: Group
name: {azure_ad_group_id}
apiGroup: rbac.authorization.k8s.io
Replace {azure_ad_group_id}
with the appropriate Azure AD group id
associated with Teleport identity.
In this case, Teleport will use the user credentials generated during the provisioning phase of the cluster.
{
"Name": "AKS Teleport Discovery",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [
"Microsoft.ContainerService/managedClusters/read",
"Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
],
"NotActions": [],
"DataActions": [],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
Step 2/2. Configure Teleport to discover AKS clusters
Teleport AKS Auto-Discovery requires a valid auth token for the Discovery and
Kubernetes services to join the cluster. Generate one by running the following
command against your Teleport Auth Service and save it in /tmp/token
on the
machine that will run Kubernetes Discovery:
tctl tokens add --type=discovery,kube
Enabling AKS Auto-Discovery requires that the discovery_service.azure
section
include at least one entry and that discovery_service.azure.types
include aks
.
It also requires configuring the kubernetes_service.resources.tags
to use the same
labels configured at discovery_service.azure.tags
or a subset of them to make
the Kubernetes Service listen to the dynamic resources created by the Discovery
Service.
Discovery Service exposes a configuration parameter - discovery_service.discovery_group
-
that allows you to group discovered resources into different sets. This parameter
is used to prevent Discovery Agents watching different sets of cloud resources
from colliding against each other and deleting resources created by another services.
When running multiple Discovery Services, you must ensure that each service is configured
with the same discovery_group
value if they are watching the same cloud resources
or a different value if they are watching different cloud resources.
It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.
- 2 Discovery Services configured with
discovery_group: "prod"
polling data from Production account. - 2 Discovery Services configured with
discovery_group: "staging"
polling data from Staging account.
version: v3
teleport:
join_params:
token_name: "/tmp/token"
method: token
proxy_server: teleport.example.com:443
auth_service:
enabled: off
proxy_service:
enabled: off
ssh_service:
enabled: off
discovery_service:
enabled: "yes"
discovery_group: "aks-prod"
azure:
- types: ["aks"]
regions: ["*"]
subscriptions: ["*"]
resource_groups: ["*"]
tags:
"env": "prod"
kubernetes_service:
enabled: "yes"
resources:
- labels:
"env": "prod" # Match Kubernetes Cluster labels specified earlier
Once you have added this configuration, start Teleport. AKS clusters matching the tags and regions specified in the Azure section will be added to the Teleport cluster automatically.
Troubleshooting
Discovery Service troubleshooting
First, check if any Kubernetes clusters have been discovered.
To do this, you can use the tctl get kube_cluster
command and check if the
expected Kubernetes clusters have already been registered with your Teleport
cluster.
If some Kubernetes clusters do not appear in the list, check if the Discovery Service selector labels match the missing Kubernetes cluster tags or look into the Discovery Service logs for permission errors.
Check that the Discovery Service is running with credentials for the correct AWS account. It can discover resources in another AWS account, but it must be configured to assume a role in the other AWS account if that's the case.
Check if there is more than one Discovery Services running:
tctl inventory status --connected
If you are running multiple Discovery Services, you must ensure that each
service is configured with the same discovery_group
value if they are watching
the same cloud Kubernetes clusters or a different value if they are watching different
cloud Kubernetes clusters.
If this is not configured correctly, a typical symptom is kube_cluster
resources being intermittently deleted from your Teleport cluster's registry.
Kubernetes Service troubleshooting
If the tctl get kube_cluster
command returns the discovered clusters, but the
tctl kube ls
command does not include them, check that you have set the
kubernetes_service.resources
section correctly.
kubernetes_service:
enabled: "yes"
resources:
- labels:
"env": "prod"
If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport.