Fork me on GitHub

Teleport

Teleport EKS Auto-Discovery (Preview)

Improve

EKS Auto-Discovery can automatically discover any EKS cluster and enroll it in Teleport if its tags match the configured labels.

The discovery process requires two separate Teleport components. The first, the Discovery Service, is responsible for watching the cloud provider and checking if there are any new clusters or if there have been any modifications to previously discovered clusters. The second, the Kubernetes Service, monitors the clusters created by the Discovery Service and is responsible for authenticating and forwarding requests into them.

This guide presents the Discovery and Kubernetes services running in the same process, however both can run independently and on different machines. Running the processes independently is beneficial if access to different clusters requires Kubernetes Service to be on the same network as the cluster.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 11.0.3.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 11.0.3, which you can download by visiting the customer portal.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

  • A Teleport Cloud account. If you do not have one, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.3.8. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.3.8 go1.19

    tsh version

    Teleport v10.3.8 go1.19

  • An AWS account with permissions to create and attach IAM policies.
  • An AWS account with system:masters RBAC access to EKS clusters.
  • A host to run the Teleport Discovery and Kubernetes services.
  • One or more EKS clusters running.

Known limitations

Due to the authorization method that AWS has chosen for EKS clusters, it is not possible for the Teleport process to configure the necessary permissions to forward requests to EKS clusters. This limitation exists because the authorization method on EKS clusters requires that the IAM role - used for authentication - exists in the aws-auth ConfigMap.

In this ConfigMap, IAM roles are mapped to Kubernetes RBAC users or groups that are then used by Kubernetes RBAC to grant access permissions. If the mapping does not exist, it results in unauthorized requests. Therefore, Teleport cannot edit the ConfigMap without first having access to the cluster.

If Teleport is not running with the same IAM identity that creates all clusters, please ensure you configure the required permissions as described in Step 2.

Note

The AWS EKS team is working on a feature request to introduce an external API to manage access to the cluster without manually editing the Configmap (aws/containers-roadmap#185).

Hopefully, once the feature is available, Teleport can leverage it to configure its access to the cluster.

Step 1/3. Set up AWS IAM credentials

For Teleport to discover EKS clusters, the instance running the Discovery Service requires an IAM policy that allows Teleport to list and describe EKS clusters:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}      

Step 2/3. Configure EKS cluster authorization

When the Kubernetes Service uses an IAM role that is different from the one that creates the clusters, you need to configure the mapping between the Teleport IAM Role and the Kubernetes RBAC permissions by editing the aws-auth Configmap on each of the discovered clusters.

To forward requests to the Kubernetes cluster, the Teleport Kubernetes Service requires cluster-wide permissions to Impersonate RBAC users and groups, to create SelfSubjectAccessReviews and SelfSubjectRulesReviews, and, finally, read access to Pods.

If your Kubernetes cluster does not have an RBAC group with the required permissions, you can create the ClusterRole, ClusterRoleBinding, and the mapping by following the Creating RBAC group guide.
If your cluster already has an RBAC group that satisfies the required permissions, you can reuse it and map it into the IAM Role used by the Teleport Kubernetes Service. For simplicity, it is also possible to map the Teleport IAM role onto a built-in Kubernetes RBAC group like system:masters, but not recommended in production.

Connect to your target cluster with your credentials and create the following resources using kubectl.

ClusterRole

Create the ClusterRole RBAC definition with the required permissions for Teleport Kubernetes Service to forward requests to the cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: teleport
rules:
- apiGroups:
  - ""
  resources:
  - users
  - groups
  - serviceaccounts
  verbs:
  - impersonate
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - "authorization.k8s.io"
  resources:
  - selfsubjectaccessreviews
  - selfsubjectrulesreviews
  verbs:
  - create

ClusterRoleBinding

Link the previously created ClusterRole into a teleport RBAC group.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: teleport
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teleport
subjects:
- kind: Group
  name: teleport
  apiGroup: rbac.authorization.k8s.io

IAM mapping

Finally, edit the configmap/aws-auth in the kube-system namespace and append the following to mapRoles. Replace {teleport_aws_iam_role} with the appropriate IAM role that Teleport Kubernetes Service will use. This step will link the Teleport IAM role into the Kubernetes RBAC group teleport, allowing Teleport Kubernetes Service to forward requests into the cluster.

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - teleport
      rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role
      username: teleport

At this point, the Teleport IAM role already has the minimum permissions to forward requests to the cluster.

To associate the Teleport IAM role with an existing Kubernetes RBAC group, edit the configmap/aws-auth in the kube-system namespace and append the following to mapRoles.

apiVersion: v1
data:
  mapRoles: |
  ...
    - groups:
      - {rbac_group}
      rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role
      username: teleport

Please replace {teleport_aws_iam_role} with the appropriate IAM role that Teleport Kubernetes Service is using and {rbac_group} with the existing Kubernetes RBAC Group that satisfies the required permissions.

At this point, the Teleport IAM role already has the minimum permissions to forward requests to the cluster.

Granting the system:masters group to the IAM role associated with the Teleport service means granting administrator-level permissions on the Kubernetes cluster. To follow least privilege principle we do not recommend using this method in production.

To associate the Teleport IAM role with the system:masters RBAC group, edit the configmap/aws-auth in the kube-system namespace and append the following to mapRoles.

apiVersion: v1
data:
  mapRoles: |
  ...
    - groups:
      - system:masters
      rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role
      username: teleport

Please replace {teleport_aws_iam_role} with the appropriate IAM role that Teleport Kubernetes Service is using.

At this point, the Teleport IAM role already has the minimum permissions to forward requests to the cluster.

If you provision your EKS clusters using tools such as terraform, eksctl or Cloudformation, you can use them to automatically configure the aws-auth Configmap and create the ClusterRole and ClusterRoleBinding resources during cluster provisioning.

Step 3/3. Configure Teleport to discover EKS clusters

Teleport EKS Auto-Discovery requires a valid Teleport auth token for the Discovery and Kubernetes services to join the cluster. Generate one by running the following command against your Teleport Auth Service and save it in /tmp/token on the machine that will run Kubernetes Auto-Discovery:

tctl tokens add --type=discovery,kube

Enabling EKS Auto-Discovery requires that the discovery_service.aws section include at least one entry and that discovery_service.aws.types include eks. It also requires configuring the kubernetes_service.resources.tags to use the same labels configured at discovery_service.aws.tags or a subset of them to make the Kubernetes Service listen to the dynamic resources created by the Discovery Service.

version: v2
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  auth_servers:
  - "teleport.example.com:3080"
auth_service:
  enabled: off
proxy_service:
  enabled: off
ssh_service:
  enabled: off
discovery_service:
  enabled: "yes"
  aws:
   - types: ["eks"]
     regions: ["*"]
     tags:
       "env": "prod" # Match EKS cluster tags where tag:env=prod
kubernetes_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod" # Match Kubernetes Cluster labels specified earlier

Once you have added this configuration, start Teleport. EKS clusters matching the tags and regions specified in the AWS section will be added to the Teleport cluster automatically.

Troubleshooting

Discovery Service

To check if the Discovery Service is working correctly, you can check if any Kubernetes clusters have been discovered. To do this, you can use the tctl get kube_cluster command and inspect if the expected clusters have already been imported into Teleport.

If some clusters do not appear in the list, check if the filtering labels match the missing cluster tags or look into the service logs for permission errors.

Kubernetes Service

If the tctl get kube_cluster command returns the discovered clusters, but the tsh kube ls command does not include them, check that you have set the kubernetes_service.resources section correctly.

kubernetes_service:
  enabled: `yes`
  resources:
  - tags:
      "env": "prod"

If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport.