# Terraform EKS Auto-Discovery Configuration This guide shows you how to use Terraform to configure Teleport and AWS to automatically enroll EKS clusters in your Teleport cluster. ## How it works Teleport cluster auto-discovery involves two components: 1. The Teleport Discovery Service that watches for new clusters or changes to previously discovered clusters. It dynamically registers each discovered cluster as a `kube_cluster` resource in your Teleport cluster. It does not need connectivity to the clusters it discovers. 2. The Teleport Kubernetes Service that monitors the dynamic `kube_cluster` resources registered by the Discovery Service. It proxies communications between users and the cluster. The [teleport-discovery-aws](https://goteleport.com/docs/reference/infrastructure-as-code/terraform-modules/teleport-discovery-aws.md) Terraform module creates the AWS IAM resources and the Teleport integration, `discovery_config`, and provision token that configure the Discovery Service to enroll EKS clusters. When the Discovery Service has the required IAM permissions, it also provisions an EKS Access Entry for each discovered cluster so the Kubernetes Service can forward traffic to it. ## Prerequisites - A running Teleport cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/get-started/deploy-community.md). - The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: **Mac/Linux** ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` **Windows - Powershell** ``` $ $TELEPORT_DOMAIN = "teleport.example.com:443" $ $TELEPORT_VERSION = (Invoke-RestMethod -Uri "https://${TELEPORT_DOMAIN}/v1/webapi/find").server_version ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v$TELEPORT_VERSION-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/installation/single-machine.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` * An AWS account with IAM permissions to create roles, policies, and an OIDC provider. See the next step for the full permissions Terraform needs. * [Terraform v1.0.0+](https://learn.hashicorp.com/tutorials/terraform/install-cli). ``` $ terraform version Terraform v1.0.0 ``` * A host to run the Teleport Discovery and Kubernetes services. Teleport Cloud runs the Discovery Service for you, but you still need to run your own Kubernetes Service. See the [manual guide](https://goteleport.com/docs/enroll-resources/auto-discovery/kubernetes/eks-discovery/eks-discovery-manual.md) for service setup. * To check that you can connect to your Teleport cluster, sign in with `tsh login`, then verify that you can run `tctl` commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email\@example.com to your Teleport username: ``` $ tsh login --proxy=teleport.example.com --user=email@example.com $ tctl status Cluster teleport.example.com Version 18.7.6 CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 ``` If you can connect to the cluster and run the `tctl status` command, you can use your current credentials to run subsequent `tctl` commands from your workstation. If you host your own Teleport cluster, you can also run `tctl` commands on the computer that hosts the Teleport Auth Service for full permissions. ## Use the `teleport-discovery-aws` Terraform module ### Step 1/5. Configure AWS Terraform provider Configure the [AWS Terraform provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) and AWS IAM permissions for Terraform to manage AWS resources. AWS IAM permissions required for AWS Terraform provider The AWS Terraform provider will need the following AWS IAM permissions to manage AWS resources created by the `teleport-discovery-aws` module: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "TerraformIdentity", "Effect": "Allow", "Action": "sts:GetCallerIdentity", "Resource": "*" }, { "Sid": "ManageIamRole", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:DeleteRole", "iam:GetRole", "iam:ListInstanceProfilesForRole", "iam:ListRolePolicies", "iam:ListRoles", "iam:ListRoleTags", "iam:TagRole", "iam:UntagRole", "iam:UpdateAssumeRolePolicy" ], "Resource": "*" }, { "Sid": "ManageIamPolicy", "Effect": "Allow", "Action": [ "iam:CreatePolicy", "iam:CreatePolicyVersion", "iam:DeletePolicy", "iam:DeletePolicyVersion", "iam:GetPolicy", "iam:GetPolicyVersion", "iam:ListPolicies", "iam:ListPolicyTags", "iam:ListPolicyVersions", "iam:TagPolicy", "iam:UntagPolicy" ], "Resource": "*" }, { "Sid": "ManageRolePolicyAttachments", "Effect": "Allow", "Action": [ "iam:AttachRolePolicy", "iam:DetachRolePolicy", "iam:ListAttachedRolePolicies" ], "Resource": "*" }, { "Sid": "ManageOidcProvider", "Effect": "Allow", "Action": [ "iam:CreateOpenIDConnectProvider", "iam:DeleteOpenIDConnectProvider", "iam:GetOpenIDConnectProvider", "iam:ListOpenIDConnectProviders", "iam:TagOpenIDConnectProvider", "iam:UntagOpenIDConnectProvider", "iam:UpdateOpenIDConnectProviderThumbprint" ], "Resource": "*" } ] } ``` ### Step 2/5. Configure Teleport Terraform provider For a local quick start, log in with `tsh` and run `tctl terraform env` in the same shell you use for `terraform` commands: ``` $ tsh login $ eval "$(tctl terraform env)" ``` See the [Local Demo](https://goteleport.com/docs/zero-trust-access/infrastructure-as-code/terraform-provider/local.md) guide for more detail on local setup. If you are running Terraform in CI, Spacelift, or another remote environment, refer to [Using the Teleport Terraform Provider](https://goteleport.com/docs/zero-trust-access/infrastructure-as-code/terraform-provider.md) for the appropriate guide. ### Step 3/5. Configure the Terraform module inputs Add the `teleport-discovery-aws` module to your Terraform configuration. **cloud** ``` module "aws_discovery" { source = "terraform.releases.teleport.dev/teleport/discovery/aws" version = "~> 18.0" # Required inputs: # Assign example.teleport.sh:443 to your Teleport cluster's proxy public address in host:port form. teleport_proxy_public_addr = "example.teleport.sh:443" # teleport_discovery_group_name must match the discovery group name in your Discovery Service config file. # Teleport Cloud clusters run the Discovery Service in the group name "cloud-discovery-group". # Do not modify this input unless you intend to run your own Discovery Service. teleport_discovery_group_name = "cloud-discovery-group" aws_matchers = [ { types = ["eks"] # EKS discovery supports "*" to discover clusters across all enabled regions. regions = ["*"] # tags filter which EKS clusters are enrolled. The wildcard matches all tags. tags = { "*" : ["*"] } # Also enroll HTTP applications running inside discovered clusters. kube_app_discovery = true } ] # Optional inputs: # Apply the additional AWS tag "origin=example" to all AWS resources created by this module apply_aws_tags = { origin = "example" } # Apply the additional Teleport label "origin=example" to all Teleport resources created by this module apply_teleport_resource_labels = { origin = "example" } } ``` **self-hosted** ``` module "aws_discovery" { source = "terraform.releases.teleport.dev/teleport/discovery/aws" version = "~> 18.0" # Required inputs: # Edit this input to the host and port of the Teleport Proxy Service in your cluster teleport_proxy_public_addr = "teleport.example.com:443" # Assign discovery-group-name to the discovery group name in your Discovery Service config file. teleport_discovery_group_name = "discovery-group-name" aws_matchers = [ { types = ["eks"] # EKS discovery supports "*" to discover clusters across all enabled regions. regions = ["*"] # tags filter which EKS clusters are enrolled. The wildcard matches all tags. tags = { "*" : ["*"] } # Also enroll HTTP applications running inside discovered clusters. kube_app_discovery = true } ] # Optional inputs: # Apply the additional AWS tag "origin=example" to all AWS resources created by this module apply_aws_tags = { origin = "example" } # Apply the additional Teleport label "origin=example" to all Teleport resources created by this module apply_teleport_resource_labels = { origin = "example" } } ``` Add a Terraform output for the module so that Terraform will display its outputs: ``` output "aws_discovery" { value = module.aws_discovery } ``` See the [teleport-discovery-aws reference](https://goteleport.com/docs/reference/infrastructure-as-code/terraform-modules/teleport-discovery-aws.md) for a complete description of the module inputs and outputs. #### Granting access to discovered EKS clusters The IAM role created by this module includes permissions for the Discovery Service to create and update EKS Access Entries. When the Discovery Service finds a new EKS cluster, it automatically provisions an Access Entry so the Teleport Kubernetes Service can forward traffic. By default, the Discovery Service grants access to its own IAM role. If your Kubernetes Service runs with a different IAM role, add `setup_access_for_arn` to the EKS matcher with that role's ARN: ``` aws_matchers = [ { types = ["eks"] regions = ["*"] tags = { "*" : ["*"] } setup_access_for_arn = "arn:aws:iam::123456789012:role/teleport-kube-service" } ] ``` ### Step 4/5. Apply the Terraform module ``` $ terraform init $ terraform apply ``` Terraform should plan to create the following resources: - AWS IAM role for Teleport Discovery Service to assume - AWS IAM policy that grants the AWS permissions necessary for Teleport to discover resources in AWS - AWS IAM policy attachment to attach the IAM policy to the Discovery Service IAM role - AWS OIDC Provider for Teleport Discovery Service to assume an IAM role using OIDC - Teleport `discovery_config` cluster resource that configures Teleport for AWS resource discovery - Teleport `integration` cluster resource for AWS OIDC - Teleport `token` cluster resource that allows Teleport nodes to use AWS IAM credentials to join the cluster Review the Terraform plan and confirm the plan actions. After Terraform finishes applying the plan, it should display the module outputs: ``` aws_discovery = { "aws_oidc_provider_arn" = "arn:aws:iam::123456789012:oidc-provider/example.teleport.sh" "teleport_discovery_config_name" = "discovery-aws-account-123456789012" "teleport_discovery_service_iam_policy_arn" = "arn:aws:iam::123456789012:policy/teleport-discovery-" "teleport_discovery_service_iam_role_arn" = "arn:aws:iam::123456789012:role/teleport-discovery-" "teleport_integration_name" = "discovery-aws-account-123456789012" "teleport_provision_token_name" = "discovery-aws-account-123456789012" } ``` The AWS resources should have the following tags: - `origin=example` - `teleport.dev/cluster=` - `teleport.dev/integration=discovery-aws-account-` - `teleport.dev/iac-tool=terraform` The Teleport resources should have the following labels: - `origin=example` - `teleport.dev/iac-tool=terraform` ### Step 5/5. Check discovery status After applying the Terraform module, the Teleport Discovery Service should start to discover EKS clusters in your AWS account and enroll them in your Teleport cluster as `kube_cluster` resources. --- NOTE It may take a few minutes for EKS clusters to be discovered and enrolled. --- Navigate to the Teleport Web UI and select `Zero Trust Access > Integrations`. By default, the integration created by the `teleport-discovery-aws` Terraform module is named `discovery-aws-account-`. Click on the integration for your AWS account to review the discovery status. The integration page provides an overview of how many EKS clusters have been discovered and any issues encountered during the discovery process. ## Updating module configuration The module inputs can be changed and re-applied to adjust the AWS discovery integration. For example, if you had previously configured specific AWS regions, you can switch to the wildcard to discover EKS clusters in all enabled regions: ``` -regions = ["us-west-1"] +regions = ["*"] ``` Apply Terraform again: ``` $ terraform apply ``` Review the Terraform plan before confirming the changes. After Terraform finishes applying its plan, the Discovery Service will pick up the change to the dynamic `discovery_config` and begin to enroll EKS clusters in the newly-configured regions. ## Troubleshooting ### Discovery Service troubleshooting First, check if any Kubernetes clusters have been discovered. To do this, you can use the `tctl get kube_cluster` command and check if the expected Kubernetes clusters have already been registered with your Teleport cluster. If some Kubernetes clusters do not appear in the list, check if the Discovery Service selector labels match the missing Kubernetes cluster tags or look into the Discovery Service logs for permission errors. Check that the Discovery Service is running with credentials for the correct AWS account. It can discover resources in another AWS account, but it must be configured to assume a role in the other AWS account if that's the case. Check if there is more than one Discovery Services running: ``` $ tctl inventory status --connected ``` If you are running multiple Discovery Services, you must ensure that each service is configured with the same `discovery_group` value if they are watching the same cloud Kubernetes clusters or a different value if they are watching different cloud Kubernetes clusters. If this is not configured correctly, a typical symptom is `kube_cluster` resources being intermittently deleted from your Teleport cluster's registry. ### Kubernetes Service troubleshooting If the `tctl get kube_cluster` command returns the discovered clusters, but the `tctl kube ls` command does not include them, check that you have set the `kubernetes_service.resources` section correctly. ``` kubernetes_service: enabled: true resources: - labels: "env": "prod" ``` If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport. ## Next steps - Read the [teleport-discovery-aws module reference](https://goteleport.com/docs/reference/infrastructure-as-code/terraform-modules/teleport-discovery-aws.md) for a complete description of the module's inputs and outputs. - Learn how Teleport can also [discover HTTP applications running inside your Kubernetes clusters](https://goteleport.com/docs/enroll-resources/auto-discovery/kubernetes-applications.md). - Review the [auto-discovery labels reference](https://goteleport.com/docs/enroll-resources/auto-discovery/reference/labels.md) for the cloud tags that Teleport imports as Kubernetes cluster labels.