Terraform EC2 Auto-Discovery Configuration for AWS Organizations
This guide shows you how to use Terraform to configure Teleport and AWS to automatically enroll EC2 instances from your AWS organization into your Teleport cluster.
How it works
The teleport-discovery-aws Terraform module creates resources in your Teleport cluster and AWS organization to enable Teleport EC2 auto-discovery.
The Teleport Discovery Service queries the AWS API to list matching Accounts, and EC2 instances in each account. For each EC2 instance that it discovers, the Discovery Service uses AWS Systems Manager (SSM) to install Teleport on the instance and join it to the cluster as a Teleport-protected server.
Prerequisites
-
A running Teleport cluster. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.
-
The
tctlandtshclients.Installing
tctlandtshclients-
Determine the version of your Teleport cluster. The
tctlandtshclients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at/v1/webapi/findand use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:- Mac/Linux
- Windows - Powershell
TELEPORT_DOMAIN=teleport.example.com:443TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"$TELEPORT_DOMAIN = "teleport.example.com:443"$TELEPORT_VERSION = (Invoke-RestMethod -Uri "https://${TELEPORT_DOMAIN}/v1/webapi/find").server_version -
Follow the instructions for your platform to install
tctlandtshclients:- Mac
- Windows - Powershell
- Linux
Download the signed macOS .pkg installer for Teleport, which includes the
tctlandtshclients:curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkgIn Finder double-click the
pkgfile to begin installation.dangerUsing Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.
curl.exe -O https://cdn.teleport.dev/teleport-v$TELEPORT_VERSION-windows-amd64-bin.zipUnzip the archive and move the `tctl` and `tsh` clients to your %PATH%
NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP.
Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\<username>) instead.
All of the Teleport binaries in Linux installations include the
tctlandtshclients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our installation page.curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gztar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gzcd teleportsudo ./installTeleport binaries have been copied to /usr/local/bin
-
- EC2 instances running Ubuntu/Debian/RHEL/Amazon Linux 2/Amazon Linux 2023 and SSM Agent version 3.1 or greater
- Check that you can connect to your Teleport cluster and verify that you can run
tctlandtshcommands using your current credentials.-
Assign teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and [email protected] to your Teleport username.
-
Authenticate to your Teleport cluster. This depends on whether your shell is interactive or not.
In an interactive shell: Run the following command. By default, this triggers a multi-factor authentication prompt:
tsh login --proxy=teleport.example.com --user=[email protected]tctl statusCluster teleport.example.com
Version 19.0.0-dev
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
On non-interactive environments: If you are running
tshandtctlas an AI agent, in a CI/CD environment, or similar, make sure theTELEPORT_IDENTITY_FILEenvironment variable is assigned to a valid file path with credentials for your cluster.tshandtctlread the file path from the environment variable and do not require a separate authentication step. If there is no identity file available, we recommend that you set up Machine ID to provision one automatically.When executing
tctlcommands with an identity file, you must pass the--auth-serverflag to provide the Teleport Auth Service address, which is not included in the identity file. If you provide the Proxy Service address,tctlconnects to the Proxy Service, which forwards traffic to and from the Teleport Auth Service. Update 443 to3025if you are contacting the Auth Service directly withtctl:tctl status --auth-server=teleport.example.com:443For
tshcommands that read an identity file, you must pass the--proxyflag, which pointstshto the address of the Teleport Proxy Service:tsh status --proxy=teleport.example.comEnsure client commands can access your identity file. Replace path/to/identity/file with the path to your identity file:
export TELEPORT_IDENTITY_FILE="${TELEPORT_IDENTITY_FILE:-path/to/identity/file}"Add the
--auth-serveror--proxyflags to all subsequenttctlandtshcommands.
tctl statuscommand, you can use your current credentials to run subsequenttctlcommands from your workstation. If you host your own Teleport cluster, you can also runtctlcommands on the computer that hosts the Teleport Auth Service for full permissions. -
- Access to the management account or a member account that is a delegated administrator.
- AWS Terraform provider configured with credentials from the management account or a member account that is a delegated administrator.
- Teleport Terraform provider configured using one of the methods described in Using the Teleport Terraform Provider.
All EC2 instances added to the Teleport cluster by the Discovery Service must
include the AmazonSSMManagedInstanceCore IAM policy in order to receive
commands from the Discovery Service. For a list of permissions included in the
policy, see the AWS
documentation.
AWS IAM permissions required for AWS Terraform provider
The AWS Terraform provider will need the following AWS IAM permissions to manage AWS resources created by the teleport-discovery-aws module:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TerraformIdentity",
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
},
{
"Sid": "ManageIamRole",
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:DeleteRole",
"iam:GetRole",
"iam:ListInstanceProfilesForRole",
"iam:ListRolePolicies",
"iam:ListRoles",
"iam:ListRoleTags",
"iam:TagRole",
"iam:UntagRole",
"iam:UpdateAssumeRolePolicy"
],
"Resource": "*"
},
{
"Sid": "ManageIamPolicy",
"Effect": "Allow",
"Action": [
"iam:CreatePolicy",
"iam:CreatePolicyVersion",
"iam:DeletePolicy",
"iam:DeletePolicyVersion",
"iam:GetPolicy",
"iam:GetPolicyVersion",
"iam:ListPolicies",
"iam:ListPolicyTags",
"iam:ListPolicyVersions",
"iam:TagPolicy",
"iam:UntagPolicy"
],
"Resource": "*"
},
{
"Sid": "ManageRolePolicyAttachments",
"Effect": "Allow",
"Action": [
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:ListAttachedRolePolicies"
],
"Resource": "*"
},
{
"Sid": "ManageOidcProvider",
"Effect": "Allow",
"Action": [
"iam:CreateOpenIDConnectProvider",
"iam:DeleteOpenIDConnectProvider",
"iam:GetOpenIDConnectProvider",
"iam:ListOpenIDConnectProviders",
"iam:TagOpenIDConnectProvider",
"iam:UntagOpenIDConnectProvider",
"iam:UpdateOpenIDConnectProviderThumbprint"
],
"Resource": "*"
},
{
"Sid": "Organization",
"Effect": "Allow",
"Action": [
"organizations:ListAccounts",
"organizations:ListRoots",
"organizations:ListAWSServiceAccessForOrganization",
"organizations:DescribeOrganization"
],
"Resource": "*"
}
]
}
Step 1/7. Install Teleport to run the Discovery Service
If you already have a running Discovery Service instance, then assign its discovery_group to discovery-group-name and proceed to step 4: Configure AWS Terraform provider.
All Teleport Cloud clusters run the Discovery Service for you, so Teleport Cloud subscribers can also skip all of these installation steps.
If you plan on running the Discovery Service on the same host already running another Teleport service (Auth or Proxy, for example), then you can skip this step.
Install Teleport on a host instance that will run the Discovery Service:
To install Teleport binaries on your Linux server, the recommended installation method is the cluster install script. This script is served by your Teleport cluster's Proxy Service and automatically selects the correct version, edition, and installation mode to match your cluster.
-
Remove any existing Teleport binaries on your system:
sudo rm -f /usr/local/bin/{tsh,teleport,tctl,tbot,fdpass-teleport,teleport-update} -
Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https://).
-
Run your cluster's install script:
curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash
Step 2/7. Configure the Discovery Service
If you are running the Discovery Service on its own host, the service requires a valid invite token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service:
tctl tokens add --type=discovery
Save the generated token in /tmp/token on the Teleport Agent instance that will run the Discovery Service.
Discovery Service exposes a configuration parameter - discovery_service.discovery_group -
that allows you to group discovered resources into different sets. This parameter
is used to prevent Discovery Agents watching different sets of cloud resources
from colliding against each other and deleting resources created by another services.
When running multiple Discovery Services, you must ensure that each service is configured
with the same discovery_group value if they are watching the same cloud resources
or a different value if they are watching different cloud resources.
It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.
- 2 Discovery Services configured with
discovery_group: "prod"polling data from Production account. - 2 Discovery Services configured with
discovery_group: "staging"polling data from Staging account.
Assign teleport.example.com:443 to the host and port of the Teleport Proxy Service in your cluster, and discovery-group-name to a name that identifies a group of resources that you will enroll:
# teleport.yaml
version: v3
teleport:
join_params:
# token_name can be a literal token string or a path to a file.
# File path is preferable to avoid including a secret in the config file.
token_name: "/tmp/token"
method: token
proxy_server: "teleport.example.com:443"
auth_service:
enabled: false
proxy_service:
enabled: false
ssh_service:
enabled: false
discovery_service:
enabled: true
discovery_group: "discovery-group-name"
Step 3/7. Start Teleport Discovery Service
Grant the Discovery Service access to credentials that it can use to authenticate to AWS.
- If you are running the Discovery Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
- If you are running the Discovery Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
- Otherwise, you must use environment variables
- Instance Metadata Service
- Kubernetes IRSA
- Environment Variables
Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.
The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.
Refer to IAM Roles for Service Accounts (IRSA) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role.
Teleport's built-in AWS client reads credentials from the following environment variables:
AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_DEFAULT_REGION
When you start the Discovery Service, the service reads environment variables from a
file at the path /etc/default/teleport. Obtain these credentials from your
organization. Ensure that /etc/default/teleport has the following content,
replacing the values of each variable:
AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>
Have multiple sources of AWS credentials?
Teleport's AWS client loads credentials from different sources in the following order:
- Environment Variables
- Shared credentials file
- Shared configuration file (Teleport always enables shared configuration)
- EC2 Instance Metadata (credentials only)
While you can provide AWS credentials via a shared credentials file or shared
configuration file, you will need to run the Discovery Service with the AWS_PROFILE
environment variable assigned to the name of your profile of choice.
If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.
Start the Discovery Service. The instructions depend on how you installed the Discovery Service and whether your system supports systemd:
- Package Manager
- TAR Archive
- No systemd
Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. On the host where you will run the Discovery Service, enable and start Teleport:
sudo systemctl enable teleportsudo systemctl start teleport
You can check the status of the Discovery Service with systemctl status teleport
and view its logs with journalctl -fu teleport.
Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. On the host where you will run the Discovery Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:
sudo teleport install systemd -o /etc/systemd/system/teleport.servicesudo systemctl enable teleportsudo systemctl start teleport
You can check the status of the Discovery Service with systemctl status teleport
and view its logs with journalctl -fu teleport.
On the host where you will run the Discovery Service, start Teleport:
sudo teleport start --config=/etc/teleport.yaml
Teleport runs in the foreground and outputs logs for the services it is running.
Step 4/7. Configure the Terraform module inputs
Add the teleport-discovery-aws module to your Terraform configuration.
- Teleport Cloud
- Self-hosted
module "aws_discovery" {
source = "terraform.releases.teleport.dev/teleport/discovery/aws"
version = "~> 18.0"
# Required inputs:
# Assign example.teleport.sh:443 to the public address of the Teleport Proxy Service in your cluster in host:port form.
teleport_proxy_public_addr = "example.teleport.sh:443"
# teleport_discovery_group_name must match the discovery group name in your Discovery Service config file.
# Teleport Cloud clusters run the Discovery Service in the group name "cloud-discovery-group".
# Do not modify this input unless you intend to run your own Discovery Service.
teleport_discovery_group_name = "cloud-discovery-group"
# Enable AWS Organization discovery.
aws_organization_discovery = {
organizational_units = {
# The wildcard "*" matches all organizational units.
# You can also specify a list of organizational unit IDs to match (eg, "ou-1a2b-rg12qq2b")
include = ["*"]
# You can also specify a list of organizational unit IDs to exclude from discovery.
# exclude = ["ou-1a2b-rg12qq2b"]
# Excluded organizational units will not be discovered, even if they are included in the "include" list.
}
}
# Each child account in the organization must have an IAM role with this name.
# This role is assumed by the Discovery Service to enroll resources from that account.
# The required trust relationship and permissions for this role can be found in the module outputs and documentation.
aws_child_account_iam_role_name = "teleport-organization-discovery-child-account-role"
# Optional inputs:
# Apply the additional AWS tag "origin=example" to all AWS resources created by this module
apply_aws_tags = { origin = "example" }
# Apply the additional Teleport label "origin=example" to all Teleport resources created by this module
apply_teleport_resource_labels = { origin = "example" }
# Discover EC2 instances using matching rules.
# Accepts "*" to discover across all enabled regions.
# The module adds account:ListRegions to the IAM policy automatically when "*" is used.
aws_matchers = [
{
types = ["ec2"]
# EC2 discovery supports a wildcard to find instances in all regions.
regions = ["*"]
tags = {
env = ["prod"]
}
}
]
}
module "aws_discovery" {
source = "terraform.releases.teleport.dev/teleport/discovery/aws"
version = "~> 19.0"
# Required inputs:
# Edit this input to the host and port of the Teleport Proxy Service in your cluster
teleport_proxy_public_addr = "teleport.example.com:443"
# Edit this input to match the discovery group name in your Discovery Service config file
teleport_discovery_group_name = "discovery-group-name"
# Enable AWS Organization discovery.
aws_organization_discovery = {
organizational_units = {
# The wildcard "*" matches all organizational units.
# You can also specify a list of organizational unit IDs to match (eg, "ou-1a2b-rg12qq2b")
include = ["*"]
# You can also specify a list of organizational unit IDs to exclude from discovery.
# exclude = ["ou-1a2b-rg12qq2b"]
# Excluded organizational units will not be discovered, even if they are included in the "include" list.
}
}
# Each child account in the organization must have an IAM role with this name.
# This role is assumed by the Discovery Service to enroll resources from that account.
# The required trust relationship and permissions for this role can be found in the module outputs and documentation.
aws_child_account_iam_role_name = "teleport-organization-discovery-child-account-role"
# Discover EC2 instances using matching rules.
# Accepts "*" to discover across all enabled regions.
# The module adds account:ListRegions to the IAM policy automatically when "*" is used.
aws_matchers = [
{
types = ["ec2"]
# EC2 discovery supports a wildcard to find instances in all regions.
regions = ["*"]
tags = {
env = ["prod"]
}
}
]
}
By default, the teleport-discovery-aws Terraform module configures an AWS OIDC integration for discovery and enrollment of EC2 instances.
However, this requires the Teleport Proxy Service to be reachable on the public internet.
If your cluster is running in a private network and the Proxy Service is not reachable on the public internet, you can disable the OIDC integration and assign the necessary IAM permissions for discovering and enrolling EC2 instances.
You can disable the AWS OIDC integration by setting the following:
discovery_service_iam_credential_source = {
use_oidc_integration = false
}
# This role must be created manually and be assigned to the Discovery Service.
aws_organization_discovery_iam_principal_arn = "arn:aws:iam::<root-account-id>:role/teleport-organization-account-enumeration"
After applying the terraform module, run terraform output to obtain the IAM role templates for the Discovery Service and Auth Service:
aws_discovery = {
# ...
"teleport_organization_account_enumeration_iam_role_template" = {
"policy" = <<-EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"organizations:ListRoots",
"organizations:ListChildren",
"organizations:ListAccountsForParent"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::*:role/teleport-organization-discovery-child-account-role",
"Condition": {
"StringEquals": {
"aws:ResourceOrgID": "<aws-organization-id>"
}
}
}
]
}
EOT
"role_name" = "teleport-organization-account-enumeration"
}
"teleport_organization_join_validation_iam_role_template" = {
"policy" = <<-EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "organizations:DescribeAccount",
"Resource": "*"
}
]
}
EOT
"role_name" = "teleport-organization-join-validation"
}
}
The IAM role attached to the Discovery Service must have the policy defined in teleport_organization_account_enumeration_iam_role_template.
The role attached to the Auth Service must have the policy defined in teleport_organization_join_validation_iam_role_template.
In setups where the Discovery Service and Auth Service are running from the same process, you can combine the two policies into a single IAM role and attach it to the instance that is running the services.
Add a Terraform output for the module so that Terraform will display its outputs:
output "aws_discovery" {
value = module.aws_discovery
}
See the teleport-discovery-aws reference for a complete description of the module inputs and outputs.
Step 5/7. Apply the Terraform module
terraform initterraform apply
Terraform should plan to create the following resources:
- AWS IAM role for Teleport Discovery Service to assume
- AWS IAM policy that grants the AWS permissions necessary for Teleport to enumerate AWS accounts and assume roles in other accounts within the AWS Organization
- AWS IAM policy attachment to attach the IAM policy to the Discovery Service IAM role
- AWS OIDC Provider for Teleport Discovery Service to assume an IAM role using OIDC.
- Teleport
discovery_configcluster resource that configures Teleport for AWS resource discovery. - Teleport
integrationcluster resource for AWS OIDC - Teleport
tokencluster resource that allows Teleport nodes to use AWS IAM credentials to join the cluster
Review the Terraform plan and confirm the plan actions.
After Terraform finishes applying the plan, it should display the module outputs:
aws_discovery = {
"aws_child_account_iam_role_template" = {
"assume_role_policy" = <<-EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::<root-account-id>:role/teleport-discovery-<random-id>"
},
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "<aws-organization-id>"
}
}
}
]
}
EOT
"policy" = <<-EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:SendCommand",
"ssm:ListCommandInvocations",
"ssm:GetCommandInvocation",
"ssm:DescribeInstanceInformation",
"ec2:DescribeInstances",
"account:ListRegions"
],
"Resource": "*"
}
]
}
EOT
"role_name" = "teleport-organization-discovery-child-account-role"
}
"aws_oidc_provider_arn" = "arn:aws:iam::<root-account-id>:oidc-provider/example.teleport.sh"
"teleport_discovery_config_name" = "discovery-aws-organization-<aws-organization-id>"
"teleport_discovery_service_iam_policy_arn" = null
"teleport_discovery_service_iam_role_arn" = "arn:aws:iam::<root-account-id>:role/teleport-discovery-<random-id>"
"teleport_integration_name" = "discovery-aws-organization-<aws-organization-id>"
"teleport_organization_account_enumeration_iam_policy_arn" = "arn:aws:iam::<root-account-id>:policy/teleport-organization-account-enumeration-<random-id>"
"teleport_organization_account_enumeration_iam_role_template" = null /* object */
"teleport_organization_join_validation_iam_policy_arn" = "arn:aws:iam::<root-account-id>:policy/teleport-organization-join-validation-<random-id>"
"teleport_organization_join_validation_iam_role_template" = null /* object */
"teleport_provision_token_name" = "discovery-aws-organization-<aws-organization-id>"
}
The AWS resources should have the following tags:
origin=exampleteleport.dev/cluster=<cluster-name>teleport.dev/integration=discovery-aws-organization-<aws-organization-id>teleport.dev/iac-tool=terraform
The Teleport resources should have the following labels:
origin=exampleteleport.dev/iac-tool=terraform
Step 6/7. Configure child accounts
Each AWS account within the organization must have an IAM role which will be assumed by the Teleport Discovery Service to discover and enroll EC2 instances in that account.
Run the following command to obtain the IAM role name, trust relationship policy and permissions policy:
terraform output
You will obtain the following output:
$ terraform output
aws_discovery = {
"aws_child_account_iam_role_template" = {
"assume_role_policy" = <<-EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"AWS": "arn:aws:iam::<root-account-id>:role/teleport-discovery-<random-id>"
},
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "<aws-organization-id>"
}
}
}
]
}
EOT
"policy" = <<-EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:SendCommand",
"ssm:ListCommandInvocations",
"ssm:GetCommandInvocation",
"ssm:DescribeInstanceInformation",
"ec2:DescribeInstances",
"account:ListRegions"
],
"Resource": "*"
}
]
}
EOT
"role_name" = "teleport-organization-discovery-child-account-role"
}
// ...
}
Create an IAM role in each target account using the role_name, assume_role_policy and policy from the Terraform output aws_child_account_iam_role_template variable.
Step 7/7. Check discovery status
After applying the Terraform module and creating the IAM role in each child account, the Teleport Discovery Service should start to discover EC2 instances in your AWS organization and enroll them in your cluster as protected resources.
It may take a few minutes for EC2 instances to be discovered and enrolled.
Navigate to the Teleport Web UI and select Zero Trust Access > Integrations.
By default, the integration created by the teleport-discovery-aws Terraform module is named discovery-aws-organization-<aws-organization-id>.
Click on the integration for your AWS organization to review the discovery status.
The integration page provides an overview of how many EC2 instances have been discovered and any issues encountered during the discovery process.
In the example above we can see there was an issue with the AWS SSM agent on the instance.
If we navigate to the "issues" tab and click on "Details" we can see more information about the issue:
In this case the issue is that the EC2 instance does not have an IAM role with the AmazonSSMManagedInstanceCore IAM policy attached to it, so the instance cannot receive SSM commands.
We can fix this by attaching the AmazonSSMManagedInstanceCore IAM policy to the EC2 instance's role and waiting for the Discovery Service to reattempt the installation with SSM. It may take several minutes for the next discovery scan to run and install Teleport on the instance.
When all of the discovered EC2 instances have successfully joined the Teleport cluster, the overview page will display a healthy status:
Updating module configuration
The module inputs can be changed and re-applied to adjust the AWS discovery integration.
For example, if you had previously used specific AWS regions (rather than the wildcard default for all regions), then you can adjust regions (within aws_matchers) to include additional regions for EC2 discovery and re-apply the module to start enrolling instances in those regions as well.
-regions = ["us-west-1"]
+regions = ["us-west-1", "us-east-1"]
Apply Terraform again:
terraform apply
Review the Terraform plan before confirming the changes.
After Terraform finishes applying its plan, the Discovery Service will pick up the change to the dynamic discovery_config and begin to enroll EC2 instances in us-east-1 as well.
Troubleshooting
Inspect AWS discovery status
Run tctl discovery status to check the state of dynamic Discovery Service
configurations, when each Discovery Service instance last reported, and the latest
AWS enrollment totals:
tctl discovery status --cloud=aws
The following example shows a healthy configuration that discovered and enrolled two resources:
Discovery config example-discovery:
Discovery group: production
Status: healthy
Last run: 1 minute ago
Service (00000000-0000-0000-0000-000000000000):
Poll interval: 5 minutes
Last update: 1 minute ago
example-integration:
AWS EC2:
Previous sync: 1 minute ago (took 12s)
Result: 2 found, 2 enrolled, 0 failed
A persistent error or not reporting yet status, a message that no Discovery
Service instances are running, or a nonzero failed count indicates a discovery
or enrollment problem.
The command reports dynamic discovery_config resources. It does not include
static discovery matchers configured in teleport.yaml.
See tctl discovery status
for the full flag reference.
Inspect instance enrollment status
The tctl discovery nodes command lists Teleport's recent attempts to enroll
AWS EC2 instances and reports whether each succeeded. It reads ssm.run audit
events emitted by the Discovery Service.
tctl discovery nodes --cloud=awstctl discovery nodes --cloud=aws --failures-onlytctl discovery nodes --cloud=aws --last=24htctl discovery nodes --cloud=aws --format=json
Sample output:
Cloud Account Region Instance ID Time Status Details----- ------------ ------------ ------------------- -------------------- ---------------------- -------------------------------------------------AWS 123456789012 eu-central-1 i-0000000000aaaaaa1 2026-04-02T15:16:03Z OnlineAWS 123456789012 eu-central-1 i-0000000000bbbbbb2 2026-04-02T15:00:24Z Failed (exit code=-1) SSM Agent lost connectionAWS 123456789012 eu-central-1 i-0000000000cccccc3 2026-04-02T15:00:19Z Failed (exit code=104) SSM Script failureAWS 123456789012 eu-central-1 i-0000000000dddddd4 2026-04-02T15:00:15Z Installed (offline) Script output: "Offloading the installation pa...
Each row's Status column tells you what stage the instance reached:
Online: the instance joined the cluster.Installed (offline): the install script succeeded but the agent isn't currently connected. Either the agent installed but never joined the cluster (bad join token, misconfiguration, or network), or it joined and later went offline.Failed (exit code=N): the install script ran and exited non-zero. The Details column shows the script output. See Installation script exit codes above for known values.
Use --last to widen the audit-event lookback window past the default one
hour (e.g. --last=24h) when investigating older failures or retries.
Use --format=json for machine-readable output that includes the full script
output, the originating user task ID, and AWS-specific instance metadata.
See tctl discovery nodes
for the full flag reference.
If Installs are showing failed or instances are failing to appear check the Command history in AWS Systems Manager -> Node Management -> Run Command. Select the instance-id of the Target to review Errors.
Installation script exit codes
When enrolling an instance into Teleport, the installation script returns a specific exit code when facing well-known issues:
- 100:
bashbinary is missing - 101:
sudobinary is missing - 102:
curlbinary is missing - 103:
/optor/do not have the required minimum space to install Teleport, at least 1250MB is required - 104: host is unable to connect to the Teleport Proxy Services's HTTPS endpoint
- 150: Teleport was installed but the agent failed to join the cluster
You can customize the installation script - as describe in the Use a custom installation script section above - and implement other checks with specific exit codes.
Whether the installation exits with a pre-flight check or with your own custom installation script, the exit code appears in ssm.run Teleport audit events, in the exit_code field.
It also appears in AWS Systems Manager -> Node Management -> Run Command -> Command history.
Example of the ssm.run event with an exit code:
{
"code": "TDS00W",
"event": "ssm.run",
"instance_id": "i-0000",
"region": "<region>",
"invocation_url": "https://<region>.console.aws.amazon.com/systems-manager/run-command/<uuid>/i-0000",
"platform_name": "Ubuntu",
"platform_type": "Linux",
"platform_version": "24.04",
"status": "curl is not installed in the instance. Please install all required tools (bash, sudo, curl) and try again.",
"stderr": "failed to run commands: exit status 102",
"stdout": "curl is missing\n",
"exit_code": 102
}
Interpreting exit code 150 (join failure)
Exit code 150 means Teleport installed successfully, but join health checks did not complete.
The status field is intentionally high-level; use stderr for root-cause details.
Common stderr patterns:
ERROR: node did not become ready (join cluster) within <duration>means Teleport kept polling/readyzuntil the configured join-health timeout elapsed without a successful join.systemd service state: ActiveState="...", SubState="...", Result="..."is a best-effort snapshot captured at timeout for troubleshooting context.join failure: token is expired or not found; ...can appear as an additional enriched line when journal logs contain an explicit token-expiry signal.failed to run commands: exit status 150can appear at the end when AWS SSM's run-shell wrapper reports the installer exit code.
Example ssm.run event for exit code 150:
{
"code": "TDS00W",
"event": "ssm.run",
"instance_id": "i-0000",
"region": "<region>",
"invocation_url": "https://<region>.console.aws.amazon.com/systems-manager/run-command/<uuid>/i-0000",
"platform_name": "Ubuntu",
"platform_type": "Linux",
"platform_version": "24.04",
"status": "Teleport was installed successfully but the agent did not become ready within the configured timeout. Check standard error output for join diagnostics.",
"stderr": "...\nERROR: node did not become ready (join cluster) within <duration>\n\nsystemd service state: ActiveState=\"active\", SubState=\"running\", Result=\"success\"\n\njoin failure: token is expired or not found; systemd service state: ActiveState=\"active\", SubState=\"running\", Result=\"success\"\n\tnode did not become ready (join cluster) within <duration>\n\nJournal output:\n...\nfailed to run commands: exit status 150",
"stdout": "",
"exit_code": 150
}
cannot unmarshal object into Go struct field
If you encounter an error similar to the following:
invalid format in plugin properties map[destinationPath:/tmp/installTeleport.sh sourceInfo:map[url:[https://example.teleport.sh:443/webapi/scripts/installer/preprod-installer](https://example.teleport.sh/webapi/scripts/installer/preprod-installer)] sourceType:HTTP];
error json: cannot unmarshal object into Go struct field DownloadContentPlugin.sourceInfo of type string
It is likely that you're running an older SSM agent version. Upgrade to SSM agent version 3.1 or greater to resolve.
InvalidInstanceId: Instances [[i-123]] not in a valid state for account 456
The following problems can cause this error:
- The Discovery Service doesn't have permission to access the managed node.
- AWS Systems Manager Agent (SSM Agent) isn't running. Verify that SSM Agent is running.
- SSM Agent isn't registered with the SSM endpoint. Try reinstalling SSM Agent.
- The discovered instance does not have permission to receive SSM commands, verify the instance includes the AmazonSSMManagedInstanceCore IAM policy.
See SSM RunCommand error codes and troubleshooting information in AWS documentation for more details:
- https://docs.aws.amazon.com/systems-manager/latest/userguide/troubleshooting-managed-instances.html
- https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_SendCommand.html#API_SendCommand_Errors
Next steps
- Read Joining Nodes via AWS IAM Role for more information on IAM Invite Tokens.
- Information on IAM best practices on EC2 instances managed by Systems Manager can be found in the AWS Cloud Operations & Migrations Blog .
- Full documentation on EC2 discovery configuration can be found through the config file reference documentation.