Skip to main content
Version: 19.x (unreleased)

EC2 Auto-Discovery Configuration for AWS Organizations for self-hosted Teleport

Report an Issue

This guide shows how to configure Teleport to automatically enroll EC2 instances from an AWS Organization in your cluster.

How it works​

The Teleport Discovery Service runs on an EC2 instance and queries the AWS API to list instances in all the accounts under an AWS Organization. For any new EC2 instance that you deploy, the Discovery Service uses AWS Systems Manager (SSM) to install Teleport on the instance and join it to the cluster as a Teleport-protected server.

If you only have a single AWS Account, read Manual EC2 Auto-Discovery Configuration.

Prerequisites​

  • A running Teleport cluster. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl and tsh clients.

    Installing tctl and tsh clients

    1. Determine the version of your Teleport cluster. The tctl and tsh clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at /v1/webapi/find and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service:

      TELEPORT_DOMAIN=teleport.example.com:443
      TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')"

    2. Follow the instructions for your platform to install tctl and tsh clients:

      Download the signed macOS .pkg installer for Teleport, which includes the tctl and tsh clients:

      curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg

      In Finder double-click the pkg file to begin installation.

      danger

      Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security.

  • AWS Organization with EC2 instances and permissions to create and attach IAM policies.
  • EC2 instances running Ubuntu/Debian/RHEL/Amazon Linux 2/Amazon Linux 2023 and SSM agent version 3.1 or greater if making use of the default Teleport install script. (For other Linux distributions, you can install Teleport manually.)
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and [email protected] to your Teleport username: 
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status
    Cluster  teleport.example.com
    Version  19.0.0-dev
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

All EC2 instances that are to be added to the Teleport cluster by the Discovery Service must include the AmazonSSMManagedInstanceCore IAM policy in order to receive commands from the Discovery Service. For a list of permissions included in the policy, see the AWS documentation.

Step 1/7. Create an EC2 invite token in Teleport​

When discovering EC2 instances, Teleport makes use of IAM invite tokens for authenticating joining Nodes.

The Organization ID starts with "o-" followed by a series of alphanumeric characters, and can be found on the left side of the AWS Organizations console.

Assign the o-organisation to your Organization ID.

Create a file called token.yaml:

# token.yaml
kind: token
version: v2
metadata:
  # the token name is not a secret because instances must prove that they are
  # running in your AWS Organization to use this token
  name: aws-discovery-iam-token
spec:
  # use the minimal set of roles required (e.g. Node, App, Kube, DB, WindowsDesktop)
  roles: [Node]

  # set the join method allowed for this token
  join_method: iam

  allow:
  # specify the AWS Organization ID which Nodes may join from
  - aws_organization_id: o-organisation

Add the token to the Teleport cluster with:

tctl create -f token.yaml

Step 2/7. Allow the Auth Service to verify organization membership​

The Auth Service must be able to verify that joining instances belong to the specified AWS Organization, by calling the organizations:DescribeAccount API.

tip

Auth Service must be run in the AWS Organization management account or in a member account that is a delegated administrator.

The required AWS APIs are part of the AWS Organizations service, and can only be called from the management account or a member account that is a delegated administrator.

If your Auth Service already has access to an IAM Role, you can attach the following policy to it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "organizations:DescribeAccount"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

If your Auth Service does not have access to an IAM Role, create one with the above policy and attach it to the Auth Service's running instance.

Step 3/7. Allow the Discovery Service to list accounts in the organization and assume roles in the target accounts​

We'll start by creating the required AWS permissions for the Discovery Service.

The Discovery Service must be able to list all the accounts in the AWS Organization. Then, for each account, it will assume a role in order to discover EC2 instances and enroll them in the Teleport cluster.

tip

Discovery Service must be run in the AWS Organization management account or in a member account that is a delegated administrator.

The required AWS APIs are part of the AWS Organizations service, and can only be called from the management account or a member account that is a delegated administrator.

Create a new IAM Role, Discovery Service role's ARN, which will be used by Discovery Service.

Allow the role to list accounts by adding the following IAM Policy:

{
    "Effect": "Allow",
    "Action": [
        "organizations:ListAccountsForParent",
        "organizations:ListChildren",
        "organizations:ListRoots"
    ],
    "Resource": "*"
}

Allow the role to assume other roles in each target account by adding the following IAM Policy:

{
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "*"
}

Step 4/7. Configure IAM Roles in each target account​

Each target AWS account must have an IAM Role which can be assumed by the Discovery Service's IAM role, and which has permissions to discover EC2 instances.

Create an IAM Role named DiscoverEC2AssumedRole in each target AWS account, including the one where the Discovery Service is running, with the following permissions:

{
    "Effect": "Allow",
    "Action": [
        "account:ListRegions",
        "ec2:DescribeInstances",
        "ssm:DescribeInstanceInformation",
        "ssm:GetCommandInvocation",
        "ssm:ListCommandInvocations",
        "ssm:SendCommand"
    ],
    "Resource": "*"
}

Add the following trust relationship, which allows the Discovery Service role's ARN to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "Discovery Service role's ARN"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 5/7. Install Teleport on the Discovery Node​

tip

If you plan on running the Discovery Service on the same Node already running another Teleport service (Auth or Proxy, for example), you can skip this step.

Install Teleport on the EC2 instance that will run the Discovery Service:

To install a Teleport Agent on your Linux server:

The recommended installation method is the cluster install script. It will select the correct version, edition, and installation mode for your cluster.

  1. Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https://).

  2. Run your cluster's install script:

    curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash

Step 6/7. Configure Teleport to discover EC2 instances​

If you are running the Discovery Service on its own host, the service requires a valid invite token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service:

tctl tokens add --type=discovery

Save the generated token in /tmp/token on the Node (EC2 instance) that will run the Discovery Service.

warning

Discovery Service exposes a configuration parameter - discovery_service.discovery_group - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services.

When running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud resources or a different value if they are watching different cloud resources.

It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.

  • 2 Discovery Services configured with discovery_group: "prod" polling data from Production account.
  • 2 Discovery Services configured with discovery_group: "staging" polling data from Staging account.

Assign teleport.example.com:443 to the host and port of the Teleport Proxy Service in your cluster, and aws-prod to a name that identifies a group of resources that you will enroll:

# teleport.yaml
version: v3
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  proxy_server: "teleport.example.com:443"
auth_service:
  enabled: false
proxy_service:
  enabled: false
ssh_service:
  enabled: false
discovery_service:
  enabled: true
  discovery_group: aws-prod

Create a matcher for the resources you want to enroll.

tip

Dynamic configuration uses Discovery Configs which can be managed using Terraform. See the Terraform discovery_config reference for more information.

Static configuration while simpler at first, has less flexibility because enrollment changes require edits to teleport.yaml and the restart of the Discovery Service.

Create a Discovery Config resource, that has the same discovery group you configured earlier, to enable EC2 instance discovery.

Create a file named discovery-aws-prod.yaml with the following content:

kind: discovery_config
version: v1
metadata:
  name: example-discovery-config
spec:
  discovery_group: aws-prod
  aws:
  - types: ["ec2"]
    regions: ["*"]
    ssm:
      document_name: "AWS-RunShellScript"
    install:
      join_method: iam
      join_token: aws-discovery-iam-token
    assume_role:
      role_name: DiscoverEC2AssumedRole
    organization:
      organization_id: o-organisation
      organizational_units:
          # Include is a list of AWS Organizational Unit IDs and children OUs to include.
          # Accounts that belong to these OUs, and their children, will be included.
          # Only exact matches or wildcard (*) are supported.
          # Required.
          include: ["*"]
          # Exclude is a list of AWS Organizational Unit IDs and children OUs to exclude.
          # Accounts that belong to these OUs, and their children, will be excluded, even if they were included.
          # Only exact matches are supported.
          # Optional. If empty, no OUs are excluded.
          exclude: []
      tags:
          "env": "prod" # Match EC2 instances where tag:env=prod

Adjust the keys under spec.aws to match your EC2 environment, specifically the tags you want to associate with the Discovery Service.

Create the Discovery Config by running the following command:

tctl create -f discovery-aws-prod.yaml

You can update the Discovery Config at any time, and the service will automatically re-apply the changes.

Step 7/7. Start Teleport​

Grant the Discovery Service access to credentials that it can use to authenticate to AWS.

  • If you are running the Discovery Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
  • If you are running the Discovery Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
  • Otherwise, you must use environment variables

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Have multiple sources of AWS credentials?

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Discovery Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Discovery Service.

On the host where you will run the Discovery Service, enable and start Teleport:

sudo systemctl enable teleport
sudo systemctl start teleport

You can check the status of the Discovery Service with systemctl status teleport and view its logs with journalctl -fu teleport.

Once you have started the Discovery Service, EC2 instances matching the tags you specified earlier will begin to be added to the Teleport cluster automatically.

Auto-discovery labels​

Teleport applies a set of default labels to resources on AWS, Azure, and Google Cloud that join a cluster via auto-discovery. See the auto-discovery labels reference

Advanced configuration​

This section covers configuration options for discovering and enrolling servers.

Install multiple Teleport agents on the same instance​

When using blue-green deployments or other multiple clusters setups, you might want to access your instances from different clusters.

Teleport supports installing and running multiple agents on the same instance, using a suffixed installation which allows you to isolate each installation.

To configure the Discovery Service to use a suffixed installation, edit the Discovery Config and set the spec.aws.install.suffix key:

kind: discovery_config
# ...
spec:
  aws:
   - install:
       suffix: "blue-cluster"

Requires agent managed updates to be enabled.

Define the group for Managed Updates​

If you are using Teleport Agent managed updates, you can configure the update group so that you can control which instances get updated together.

To set the update group, edit the Discovery Config and set the spec.aws.install.update_group key:

kind: discovery_config
# ...
spec:
  aws:
   - install:
       update_group: "update-group-1"

Configure HTTP Proxy during installation​

For instances which require a proxy to access the installation files, you can configure HTTP Proxy settings in the Discovery Service.

To set the HTTP proxy settings, edit the Discovery Config and set the spec.aws.install.http_proxy_settings key:

kind: discovery_config
# ...
spec:
  aws:
   - install:
       http_proxy_settings:
         https_proxy: http://172.31.5.130:3128
         http_proxy: http://172.31.5.130:3128
         no_proxy: my-local-domain

Use a custom installation script​

To customize an installer, your user must have a role that allows list, create, read and update verbs on the installer resource.

Create a file called installer-manager.yaml with the following content:

kind: role
version: v5
metadata:
  name: installer-manager
spec:
  allow:
    rules:
      - resources: [installer]
        verbs: [list, create, read, update]

Create the role:

tctl create -f installer-manager.yaml
role 'installer-manager' has been created
tip

You can also create and edit roles using the Web UI. Go to Access -> Roles and click Create New Role or pick an existing role to edit.

The preset editor role has the required permissions by default.

To customize the default installer script, execute the following command on your workstation:

tctl edit installer/default-installer

After making the desired changes to the default installer, save and close the file in your text editor.

Multiple installer resources can exist and be specified in the aws.install.script_name section:

Edit the Discovery Config to specify a custom installer script:

kind: discovery_config
# ...
spec:
  aws:
    - types: ["ec2"]
      tags:
       - "env": "prod"
      regions: ["us-west1", "us-east1"]
      install:
        script_name: "default-installer"
      ssm:
        document_name: "AWS-RunShellScript"
    - types: ["ec2"]
      tags:
       - "env": "devel"
      regions: ["us-west1", "us-east1"]
      install:
        script_name: "devel-installer"
      ssm:
        document_name: "AWS-RunShellScript"

The installer resource has the following templating options:

  • {{ .MajorVersion }}: the major version of Teleport to use when installing from the repository.
  • {{ .PublicProxyAddr }}: the public address of the Teleport Proxy Service to connect to.
  • {{ .RepoChannel }}: Optional package repository (apt/yum) channel name. Has format <channel>/<version> e.g. stable/v19. See installation for more details.
  • {{ .AutomaticUpgrades }}: indicates whether Automatic Updates are enabled or disabled. Its value is either true or false. See Automatic Agent Updates for more information.
  • {{ .TeleportPackage }}: the Teleport package to use. Its value is either teleport-ent or teleport depending on whether the cluster is enterprise or not.

These can be used as follows:

kind: installer
metadata:
  name: default-installer
spec:
  script: |
    echo {{ .PublicProxyAddr }}
    echo Teleport-{{ .MajorVersion }}
    echo Repository Channel: {{ .RepoChannel }}
version: v1

Which, when retrieved for installation, will evaluate to a script with the following contents:

echo teleport.example.com
echo Teleport-19.0.0-dev
echo Repository Channel: stable/v19.0.0-dev

The default installer will take the following actions:

  • Add an official Teleport repository to supported Linux distributions.
  • Install Teleport via apt or yum.
  • Generate the Teleport config file and write it to /etc/teleport.yaml.
  • Enable and start the Teleport service.

Use a custom SSM Document​

When executing the installation script on discovered EC2 instances, the Discovery Service uses an SSM document.

The default AWS-RunShellScript SSM document works in most cases and is always available in AWS.

However, if you need to customize the installation process for your environment, you can create a custom SSM Document and configure the Discovery Service to use it during installation.

The custom document's parameters must include env, scriptName and token.

The recommended approach is to use the following document and customize it as needed:

schemaVersion: '2.2'
description: aws:runShellScript
parameters:
  token:
    type: String
    description: "(Required) The Teleport invite token to use when joining the cluster."
  scriptName:
    type: String
    description: "(Required) The Teleport installer script to use when joining the cluster."
  env:
    type: String
    description: "Environment variables exported to the script. Format 'ENV=var FOO=bar'"
    default: "X=$X"
mainSteps:
- action: aws:downloadContent
  name: downloadContent
  inputs:
    sourceType: "HTTP"
    destinationPath: "/tmp/installTeleport.sh"
    sourceInfo:
      url: "https://teleport.example.com:443/webapi/scripts/installer/{{ scriptName }}"
- action: aws:runShellScript
  name: runShellScript
  inputs:
    timeoutSeconds: '300'
    runCommand:
      - export {{ env }}; /bin/sh /tmp/installTeleport.sh "{{ token }}"

Create this document using AWS Systems Manager in each region where you plan to discover instances.

Edit your Discovery Service configuration to use the custom SSM Document, by setting the ssm.document_name key:

# teleport.yaml
version: v3
# ...
discovery_service:
  enabled: true
  aws:
   - ssm:
       document_name: "TeleportDiscoveryInstaller"

Discover instances in all active regions​

The Discovery Service can be configured to scan all active AWS regions for EC2 instances.

Edit the AWS matcher and set the regions key to wildcard (*):

# teleport.yaml
version: v3
# ...
discovery_service:
  enabled: true
  aws:
   - regions: ["*"]
     # other fields

Add the necessary IAM permissions to allow the Discovery Service to list regions:

{
    "Effect": "Allow",
    "Action": [
        // existing permissions
        "account:ListRegions"
    ],
    "Resource": "*"
}

Troubleshooting​

If Installs are showing failed or instances are failing to appear check the Command history in AWS System Manager -> Node Management -> Run Command. Select the instance-id of the Target to review Errors.

cannot unmarshal object into Go struct field​

If you encounter an error similar to the following:

invalid format in plugin properties map[destinationPath:/tmp/installTeleport.sh sourceInfo:map[url:[https://example.teleport.sh:443/webapi/scripts/installer/preprod-installer](https://example.teleport.sh/webapi/scripts/installer/preprod-installer)] sourceType:HTTP];
error json: cannot unmarshal object into Go struct field DownloadContentPlugin.sourceInfo of type string

It is likely that you're running an older SSM agent version. Upgrade to SSM agent version 3.1 or greater to resolve.

InvalidInstanceId: Instances [[i-123]] not in a valid state for account 456​

The following problems can cause this error:

  • The Discovery Service doesn't have permission to access the managed node.
  • AWS Systems Manager Agent (SSM Agent) isn't running. Verify that SSM Agent is running.
  • SSM Agent isn't registered with the SSM endpoint. Try reinstalling SSM Agent.
  • The discovered instance does not have permission to receive SSM commands, verify the instance includes the AmazonSSMManagedInstanceCore IAM policy.

See SSM RunCommand error codes and troubleshooting information in AWS documentation for more details:

Next steps​