# Server Auto-Discovery for Amazon EC2 This guide shows you how to configure Teleport to automatically enroll EC2 instances in your cluster. ## How it works In the setup we describe in this guide, the Teleport Discovery Service connects to Amazon EC2 and reconciles the servers enrolled on the Auth Service backend with servers it lists from the EC2 API. If an EC2 instance matches a configured label and is not enrolled in your cluster, the Discovery Service executes a script on these discovered instances using AWS Systems Manager that installs Teleport, starts it and joins the cluster using the [IAM join method](https://goteleport.com/docs/ver/17.x/enroll-resources/agents/aws-iam.md). The Teleport Discovery Service uses an IAM invite token with a long time-to-live (TTL), so that new instances can be discovered and added to the Teleport cluster for the lifetime of the token. ## Prerequisites - A running Teleport cluster. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/ver/17.x/get-started/deploy-community.md). - The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/ver/17.x/installation.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` * AWS account with EC2 instances and permissions to create and attach IAM policies. * EC2 instances running Ubuntu/Debian/RHEL/Amazon Linux 2/Amazon Linux 2023 and SSM agent version 3.1 or greater if making use of the default Teleport install script. (For other Linux distributions, you can install Teleport manually.) * To check that you can connect to your Teleport cluster, sign in with `tsh login`, then verify that you can run `tctl` commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and email\@example.com to your Teleport username: ``` $ tsh login --proxy=teleport.example.com --user=email@example.com $ tctl status Cluster teleport.example.com Version 17.7.20 CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678 ``` If you can connect to the cluster and run the `tctl status` command, you can use your current credentials to run subsequent `tctl` commands from your workstation. If you host your own Teleport cluster, you can also run `tctl` commands on the computer that hosts the Teleport Auth Service for full permissions. ## Step 1/7. Create an EC2 invite token When discovering EC2 instances, Teleport makes use of IAM invite tokens for authenticating joining Nodes. Create a file called `token.yaml`: ``` # token.yaml kind: token version: v2 metadata: # the token name is not a secret because instances must prove that they are # running in your AWS account to use this token name: aws-discovery-iam-token spec: # use the minimal set of roles required (e.g. Node, App, Kube, DB, WindowsDesktop) roles: [Node] # set the join method allowed for this token join_method: iam allow: # specify the AWS account which Nodes may join from - aws_account: "123456789" ``` Assign the `aws_account` field to your AWS account number. Add the token to the Teleport cluster with: ``` $ tctl create -f token.yaml ``` ## Step 2/7. Define an IAM policy The `teleport discovery bootstrap` command will automate the process of defining and implementing IAM policies required to make auto-discovery work. It requires only a single pre-defined policy, attached to the EC2 instance running the command: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:GetPolicy", "iam:TagPolicy", "iam:ListPolicyVersions", "iam:CreatePolicyVersion", "iam:CreatePolicy", "ssm:CreateDocument", "iam:DeletePolicyVersion", "iam:AttachRolePolicy", "iam:PutRolePermissionsBoundary" ], "Resource": "*" } ] } ``` Create this policy and apply it to the Node (EC2 instance) that will run the Discovery Service. ## Step 3/7. Install Teleport on the Discovery Node --- TIP If you plan on running the Discovery Service on the same Node already running another Teleport service (Auth or Proxy, for example), you can skip this step. --- Install Teleport on the EC2 instance that will run the Discovery Service: To install a Teleport Agent on your Linux server: The easiest installation method, for *Teleport versions 17.3 and above*, is the cluster install script. It will use the best version, edition, and installation mode for your cluster. 1. Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https\://). 2. Run your cluster's install script: ``` $ curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash ``` On *older Teleport versions*: 1. Assign edition to one of the following, depending on your Teleport edition: | Edition | Value | | --------------------------------- | ------------ | | Teleport Enterprise Cloud | `cloud` | | Teleport Enterprise (Self-Hosted) | `enterprise` | | Teleport Community Edition | `oss` | 2. Get the version of Teleport to install. If you have automatic agent updates enabled in your cluster, query the latest Teleport version that is compatible with the updater: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/automaticupgrades/channel/default/version | sed 's/v//')" ``` Otherwise, get the version of your Teleport cluster: ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/ping | jq -r '.server_version')" ``` 3. Install Teleport on your Linux server: ``` $ curl https://cdn.teleport.dev/install.sh | bash -s ${TELEPORT_VERSION} edition ``` The installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the [installation guide](https://goteleport.com/docs/ver/17.x/installation/linux.md). ## Step 4/7. Configure Teleport to discover EC2 instances If you are running the Discovery Service on its own host, the service requires a valid invite token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service: ``` $ tctl tokens add --type=discovery ``` Save the generated token in `/tmp/token` on the Node (EC2 instance) that will run the Discovery Service. In order to enable EC2 instance discovery the `discovery_service.aws` section of `teleport.yaml` must include at least one entry: --- WARNING Discovery Service exposes a configuration parameter - `discovery_service.discovery_group` - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services. When running multiple Discovery Services, you must ensure that each service is configured with the same `discovery_group` value if they are watching the same cloud resources or a different value if they are watching different cloud resources. It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration. - 2 Discovery Services configured with `discovery_group: "prod"` polling data from Production account. - 2 Discovery Services configured with `discovery_group: "staging"` polling data from Staging account. --- ``` version: v3 teleport: join_params: token_name: "/tmp/token" method: token proxy_server: "teleport.example.com:443" auth_service: enabled: false proxy_service: enabled: false ssh_service: enabled: false discovery_service: enabled: true discovery_group: "aws-prod" aws: - types: ["ec2"] regions: ["us-east-1","us-west-1"] install: join_params: token_name: aws-discovery-iam-token method: iam tags: "env": "prod" # Match EC2 instances where tag:env=prod ``` - Edit the `teleport.auth_servers` key to match your Auth Service or Proxy Service's URI and port. - Adjust the keys under `discovery_service.aws` to match your EC2 environment, specifically the regions and tags you want to associate with the Discovery Service. ## Step 5/7. Bootstrap the Discovery Service AWS configuration On the same Node as above, run `teleport discovery bootstrap`. This command will generate and display the additional IAM policies and AWS Systems Manager (SSM) documents required to enable the Discovery Service: ``` $ sudo teleport discovery bootstrap Reading configuration at "/etc/teleport.yaml"... AWS 1. Create IAM Policy "TeleportEC2Discovery": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ssm:DescribeInstanceInformation", "ssm:GetCommandInvocation", "ssm:ListCommandInvocations", "ssm:SendCommand" ], "Resource": [ "*" ] } ] } 2. Create IAM Policy "TeleportEC2DiscoveryBoundary": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ssm:DescribeInstanceInformation", "ssm:GetCommandInvocation", "ssm:ListCommandInvocations", "ssm:SendCommand" ], "Resource": [ "*" ] } ] } 3. Create SSM Document "TeleportDiscoveryInstaller": schemaVersion: '2.2' description: aws:runShellScript parameters: token: type: String description: "(Required) The Teleport invite token to use when joining the cluster." scriptName: type: String description: "(Required) The Teleport installer script to use when joining the cluster." env: type: String description: "Environment variables exported to the script. Format 'ENV=var FOO=bar'" default: "X=$X" mainSteps: - action: aws:downloadContent name: downloadContent inputs: sourceType: "HTTP" destinationPath: "/tmp/installTeleport.sh" sourceInfo: url: "https://teleport.example.com:443/webapi/scripts/installer/{{ scriptName }}" - action: aws:runShellScript name: runShellScript inputs: timeoutSeconds: '300' runCommand: - /bin/sh /tmp/installTeleport.sh "{{ token }}" 4. Attach IAM policies to "yourUser-discovery-role". Confirm? [y/N]: y ``` Review the policies and confirm: ``` Confirm? [y/N]: y ✅[AWS] Create IAM Policy "TeleportEC2Discovery"... done. ✅[AWS] Create IAM Policy "TeleportEC2DiscoveryBoundary"... done. ✅[AWS] Create IAM SSM Document "TeleportDiscoveryInstaller"... done. ✅[AWS] Attach IAM policies to "alex-discovery-role"... done. ``` All EC2 instances that are to be added to the Teleport cluster by the Discovery Service must include the `AmazonSSMManagedInstanceCore` IAM policy in order to receive commands from the Discovery Service. This policy includes the following permissions: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ssm:DescribeAssociation", "ssm:GetDeployablePatchSnapshotForInstance", "ssm:GetDocument", "ssm:DescribeDocument", "ssm:GetManifest", "ssm:GetParameter", "ssm:GetParameters", "ssm:ListAssociations", "ssm:ListInstanceAssociations", "ssm:PutInventory", "ssm:PutComplianceItems", "ssm:PutConfigurePackageResult", "ssm:UpdateAssociationStatus", "ssm:UpdateInstanceAssociationStatus", "ssm:UpdateInstanceInformation" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ssmmessages:CreateControlChannel", "ssmmessages:CreateDataChannel", "ssmmessages:OpenControlChannel", "ssmmessages:OpenDataChannel" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ec2messages:AcknowledgeMessage", "ec2messages:DeleteMessage", "ec2messages:FailMessage", "ec2messages:GetEndpoint", "ec2messages:GetMessages", "ec2messages:SendReply" ], "Resource": "*" } ] } ``` ## Step 6/7. \[Optional] Customize the default installer script To customize an installer, your user must have a role that allows `list`, `create`, `read` and `update` verbs on the `installer` resource. Create a file called `installer-manager.yaml` with the following content: ``` kind: role version: v5 metadata: name: installer-manager spec: allow: rules: - resources: [installer] verbs: [list, create, read, update] ``` Create the role: ``` $ tctl create -f installer-manager.yaml role 'installer-manager' has been created ``` --- TIP You can also create and edit roles using the Web UI. Go to **Access -> Roles** and click **Create New Role** or pick an existing role to edit. --- The preset `editor` role has the required permissions by default. To customize the default installer script, execute the following command on your workstation: ``` $ tctl edit installer/default-installer ``` After making the desired changes to the default installer, save and close the file in your text editor. Multiple `installer` resources can exist and be specified in the `aws.install.script_name` section of a `discovery_service.aws` list item in `teleport.yaml`: ``` discovery_service: # ... aws: - types: ["ec2"] tags: - "env": "prod" install: # optional section when default-installer is used. script_name: "default-installer" - types: ["ec2"] tags: - "env": "devel" install: script_name: "devel-installer" ``` --- The `installer` resource has the following templating options: - `{{ .MajorVersion }}`: the major version of Teleport to use when installing from the repository. - `{{ .PublicProxyAddr }}`: the public address of the Teleport Proxy Service to connect to. - `{{ .RepoChannel }}`: Optional package repository (apt/yum) channel name. Has format `/` e.g. stable/v17. See [installation](https://goteleport.com/docs/ver/17.x/installation/linux.md) for more details. - `{{ .AutomaticUpgrades }}`: indicates whether Automatic Updates are enabled or disabled. Its value is either `true` or `false`. See [Automatic Agent Updates](https://goteleport.com/docs/ver/17.x/upgrading/agent-managed-updates.md) for more information. - `{{ .TeleportPackage }}`: the Teleport package to use. Its value is either `teleport-ent` or `teleport` depending on whether the cluster is enterprise or not. These can be used as follows: ``` kind: installer metadata: name: default-installer spec: script: | echo {{ .PublicProxyAddr }} echo Teleport-{{ .MajorVersion }} echo Repository Channel: {{ .RepoChannel }} version: v1 ``` Which, when retrieved for installation, will evaluate to a script with the following contents: ``` echo teleport.example.com echo Teleport-17.7.20 echo Repository Channel: stable/v17.7.20 ``` The default installer will take the following actions: - Add an official Teleport repository to supported Linux distributions. - Install Teleport via `apt` or `yum`. - Generate the Teleport config file and write it to `/etc/teleport.yaml`. - Enable and start the Teleport service. ## Step 7/7. Start Teleport Grant the Discovery Service access to credentials that it can use to authenticate to AWS. - If you are running the Discovery Service on an EC2 instance, you may use the EC2 Instance Metadata Service method - If you are running the Discovery Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA) - Otherwise, you must use environment variables **Instance Metadata Service** Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials. The EC2 instance should be configured to use an EC2 instance profile. For more information, see: [Using Instance Profiles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html). **Kubernetes IRSA** Refer to [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role. **Environment Variables** Teleport's built-in AWS client reads credentials from the following environment variables: - `AWS_ACCESS_KEY_ID` - `AWS_SECRET_ACCESS_KEY` - `AWS_DEFAULT_REGION` When you start the Discovery Service, the service reads environment variables from a file at the path `/etc/default/teleport`. Obtain these credentials from your organization. Ensure that `/etc/default/teleport` has the following content, replacing the values of each variable: ``` AWS_ACCESS_KEY_ID=00000000000000000000 AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000 AWS_DEFAULT_REGION= ``` Have multiple sources of AWS credentials? Teleport's AWS client loads credentials from different sources in the following order: - Environment Variables - Shared credentials file - Shared configuration file (Teleport always enables shared configuration) - EC2 Instance Metadata (credentials only) While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Discovery Service with the `AWS_PROFILE` environment variable assigned to the name of your profile of choice. If you have a specific use case that the instructions above do not account for, consult the documentation for the [AWS SDK for Go](https://docs.aws.amazon.com/sdk-for-go/api/aws/session/) for a detailed description of credential loading behavior. Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Discovery Service. **Package Manager** On the host where you will run the Discovery Service, enable and start Teleport: ``` $ sudo systemctl enable teleport $ sudo systemctl start teleport ``` **TAR Archive** On the host where you will run the Discovery Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport: ``` $ sudo teleport install systemd -o /etc/systemd/system/teleport.service $ sudo systemctl enable teleport $ sudo systemctl start teleport ``` You can check the status of the Discovery Service with `systemctl status teleport` and view its logs with `journalctl -fu teleport`. Once you have started the Discovery Service, EC2 instances matching the tags you specified earlier will begin to be added to the Teleport cluster automatically. ## Troubleshooting If Installs are showing failed or instances are failing to appear check the Command history in AWS System Manager -> Node Management -> Run Command. Select the instance-id of the Target to review Errors. ### `cannot unmarshal object into Go struct field` If you encounter an error similar to the following: ``` invalid format in plugin properties map[destinationPath:/tmp/installTeleport.sh sourceInfo:map[url:[https://example.teleport.sh:443/webapi/scripts/installer/preprod-installer](https://example.teleport.sh/webapi/scripts/installer/preprod-installer)] sourceType:HTTP]; error json: cannot unmarshal object into Go struct field DownloadContentPlugin.sourceInfo of type string ``` It is likely that you're running an older SSM agent version. Upgrade to SSM agent version 3.1 or greater to resolve. ### `InvalidInstanceId: Instances [[i-123]] not in a valid state for account 456` The following problems can cause this error: - The Discovery Service doesn't have permission to access the managed node. - AWS Systems Manager Agent (SSM Agent) isn't running. Verify that SSM Agent is running. - SSM Agent isn't registered with the SSM endpoint. Try reinstalling SSM Agent. - The discovered instance does not have permission to receive SSM commands, verify the instance includes the AmazonSSMManagedInstanceCore IAM policy. See SSM RunCommand error codes and troubleshooting information in AWS documentation for more details: - - [https://docs.aws.amazon.com/systems-manager/latest/APIReference/API\_SendCommand.html#API\_SendCommand\_Errors](https://docs.aws.amazon.com/systems-manager/latest/apireference/api_sendcommand.html#api_sendcommand_errors) ## Next steps - Read [Joining Nodes via AWS IAM Role](https://goteleport.com/docs/ver/17.x/enroll-resources/agents/aws-iam.md) for more information on IAM Invite Tokens. - Information on IAM best practices on EC2 instances managed by Systems Manager can be found for in the [AWS Cloud Operations & Migrations Blog ](https://aws.amazon.com/blogs/mt/applying-managed-instance-policy-best-practices/). - Full documentation on EC2 discovery configuration can be found through the [config file reference documentation](https://goteleport.com/docs/ver/17.x/reference/config.md).