Fork me on GitHub

Teleport

Configure Teleport to Automatically Enroll EC2 instances (Preview)

Improve

The Teleport Discovery Service can connect to Amazon EC2 and automatically discover and enroll EC2 instances matching configured labels. It will then execute an install script on these discovered instances using AWS Systems Manager that will install Teleport, start it and join the cluster.

In this scenario, Teleport Discovery Service uses an IAM invite token with a long time-to-live (TTL), so that new instances can be discovered and added to the Teleport cluster for the lifetime of the token.

Prerequisites

  • A running Teleport cluster. For details on how to set this up, see one of our Getting Started guides.

  • The tctl admin tool and tsh client tool version >= 12.1.1.

    tctl version

    Teleport v12.1.1 go1.19

    tsh version

    Teleport v12.1.1 go1.19

    See Installation for details.

  • A running Teleport cluster. For details on how to set this up, see our Enterprise Getting Started guide.

  • The Enterprise tctl admin tool and tsh client tool version >= 12.1.1, which you can download by visiting the customer portal.

    tctl version

    Teleport Enterprise v12.1.1 go1.19

    tsh version

    Teleport v12.1.1 go1.19

Cloud is not available for Teleport v.
Please use the latest version of Teleport Enterprise documentation.
  • AWS account with EC2 instances and permissions to create and attach IAM policies.
  • EC2 instances running Ubuntu/Debian/RHEL/Amazon Linux 2 and SSM agent version 3.1 or greater if making use of the default Teleport install script. (For other Linux distributions, you can install Teleport manually.)

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 12.1.1

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 12.1.1

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Step 1/7. Create an EC2 invite token

When discovering EC2 instances, Teleport makes use of IAM invite tokens for authenticating joining Nodes.

Create a file called token.yaml:

# token.yaml
kind: token
version: v2
metadata:
  # the token name is not a secret because instances must prove that they are
  # running in your AWS account to use this token
  name: aws-discovery-iam-token
  # set a long expiry time, as the default for tokens is only 30 minutes
  expires: "3000-01-01T00:00:00Z"
spec:
  # use the minimal set of roles required
  roles: [Node]

  # set the join method allowed for this token
  join_method: iam

  allow:
  # specify the AWS account which Nodes may join from
  - aws_account: "123456789"

Assign the aws_account field to your AWS account number. Add the token to the Teleport cluster with:

tctl create -f token.yaml

Step 2/7. Define an IAM policy

The teleport discovery bootstrap command will automate the process of defining and implementing IAM policies required to make auto-discovery work. It requires only a single pre-defined policy, attached to the EC2 instance running the command:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetPolicy",
                "iam:TagPolicy",
                "iam:ListPolicyVersions",
                "iam:CreatePolicyVersion",
                "iam:CreatePolicy",
                "ssm:CreateDocument",
                "iam:DeletePolicyVersion",
                "iam:AttachRolePolicy",
                "iam:PutRolePermissionsBoundary"
            ],
            "Resource": "*"
        }
    ]
}

Create this policy and apply it to the Node (EC2 instance) that will run the Discovery Service.

Step 3/7. Install Teleport on the Discovery Node

Tip

If you plan on running the Discovery Service on the same Node already running another Teleport service (Auth or Proxy, for example), you can skip this step.

Install Teleport on the EC2 instance that will run the Discovery Service:

Use the appropriate commands for your environment to install your package.

Teleport Edition

Add the Teleport repository to your repository list:

Download Teleport's PGP public key

sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc

Source variables about OS version

source /etc/os-release

Add the Teleport APT repository for v12. You'll need to update this

file for each major release of Teleport.

Note: if using a fork of Debian or Ubuntu you may need to use '$ID_LIKE'

and the codename your distro was forked from instead of '$ID' and '$VERSION_CODENAME'.

Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L42-L67

echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v12" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport

Source variables about OS version

source /etc/os-release

Add the Teleport YUM repository for v12. You'll need to update this

file for each major release of Teleport.

Note: if using a fork of RHEL/CentOS or Amazon Linux you may need to use '$ID_LIKE'

and the codename your distro was forked from instead of '$ID'

Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L133-L153

sudo yum-config-manager --add-repo $(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v12/teleport.repo")
sudo yum install teleport

Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)

echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path

Optional: Use DNF on newer distributions

$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo

$ sudo dnf install teleport

In the example commands below, update $SYSTEM-ARCH with the appropriate value (amd64, arm64, or arm). All example commands using this variable will update after one is filled out.

curl https://get.gravitational.com/teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://cdn.teleport.dev/teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
shasum -a 256 teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz

Verify that the checksums match

tar -xvf teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
cd teleport
sudo ./install

In the example commands below, update $SYSTEM-ARCH with the appropriate value (amd64, arm64, or arm). All example commands using this variable will update after one is filled out.

After Downloading the .deb file for your system architecture, install it with dpkg. The example below assumes the root user:

dpkg -i ~/Downloads/teleport-ent_12.1.1_$SYSTEM-ARCH.deb

Selecting previously unselected package teleport-ent.

(Reading database ... 30810 files and directories currently installed.)

Preparing to unpack teleport-ent_12.1.1_$SYSTEM_ARCH.deb ...

Unpacking teleport-ent 12.1.1 ...

Setting up teleport-ent 12.1.1 ...

After Downloading the .rpm file for your system architecture, install it with rpm:

rpm -i ~/Downloads/teleport-ent-12.1.1.$SYSTEM-ARCH.rpm

warning: teleport-ent-12.1.1.$SYSTEM-ARCH.rpm: Header V4 RSA/SHA512 Signature, key ID 6282c411: NOKEY

curl https://get.gravitational.com/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://cdn.teleport.dev/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
shasum -a 256 teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz

Verify that the checksums match

tar -xvf teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
cd teleport-ent
sudo ./install

For FedRAMP/FIPS-compliant installations of Teleport Enterprise, package URLs will be slightly different:

curl https://get.gravitational.com/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz.sha256

<checksum> <filename>

curl -O https://cdn.teleport.dev/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz
shasum -a 256 teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz

Verify that the checksums match

tar -xvf teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz
cd teleport-ent
sudo ./install
Cloud is not available for Teleport v.
Please use the latest version of Teleport Enterprise documentation.

Step 4/7. Configure Teleport to discover EC2 instances

If you are running the Discovery Service on its own host, the service requires a valid invite token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service:

tctl tokens add --type=discovery

Save the generated token in /tmp/token on the Node (EC2 instance) that will run the Discovery Service.

In order to enable EC2 instance discovery the discovery_service.aws section of teleport.yaml must include at least one entry:

version: v2
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  auth_servers:
  - "teleport.example.com:3080"
auth_service:
  enabled: off
proxy_service:
  enabled: off
ssh_service:
  enabled: off
discovery_service:
  enabled: "yes"
  aws:
   - types: ["ec2"]
     regions: ["us-east-1","us-west-1"]
     tags:
       "env": "prod" # Match EC2 instances where tag:env=prod
  • Edit the teleport.auth_servers key to match your Auth Service or Proxy Service's URI and port.
  • Adjust the keys under discovery_service.aws to match your EC2 environment, specifically the regions and tags you want to associate with the Discovery Service.

Step 5/7. Bootstrap the Discovery Service AWS configuration

On the same Node as above, run teleport discovery bootstrap. This command will generate and display the additional IAM policies and AWS Systems Manager (SSM) documents required to enable the Discovery Service:

sudo ./teleport discovery bootstrap

Reading configuration at "/etc/teleport.yaml"...

AWS

1. Create IAM Policy "TeleportEC2Discovery":

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"ec2:DescribeInstances",

"ssm:GetCommandInvocation",

"ssm:SendCommand"

],

"Resource": [

"*"

]

}

]

}

2. Create IAM Policy "TeleportEC2DiscoveryBoundary":

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"ec2:DescribeInstances",

"ssm:GetCommandInvocation",

"ssm:SendCommand"

],

"Resource": [

"*"

]

}

]

}

3. Create SSM Document "TeleportDiscoveryInstaller":

schemaVersion: '2.2'

description: aws:runShellScript

parameters:

token:

type: String

description: "(Required) The Teleport invite token to use when joining the cluster."

scriptName:

type: String

description: "(Required) The Teleport installer script to use when joining the cluster."

mainSteps:

- action: aws:downloadContent

name: downloadContent

inputs:

sourceType: "HTTP"

destinationPath: "/tmp/installTeleport.sh"

sourceInfo:

url: "https://teleport.example.com:443/webapi/scripts/installer/{{ scriptName }}"

- action: aws:runShellScript

name: runShellScript

inputs:

timeoutSeconds: '300'

runCommand:

- /bin/sh /tmp/installTeleport.sh "{{ token }}"

4. Attach IAM policies to "yourUser-discovery-role".

Confirm? [y/N]: y

Review the policies and confirm:

Confirm? [y/N]: y

✅[AWS] Create IAM Policy "TeleportEC2Discovery"... done.

✅[AWS] Create IAM Policy "TeleportEC2DiscoveryBoundary"... done.

✅[AWS] Create IAM SSM Document "TeleportDiscoveryInstaller"... done.

✅[AWS] Attach IAM policies to "alex-discovery-role"... done.

All EC2 instances that are to be added to the Teleport cluster by the Discovery Service must include the AmazonSSMManagedInstanceCore IAM policy in order to receive commands from the Discovery Service.

This policy includes the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeAssociation",
                "ssm:GetDeployablePatchSnapshotForInstance",
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetManifest",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:ListAssociations",
                "ssm:ListInstanceAssociations",
                "ssm:PutInventory",
                "ssm:PutComplianceItems",
                "ssm:PutConfigurePackageResult",
                "ssm:UpdateAssociationStatus",
                "ssm:UpdateInstanceAssociationStatus",
                "ssm:UpdateInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        }
    ]
}

Step 6/7. [Optional] Customize the default installer script

To customize the default installer script, execute the following command on your workstation:

tctl get installer/default-installer > teleport-default-installer.yaml

The resulting teleport-default-installer.yaml can the be edited to change what gets executed when enrolling discovered EC2 instances.

After making the desired changes to the default installer, the resource can be updated by executing:

tctl create -f teleport-default-installer.yaml

Multiple installer resources can exist and be specified in the aws.install.script_name section of a discovery_service.aws list item in teleport.yaml:

discovery_service:
  aws:
    - types: ["ec2"] 
      tags:
       - "env": "prod"
      install: # optional section when default-installer is used.
        script_name: "default-installer"
    - types: ["ec2"] 
      tags:
       - "env": "devel"
      install:
        script_name: "devel-installer"

The installer resource has the following templating options:

  • {{ .MajorVersion }}: the major version of Teleport to use when installing from the repository.
  • {{ .PublicProxyAddr }}: the public address of the Teleport Proxy Service to connect to.

These can be used as follows:

kind: installer
metadata:
  name: default-installer
spec:
  script: |
    echo {{ .PublicProxyAddr }}
    echo Teleport-{{ .MajorVersion }}
version: v1

Which, when retrieved by the Systems Manager document will evaluate to a script with the following contents:

echo teleport.example.com
echo Teleport-12.1.1

The default installer will take the following actions:

  • Add an official Teleport repository to supported Linux distributions.
  • Install Teleport via apt or yum.
  • Generate the Teleport config file and write it to /etc/teleport.yaml.
  • Enable and start the Teleport service.

Step 7/7. Start Teleport

Grant the Discovery Service access to credentials that it can use to authenticate to AWS. If you are running the Discovery Service on an EC2 instance, you should use the EC2 Instance Metadata Service method. Otherwise, you must use environment variables:

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

Teleport's built-in AWS client reads credentials from the following environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION

When you start the Discovery Service, the service reads environment variables from a file at the path /etc/default/teleport. Obtain these credentials from your organization. Ensure that /etc/default/teleport has the following content, replacing the values of each variable:

AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Discovery Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Discovery Service.

On the host where you will run the Discovery Service, start Teleport:

sudo systemctl start teleport

On the host where you will run the Discovery Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:

sudo teleport install systemd -o /etc/systemd/system/teleport.service
sudo systemctl enable teleport
sudo systemctl start teleport

Once you have started the Discovery Service, EC2 instances matching the tags you specified earlier will begin to be added to the Teleport cluster automatically.

Troubleshooting

If Installs are showing failed or instances are failing to appear check the Command history in AWS System Manager -> Node Management -> Run Command. Select the instance-id of the Target to review Errors.

Next steps