Access AWS With Teleport Application Access

You can protect the AWS Management Console and AWS APIs with Teleport. This makes it possible to manage access to AWS infrastructure with Teleport features like Access Requests, Access Lists, and identity locking. You can also configure Teleport to provide different levels of AWS access automatically when users authenticate using your single sign-on solution.

This guide will explain how to:

• Access the AWS Management Console through Teleport.
• Access the AWS Command Line Interface (CLI) through Teleport.
• Access applications using AWS SDKs through Teleport.

In this setup, the Teleport Application Service has an AWS IAM role that can assume one or more target IAM roles. Teleport users access the AWS Management Console and APIs through the Teleport Web UI and tsh. When a user visits the AWS Management Console or executes a command with an AWS client application, the Teleport Application Service checks the user's RBAC permissions and, if they are authorized, forwards the user's requests to AWS.

Prerequisites

• A running Teleport cluster version 16.4.8 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

• The tctl admin tool and tsh client tool.

  Visit Installation for instructions on downloading tctl and tsh.

• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials.

  For example:

  tsh login --proxy=teleport.example.com --user=[email protected]
  tctl status
  Cluster  teleport.example.com
  Version  16.4.8
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

• An AWS EC2 instance or Elastic Kubernetes Service (EKS) cluster where you will run the Teleport Application Service. EC2 instances must be running a Linux distribution. We recommend starting with a fresh demo instance or EKS cluster to get familiar with the procedure before following this guide in production.

• Permissions to create IAM roles and policies in the AWS account you want to connect.

• aws command line interface (CLI) tool in PATH. Read the AWS documentation to install or update the latest version of the AWS CLI.

• If you plan to run the Teleport Application Service on EKS, you will need an IAM OIDC provider running in your Kubernetes cluster. See the AWS documentation for how to create an IAM OIDC provider.

  To check whether you have an IAM OIDC provider running in your cluster, run the following aws command, assigning eks-region to the region where your EKS cluster is running and cluster-name to the name of your Kubernetes cluster:

  aws --region=eks-region eks describe-cluster --name cluster-name --query "cluster.identity.oidc.issuer" --output text

  If you have an IAM OIDC provider associated with your cluster, this command will print its ID.

Step 1/4. Configure AWS IAM

In this section, you will configure AWS IAM resources to allow the Teleport Application Service to proxy AWS APIs.

The rest of the guide assumes that you are protecting access to an IAM role called ExampleReadOnlyAccess, which allows read-only access to AWS resources through the AWS-managed ReadOnlyAccess policy. We recommend following this guide in a demo environment with the ExampleReadOnlyAccess role, and protecting production AWS roles when you are familiar with the procedure.

You will create the following resources:

Create a role for the Teleport Application Service

In this section, you will create an IAM role that allows the Teleport Application Service to assume other IAM roles in order to proxy user traffic to AWS APIs.

1. Define a trust policy to enable the Teleport Application Service to assume the role you will create.

   • EC2
   • EKS

   Create a file called app-service-tp.json:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow", 
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

   Retrieve your OIDC issuer ID, assigning cluster-name to the name of your EKS cluster and eks-region to the AWS region where your EKS cluster is running:

   aws --region=eks-region eks describe-cluster --name cluster-name --query "cluster.identity.oidc.issuer" --output text | grep -Eo "[A-Z0-9]+$"
   
   AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

   Create a file called app-service-tp.json with the following content, assigning oidc-issuer to the issuer string you retrieved and EKS_ACCOUNT to the ID of the AWS account that belongs to your EKS cluster:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::EKS_ACCOUNT:oidc-provider/oidc.eks.EKS_REGION.amazonaws.com/id/OIDC_ISSUER"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "oidc.eks.EKS_REGION.amazonaws.com/id/OIDC_ISSUER:aud": "sts.amazonaws.com",
                        "oidc.eks.EKS_REGION.amazonaws.com/id/OIDC_ISSUER:sub": "system:serviceaccount:teleport-agent:teleport-kube-agent"
   
                   }
               }
           }
       ]
   }
   

   Later in this guide, we will install a Helm chart to deploy the Teleport Application Service. The chart assigns the Teleport pod to a service account called teleport-kube-agent. This trust policy authorizes workloads with this service account in the teleport-agent namespace to assume the IAM role we create in this section.

2. Create a role for the Teleport Application Service:

   aws iam create-role --role-name "TeleportAWSAccess" \--assume-role-policy-document file://app-service-tp.json

3. Get the ARN of the ExampleReadOnlyAccess role:

   ROLE_ARN=$(aws iam get-role --role-name ExampleReadOnlyAccess --query "Role.Arn" --output text)

4. Define an IAM policy to allow the Teleport Application Service to assume the ExampleReadOnlyAccess role:

   cat<<EOF > teleport-assume-role.json{ "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "sts:AssumeRole",      "Resource": "${ROLE_ARN}"    }  ]}EOF
   
   POLICY_ARN=$(aws iam create-policy --policy-name=AssumeExampleReadOnlyRole \--policy-document file://teleport-assume-role.json | jq -r '.Policy.Arn')
   aws iam attach-role-policy --role-name TeleportAWSAccess \--policy-arn ${POLICY_ARN}

Configure a role for Teleport users to request

In this section, you will create a role that Teleport users can request access to when making requests to AWS APIs. The Teleport Application Service assumes this role when proxying requests:

1. Obtain AWS credentials for the account where you will run the Teleport Application Service and make them available to your terminal shell.

2. Create a trust policy document, which authorizes an entity to assume the role you want to protect access to. To do so, create a file called ro-access.json with the following content, replacing AWS_ACCESS_ACCOUNT with the ID of the AWS account where you will run the Teleport Application Service:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::AWS_ACCESS_ACCOUNT:role/TeleportAWSAccess"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

   In the setup we show in this guide, the Teleport Application Service assumes the TeleportAWSAccess role, then uses that role to assume the ExampleReadOnlyAccess role. With the trust policy above, AWS authorizes this operation.

   Cross-account access

   If you are configuring the Application Service to proxy access to IAM roles in another AWS account, we recommend checking the external ID of the AWS account where the Application Service runs. Add the external ID to the trust policy as follows, assigning EXTERNAL_ID to the external ID:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::AWS_ACCESS_ACCOUNT:role/TeleportAWSAccess"
         },
         "Action": "sts:AssumeRole",
         "Condition": { 
           "StringEquals": { 
             "sts:ExternalId": "EXTERNAL_ID" 
           } 
         }
       }
     ]
   }
   

   See the AWS documentation for details on external IDs.

3. Run the following commands to create the ExampleReadOnlyAccess role:

   aws iam create-role --role-name "ExampleReadOnlyAccess" \--assume-role-policy-document file://ro-access.json

4. Get the ARN of the AWS-managed ReadOnlyAccess policy so you can attach it to your role:

   ARN=$(aws iam list-policies --output text --query "Policies[?PolicyName=='ReadOnlyAccess'].Arn")

5. Attach the ReadOnlyAccess policy to the role:

   aws iam attach-role-policy --role-name ExampleReadOnlyAccess --policy-arn $ARN

Associate a role with the Teleport Application Service

Now that you have created a role for the Teleport Application Service, associate the role with the service.

kubectl annotate serviceaccount \-n teleport-agent \teleport-kube-agent eks.amazonaws.com/role-arn=arn:aws:iam::EKS_ACCOUNT:role/TeleportAWSAccess

Step 2/4. Configure Teleport IAM role mapping

In this step, you will define a Teleport role that confers access to the ExampleReadOnlyAccess IAM role you created in the previous step.

1. Create a file called aws-ro-access.yaml with the following content, replacing AWS_ACCOUNT with the ID of the account where you created the ExampleReadOnlyAccess role:

   kind: role
   version: v5
   metadata:
     name: aws-ro-access
   spec:
     allow:
       app_labels:
         '*': '*'
       aws_role_arns:
       - arn:aws:iam::AWS_ACCOUNT:role/ExampleReadOnlyAccess
   

   The Teleport role called aws-ro-access enables users to visit the AWS Management Console or make requests to AWS APIs with the ExampleReadOnlyAccess role.

2. Create the role:

   tctl create aws-ro-access.yaml

3. Assign the aws-ro-access role to your Teleport user by running the appropriate commands for your authentication provider:

   • Local User
   • GitHub
   • SAML
   • OIDC

   1. Retrieve your local user's roles as a comma-separated list:

      ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')

   2. Edit your local user to add the new role:

      tctl users update $(tsh status -f json | jq -r '.active.username') \  --set-roles "${ROLES?},aws-ro-access"

   3. Sign out of the Teleport cluster and sign in again to assume the new role.

   1. Open your github authentication connector in a text editor:

      tctl edit github/github

   2. Edit the github connector, adding aws-ro-access to the teams_to_roles section.

      The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.

      Here is an example:

        teams_to_roles:
          - organization: octocats
            team: admins
            roles:
              - access
      +       - aws-ro-access
      

   3. Apply your changes by saving closing the file in your editor.

   4. Sign out of the Teleport cluster and sign in again to assume the new role.

   1. Retrieve your saml configuration resource:

      tctl get --with-secrets saml/mysaml > saml.yaml

      Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the saml.yaml file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource.

   2. Edit saml.yaml, adding aws-ro-access to the attributes_to_roles section.

      The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

      Here is an example:

        attributes_to_roles:
          - name: "groups"
            value: "my-group"
            roles:
              - access
      +       - aws-ro-access
      

   3. Apply your changes:

      tctl create -f saml.yaml

   4. Sign out of the Teleport cluster and sign in again to assume the new role.

   1. Retrieve your oidc configuration resource:

      tctl get oidc/myoidc --with-secrets > oidc.yaml

      Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the oidc.yaml file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource.

   2. Edit oidc.yaml, adding aws-ro-access to the claims_to_roles section.

      The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

      Here is an example:

        claims_to_roles:
          - name: "groups"
            value: "my-group"
            roles:
              - access
      +       - aws-ro-access
      

   3. Apply your changes:

      tctl create -f oidc.yaml

   4. Sign out of the Teleport cluster and sign in again to assume the new role.

Step 3/4. Set up the Teleport Application Service

Now that you have configured RBAC resources in AWS and Teleport, the remaining step is to launch the Teleport Application Service.

Get a join token

Establish trust between your Teleport cluster and your new Application Service instance by creating a join token:

tctl tokens add --type=app --ttl=1h --format=text
abcd123-insecure-do-not-use-this

echo join-token | sudo tee /tmp/token

Note that the URI you configure for your AWS app must start with one of the following values in order to be recognized as an AWS console:

app_service:
  enabled: "yes"
  apps:
  - name: "awsconsole-test"
    uri: "https://console.aws.amazon.com/ec2/v2/home"
    labels:
      aws_account_id: "1234567890"
      env: test
  - name: "awsconsole-prod"
    uri: "https://console.aws.amazon.com/ec2/v2/home"
    labels:
      aws_account_id: "0987654321"
      env: prod
  - name: "awsconsole-third-party"
    uri: "https://console.aws.amazon.com/ec2/v2/home"
    labels:
      aws_account_id: "1234554321"
    aws:
      external_id: "example-external-id"

Start the Teleport Application Service

If you deployed the Teleport Application Service on Kubernetes, it will have already started, and you can skip to Step 4.

1. Grant the Application Service access to credentials that it can use to authenticate to AWS.
   • If you are running the Application Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
   • If you are running the Application Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
   • Otherwise, you must use environment variables
   • Instance Metadata Service
   • Kubernetes IRSA
   • Environment Variables

   Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

   The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

   Refer to IAM Roles for Service Accounts (IRSA) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role.

   Teleport's built-in AWS client reads credentials from the following environment variables:

   • AWS_ACCESS_KEY_ID
   • AWS_SECRET_ACCESS_KEY
   • AWS_DEFAULT_REGION

   When you start the Application Service, the service reads environment variables from a file at the path /etc/default/teleport. Obtain these credentials from your organization. Ensure that /etc/default/teleport has the following content, replacing the values of each variable:

   AWS_ACCESS_KEY_ID=00000000000000000000
   AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
   AWS_DEFAULT_REGION=<YOUR_REGION>
   

   Have multiple sources of AWS credentials?

   Teleport's AWS client loads credentials from different sources in the following order:

   • Environment Variables
   • Shared credentials file
   • Shared configuration file (Teleport always enables shared configuration)
   • EC2 Instance Metadata (credentials only)

   While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Application Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

   If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

2. Configure the Application Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Application Service.
   • Package Manager
   • TAR Archive

   On the host where you will run the Application Service, enable and start Teleport:

   sudo systemctl enable teleport
   sudo systemctl start teleport

   On the host where you will run the Application Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:

   sudo teleport install systemd -o /etc/systemd/system/teleport.service
   sudo systemctl enable teleport
   sudo systemctl start teleport

   You can check the status of the Application Service with systemctl status teleport and view its logs with journalctl -fu teleport.

Step 4/4. Access AWS resources

Now that you have configured the Teleport Application Service to proxy requests to AWS, users can access AWS resources through Teleport.

Access the AWS console

1. Visit the home page of the Teleport Web UI or click the Resources tab. If the Teleport Application Service is proxying the AWS Management Console as expected, the Web UI will display the name of the application you registered, which this guide assumes is awsconsole. (Enter the name of the application into the search box if there are too many resources to view all of them at once.)

2. Click the Launch button for the AWS Console application, then click on the role you would like to assume when signing in to the AWS Console:

   

3. You will get redirected to the AWS Management Console, signed in with the selected role. You should see your Teleport user name as a federated login assigned to ExampleReadOnlyRole in the top-right corner of the AWS Console:

   

Access the AWS CLI

1. On your desktop, log into the AWS Management Console app you configured while following this guide:

   tsh apps login --aws-role ExampleReadOnlyAccess awsconsole
   Logged into AWS app "awsconsole".
   Your IAM role:  arn:aws:iam::000000000000:role/ExampleReadOnlyAccess
   Example AWS CLI command:  tsh aws s3 ls
   Or start a local proxy:  tsh proxy aws --app awsconsole

   The --aws-role flag allows you to specify the AWS IAM role to assume when accessing AWS APIs. You can either provide a role name like --aws-role ExampleReadOnlyAccess or a full role ARN like arn:aws:iam::0000000000:role/ExampleReadOnlyAccess.

2. Now you can use the tsh aws command like the native aws command-line tool:

   tsh aws s3 ls

3. Run the following command to log out of the AWS application and remove credentials:

   tsh apps logout awsconsole

Access applications using AWS SDKs

1. On your desktop, log into the AWS Management Console app you configured while following this guide:

   tsh apps login --aws-role ExampleReadOnlyAccess awsconsole

2. Open a new terminal and use the following command to start a local HTTPS proxy server that forwards AWS API traffic to the Teleport Application Service. Leave the terminal open since it runs in the foreground:

   tsh proxy aws -p 23456
   Started AWS proxy on http://127.0.0.1:23456.
   Use the following credentials and HTTPS proxy setting to connect to the proxy:  export AWS_ACCESS_KEY_ID=abcd1234-this-is-an-example  export AWS_SECRET_ACCESS_KEY=zyxw9876-this-is-an-example  export AWS_CA_BUNDLE=<ca-bundle-path>  export HTTPS_PROXY=http://127.0.0.1:23456

3. Copy the export commands from the terminal and paste them into a new terminal window.

4. You can then run applications that use AWS SDKs to communicate with AWS APIs through the local proxy. For example, open a Python console and use the boto3 client to fetch EC2 instances in your AWS account:

   >>> import boto3
   >>> ec2 = boto3.resource('ec2')
   >>> for instance in ec2.instances.all():
   ...   print(instance.id)
   ...   
   

   It is important to check how AWS credentials and HTTPS proxy setting can be configured for your application. For example, command line tools like terraform and eksctl support setting AWS credentials and the HTTPS proxy using environment variables, as shown above.

   However, some AWS SDKs may require extra environment variables (e.g. AWS_SDK_LOAD_CONFIG=true for AWS SDK for Go v2) or require configuring the HTTPS proxy through code (e.g. AWS SDK for JavaScript).

5. To log out of the AWS application and remove credentials:

   tsh apps logout awsconsole

Use CloudTrail to see Teleport user activity

To view CloudTrail events for your federated sessions, navigate to the CloudTrail dashboard and go to "Event history".

Each Teleport federated login session uses a Teleport username as the federated username which you can search for to get the events history:

Troubleshooting

Read this section if you run into issues while following this guide.

Internal Server Error or fails to connect in Web UI

When visiting the AWS Management Console from the Teleport Web UI, you may see an InternalServer Error message or other connection issues instead of the AWS Management Console.

If this happens, check the Teleport Application Service logs:

journalctl -u teleport

kubectl -n teleport-agent logs statefulset/teleport-kube-agent

If the Teleport Application Service encounters an error sending a request to the AWS API, the logs will show the error message stack trace.

Within the logs you may see a connection failure such as a i/o timeout regarding sts.amazonaws.com:443. The Teleport Application Service must be able to connect to https://sts.amazonaws.com and https://signin.aws.amazon.com/federation to create an authorized AWS console session.

The Application Service is not authorized to assume a role

If the Teleport Application Service fails to assume the ExampleReadOnlyAccess role when a user attempts to access AWS, you will see an error message similar to the following in the service's logs:

AccessDenied: User: arn:aws:sts::000000000000:assumed-role/ROLE_NAME is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::000000000000:role/ExampleReadOnlyAccess

The AccessDenied message contains the principal assumed by the Teleport Application Service in order to execute the sts:AssumeRole action.

If the principal is the TeleportAWSAccess role, check the trust policy of the ExampleReadOnlyAccess role to ensure that it contains that principal.

Otherwise, the Teleport Application Service is not assuming the TeleportAWSAccess role as expected. Follow the instructions to associate a role with the Teleport Application Service.

remote error: tls: bad certificate error during SSM sessions

You may encounter the remote error: tls: bad certificate error when starting System Session Manager (SSM) sessions using the tsh aws ssm start-session or tsh aws ecs execute-command commands.

The issue is that tsh cannot properly proxy WebSocket connections sent by SSM.

Please upgrade to the latest version of tsh where workarounds have been implemented for tsh aws ssm start-session and tsh aws ecs execute-command. For more information on the tsh workarounds, see the pull requests that introduced them:

• https://github.com/gravitational/teleport/pull/30510
• https://github.com/gravitational/teleport/pull/33705

If you are using tsh proxy aws or if your tsh version does not contain the above fixes, add the following domain to the NO_PROXY environment variable before running tsh commands to ensure the WebSocket connections bypass tsh:

export NO_PROXY=ssmmessages.us-west-1.amazonaws.com

Replace us-west-1 with the AWS region you are accessing.

Management Console expires in one hour

By default, your AWS Management Console session will expire when your Teleport web session expires, with a maximum session duration of 12 hours. You can adjust the duration of your Teleport session by modifying the max_session_ttl parameter in your Teleport roles.

However, due to limitations in AWS IAM role chaining, Management Console sessions will be restricted to one hour if Teleport is running with temporary security credentials.

For example, if the Teleport Application Service is deployed on EKS with an IAM role for service accounts (IRSA), or if the Teleport Application Services assumes a web or SSO identity, the AWS Management Console session will be limited to an hour.

In such cases, it is recommended to deploy the Application Service on an EC2 instance and attach an IAM role to it.

No credential providers error

If you see the error NoCredentialProviders: no valid providers in chain in Application Service logs then Teleport is not detecting the required credentials to connect via AWS IAM permissions. Check whether the credentials or security role has been applied in the machine running the Teleport Application Service.

When running on EKS, this error may occur if the Teleport Application Service cannot access IMDSv2 when the PUT requests hop limit on the worker node instance is set to 1. You can use the following commands to check the hop limit:

aws ec2 describe-instances --instance-ids <node-instance-id> | grep HttpPutResponseHopLimit
                        "HttpPutResponseHopLimit": 1,

See IMDSv2 support for EKS and EKS best practices for more details.

Next steps

Now that you know how to set up Teleport to protect access to the AWS Management Console and APIs, you can tailor your setup to the needs of your organization.

Refine your AWS IAM role mapping

The aws_role_arns field supports template variables so they can be populated dynamically when a user authenticates to Teleport.

For example, you can configure your identity provider to define a SAML attribute or OIDC claim called aws_role_arns, then use this field to list each user's permitted AWS role ARNs on your IdP. If you define a Teleport role to mention the {{external.aws_role_arns}} variable, the Auth Service will fill in the user's permitted ARNs based on data from the IdP:

    aws_role_arns:
    - {{external.aws_role_arns}}

See the Teleport Access Controls Reference for all of the variables and functions you can use in the aws_role_arns field.

Register the AWS application dynamically

You can deploy a pool of Teleport agents to run the Teleport Application Service, then enroll an AWS application in your Teleport cluster as a dynamic resource. Read more about dynamically registering applications.

Choose an alternative agent join method

This guide shows you how to use the join token method to enroll the Teleport Application Service in your cluster. This is one of several available methods, and we recommend reading the Join Services to your Teleport Cluster guide to configure the most appropriate method for your environment.

Further reading

• Learn more about AWS federation in the AWS documentation.
• Read the AWS documentation to learn more about how trust policies work.