Access AWS With Teleport Application Access
You can protect the AWS Management Console and AWS APIs with Teleport. This makes it possible to manage access to AWS infrastructure with Teleport features like Access Requests, Access Lists, and identity locking. You can also configure Teleport to provide different levels of AWS access automatically when users authenticate using your single sign-on solution.
This guide will explain how to:
- Access the AWS Management Console through Teleport.
- Access the AWS Command Line Interface (CLI) through Teleport.
- Access applications using AWS SDKs through Teleport.
In this setup, the Teleport Application Service has an AWS IAM role that can
assume one or more target IAM roles. Teleport users access the AWS Management
Console and APIs through the Teleport Web UI and tsh
. When a user visits the
AWS Management Console or executes a command with an AWS client application, the
Teleport Application Service checks the user's RBAC permissions and, if they are
authorized, forwards the user's requests to AWS.
Prerequisites
-
A running Teleport cluster version 16.4.7 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.
-
The
tctl
admin tool andtsh
client tool.Visit Installation for instructions on downloading
tctl
andtsh
.
-
To check that you can connect to your Teleport cluster, sign in with
tsh login
, then verify that you can runtctl
commands using your current credentials.For example:
$ tsh login --proxy=teleport.example.com [email protected]
$ tctl status
# Cluster teleport.example.com
# Version 16.4.7
# CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678If you can connect to the cluster and run the
tctl status
command, you can use your current credentials to run subsequenttctl
commands from your workstation. If you host your own Teleport cluster, you can also runtctl
commands on the computer that hosts the Teleport Auth Service for full permissions. -
An AWS EC2 instance or Elastic Kubernetes Service (EKS) cluster where you will run the Teleport Application Service. EC2 instances must be running a Linux distribution. We recommend starting with a fresh demo instance or EKS cluster to get familiar with the procedure before following this guide in production.
-
Permissions to create IAM roles and policies in the AWS account you want to connect.
-
aws
command line interface (CLI) tool in PATH. Read the AWS documentation to install or update the latest version of the AWS CLI. -
If you plan to run the Teleport Application Service on EKS, you will need an IAM OIDC provider running in your Kubernetes cluster. See the AWS documentation for how to create an IAM OIDC provider.
To check whether you have an IAM OIDC provider running in your cluster, run the following
aws
command, assigning eks-region to the region where your EKS cluster is running and cluster-name to the name of your Kubernetes cluster:$ aws --region=eks-region eks describe-cluster --name cluster-name --query "cluster.identity.oidc.issuer" --output text
If you have an IAM OIDC provider associated with your cluster, this command will print its ID.
Step 1/4. Configure AWS IAM
In this section, you will configure AWS IAM resources to allow the Teleport Application Service to proxy AWS APIs.
The rest of the guide assumes that you are protecting access to an IAM role
called ExampleReadOnlyAccess
, which allows read-only access to AWS resources
through the AWS-managed ReadOnlyAccess
policy. We recommend following this
guide in a demo environment with the ExampleReadOnlyAccess
role, and
protecting production AWS roles when you are familiar with the procedure.
You will create the following resources:
Name | Resource | Function |
---|---|---|
ExampleReadOnlyAccess | IAM role | Example role to protect access to with Teleport. |
TeleportAWSAccess | IAM role | Allows the Application Service to assume other roles in order to proxy user requests to AWS. |
AssumeRole | IAM policy | Allows the Application Service to assume other roles in order to proxy user requests to AWS. |
TeleportAWSAccess (for EC2 deployments) | EC2 instance profile | Associates the TeleportAWSAccess role with your EC2 instance. |
Create a role for the Teleport Application Service
In this section, you will create an IAM role that allows the Teleport Application Service to assume other IAM roles in order to proxy user traffic to AWS APIs.
-
Define a trust policy to enable the Teleport Application Service to assume the role you will create.
- EC2
- EKS
Create a file called
app-service-tp.json
:{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}Retrieve your OIDC issuer ID, assigning cluster-name to the name of your EKS cluster and eks-region to the AWS region where your EKS cluster is running:
$ aws --region=eks-region eks describe-cluster --name cluster-name --query "cluster.identity.oidc.issuer" --output text | grep -Eo "[A-Z0-9]+$"
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACreate a file called
app-service-tp.json
with the following content, assigning oidc-issuer to the issuer string you retrieved and EKS_ACCOUNT to the ID of the AWS account that belongs to your EKS cluster:{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::EKS_ACCOUNT:oidc-provider/oidc.eks.EKS_REGION.amazonaws.com/id/OIDC_ISSUER"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.EKS_REGION.amazonaws.com/id/OIDC_ISSUER:aud": "sts.amazonaws.com",
"oidc.eks.EKS_REGION.amazonaws.com/id/OIDC_ISSUER:sub": "system:serviceaccount:teleport-agent:teleport-kube-agent"
}
}
}
]
}Later in this guide, we will install a Helm chart to deploy the Teleport Application Service. The chart assigns the Teleport pod to a service account called
teleport-kube-agent
. This trust policy authorizes workloads with this service account in theteleport-agent
namespace to assume the IAM role we create in this section. -
Create a role for the Teleport Application Service:
$ aws iam create-role --role-name "TeleportAWSAccess" \
--assume-role-policy-document file://app-service-tp.json -
Get the ARN of the
ExampleReadOnlyAccess
role:$ ROLE_ARN=$(aws iam get-role --role-name ExampleReadOnlyAccess --query "Role.Arn" --output text)
-
Define an IAM policy to allow the Teleport Application Service to assume the
ExampleReadOnlyAccess
role:$ cat<<EOF > teleport-assume-role.json
{ "Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "${ROLE_ARN}"
}
]
}
EOF
$ POLICY_ARN=$(aws iam create-policy --policy-name=AssumeExampleReadOnlyRole \
--policy-document file://teleport-assume-role.json | jq -r '.Policy.Arn')
$ aws iam attach-role-policy --role-name TeleportAWSAccess \
--policy-arn ${POLICY_ARN}
Configure a role for Teleport users to request
In this section, you will create a role that Teleport users can request access to when making requests to AWS APIs. The Teleport Application Service assumes this role when proxying requests:
-
Obtain AWS credentials for the account where you will run the Teleport Application Service and make them available to your terminal shell.
-
Create a trust policy document, which authorizes an entity to assume the role you want to protect access to. To do so, create a file called
ro-access.json
with the following content, replacing AWS_ACCESS_ACCOUNT with the ID of the AWS account where you will run the Teleport Application Service:{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWS_ACCESS_ACCOUNT:role/TeleportAWSAccess"
},
"Action": "sts:AssumeRole"
}
]
}In the setup we show in this guide, the Teleport Application Service assumes the
TeleportAWSAccess
role, then uses that role to assume theExampleReadOnlyAccess
role. With the trust policy above, AWS authorizes this operation.Cross-account access
If you are configuring the Application Service to proxy access to IAM roles in another AWS account, we recommend checking the external ID of the AWS account where the Application Service runs. Add the external ID to the trust policy as follows, assigning EXTERNAL_ID to the external ID:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWS_ACCESS_ACCOUNT:role/TeleportAWSAccess"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "EXTERNAL_ID"
}
}
}
]
}See the AWS documentation for details on external IDs.
-
Run the following commands to create the
ExampleReadOnlyAccess
role:$ aws iam create-role --role-name "ExampleReadOnlyAccess" \
--assume-role-policy-document file://ro-access.json -
Get the ARN of the AWS-managed
ReadOnlyAccess
policy so you can attach it to your role:$ ARN=$(aws iam list-policies --output text --query "Policies[?PolicyName=='ReadOnlyAccess'].Arn")
-
Attach the
ReadOnlyAccess
policy to the role:$ aws iam attach-role-policy --role-name ExampleReadOnlyAccess --policy-arn $ARN
Associate a role with the Teleport Application Service
Now that you have created a role for the Teleport Application Service, associate the role with the service.
- EC2
- EKS
In this section, you will add the TeleportAWSAccess
role to the instance
profile of the EC2 instance where you will install the Teleport Application
Service.
-
Create an instance profile:
$ aws iam create-instance-profile --instance-profile-name TeleportAWSAccess
-
Add the
TeleportAWSAccess
role to the profile:$ aws iam add-role-to-instance-profile \
--instance-profile-name TeleportAWSAccess \
--role-name TeleportAWSAccess -
Obtain the ID of the EC2 instance where you will install the Teleport Application Service.
-
Run the following command to associate the new instance profile with your instance:
$ aws ec2 associate-iam-instance-profile --iam-instance-profile Name="TeleportAWSAccess" \
--instance-id INSTANCE_ID --region AWS_REGION
Annotate the service account used by the Teleport Application Service to
instruct it to assume the TeleportAWSAccess
role:
$ kubectl annotate serviceaccount \
-n teleport-agent \
teleport-kube-agent eks.amazonaws.com/role-arn=arn:aws:iam::EKS_ACCOUNT:role/TeleportAWSAccess
Step 2/4. Configure Teleport IAM role mapping
In this step, you will define a Teleport role that confers access to the
ExampleReadOnlyAccess
IAM role you created in the previous step.
-
Create a file called
aws-ro-access.yaml
with the following content, replacing AWS_ACCOUNT with the ID of the account where you created theExampleReadOnlyAccess
role:kind: role
version: v5
metadata:
name: aws-ro-access
spec:
allow:
app_labels:
'*': '*'
aws_role_arns:
- arn:aws:iam::AWS_ACCOUNT:role/ExampleReadOnlyAccessThe Teleport role called
aws-ro-access
enables users to visit the AWS Management Console or make requests to AWS APIs with theExampleReadOnlyAccess
role. -
Create the role:
$ tctl create aws-ro-access.yaml
-
Assign the
aws-ro-access
role to your Teleport user by running the appropriate commands for your authentication provider:- Local User
- GitHub
- SAML
- OIDC
-
Retrieve your local user's roles as a comma-separated list:
$ ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')
-
Edit your local user to add the new role:
$ tctl users update $(tsh status -f json | jq -r '.active.username') \
--set-roles "${ROLES?},aws-ro-access" -
Sign out of the Teleport cluster and sign in again to assume the new role.
-
Open your
github
authentication connector in a text editor:$ tctl edit github/github
-
Edit the
github
connector, addingaws-ro-access
to theteams_to_roles
section.The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.
Here is an example:
teams_to_roles:
- organization: octocats
team: admins
roles:
- access
+ - aws-ro-access -
Apply your changes by saving closing the file in your editor.
-
Sign out of the Teleport cluster and sign in again to assume the new role.
-
Retrieve your
saml
configuration resource:$ tctl get --with-secrets saml/mysaml > saml.yaml
Note that the
--with-secrets
flag adds the value ofspec.signing_key_pair.private_key
to thesaml.yaml
file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource. -
Edit
saml.yaml
, addingaws-ro-access
to theattributes_to_roles
section.The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.
Here is an example:
attributes_to_roles:
- name: "groups"
value: "my-group"
roles:
- access
+ - aws-ro-access -
Apply your changes:
$ tctl create -f saml.yaml
-
Sign out of the Teleport cluster and sign in again to assume the new role.
-
Retrieve your
oidc
configuration resource:$ tctl get oidc/myoidc --with-secrets > oidc.yaml
Note that the
--with-secrets
flag adds the value ofspec.signing_key_pair.private_key
to theoidc.yaml
file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource. -
Edit
oidc.yaml
, addingaws-ro-access
to theclaims_to_roles
section.The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.
Here is an example:
claims_to_roles:
- name: "groups"
value: "my-group"
roles:
- access
+ - aws-ro-access -
Apply your changes:
$ tctl create -f oidc.yaml
-
Sign out of the Teleport cluster and sign in again to assume the new role.
Step 3/4. Set up the Teleport Application Service
Now that you have configured RBAC resources in AWS and Teleport, the remaining step is to launch the Teleport Application Service.
Get a join token
Establish trust between your Teleport cluster and your new Application Service instance by creating a join token:
$ tctl tokens add --type=app --ttl=1h --format=text
abcd123-insecure-do-not-use-this
- EC2
- EKS
On the host where you will install the Teleport Application Service, create a
file called /tmp/token
that consists only of your token:
$ echo join-token | sudo tee /tmp/token
Install the Teleport Application Service
Run the following commands on the host where you will install the Teleport Application Service:
Install Teleport on your Linux server:
-
Assign edition to one of the following, depending on your Teleport edition:
Edition Value Teleport Enterprise Cloud cloud
Teleport Enterprise (Self-Hosted) enterprise
Teleport Community Edition oss
-
Get the version of Teleport to install. If you have automatic agent updates enabled in your cluster, query the latest Teleport version that is compatible with the updater:
$ TELEPORT_DOMAIN=example.teleport.com
$ TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/automaticupgrades/channel/default/version | sed 's/v//')"Otherwise, get the version of your Teleport cluster:
$ TELEPORT_DOMAIN=example.teleport.com
$ TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/ping | jq -r '.server_version')" -
Install Teleport on your Linux server:
$ curl https://cdn.teleport.dev/install-v16.4.7.sh | bash -s ${TELEPORT_VERSION} edition
The installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the installation guide.
Configure the Teleport Application Service
-
On the host where you will run the Teleport Application Service, create a file at
/etc/teleport.yaml
with the following content:version: v3
teleport:
join_params:
token_name: "/tmp/token"
method: token
proxy_server: "PROXY_ADDR"
auth_service:
enabled: off
proxy_service:
enabled: off
ssh_service:
enabled: off
app_service:
enabled: "yes"
apps:
- name: "awsconsole"
# The public AWS Console is used after authenticating the user from Teleport
uri: "https://console.aws.amazon.com/ec2/v2/home" -
Edit
/etc/teleport.yaml
to replace PROXY_ADDR with the host and port of your Teleport Proxy Service or Teleport Cloud tenant, e.g.,example.teleport.sh:443
.The
app_service
field configures the Teleport Application Service. Each item withinapp_service.apps
is an application configuration.
-
Retrieve the join token you created earlier in this guide by running the following command and copy the token with the
App
type:$ tctl tokens ls
Token Type Labels Expiry Time (UTC)
--------------------------------- ---- ------ ----------------------------
abcd123-insecure-do-not-use-this App 14 Jun 23 21:21 UTC (20m15s) -
Create a Helm values file called
values.yaml
, assigning token to the value of the join token you retrieved above, example.teleport.sh:443 to the host and port of your Teleport Proxy Service (e.g.,teleport.example.com:443
):authToken: token
proxyAddr: example.teleport.sh:443
roles: app
apps:
- name: "awsconsole"
# The public AWS Console is used after authenticating the user from Teleport
uri: "https://console.aws.amazon.com/ec2/v2/home" -
Set up the Teleport Helm repository.
Allow Helm to install charts that are hosted in the Teleport Helm repository:
$ helm repo add teleport https://charts.releases.teleport.dev
Update the cache of charts from the remote repository so you can upgrade to all available releases:
$ helm repo update
-
Install the Helm chart for Teleport agent services,
teleport-kube-agent
:$ helm -n teleport-agent install teleport-kube-agent teleport/teleport-kube-agent \
--values values.yaml --create-namespace -
Make sure that the Teleport agent pod is running. You should see one
teleport-kube-agent
pod with a single ready container:$ kubectl -n teleport-agent get pods
NAME READY STATUS RESTARTS AGE
teleport-kube-agent-0 1/1 Running 0 32s
Note that the URI you configure for your AWS app must start with one of the following values in order to be recognized as an AWS console:
Regions | AWS Console URL |
---|---|
Standard AWS regions | https://console.aws.amazon.com |
AWS GovCloud (US) regions | https://console.amazonaws-us-gov.com |
AWS China regions | https://console.amazonaws.cn |
Multiple AWS accounts
If you have multiple AWS accounts and would like to logically separate them
in the UI, register an application entry for each and set aws_account_id
label to the account ID:
app_service:
enabled: "yes"
apps:
- name: "awsconsole-test"
uri: "https://console.aws.amazon.com/ec2/v2/home"
labels:
aws_account_id: "1234567890"
env: test
- name: "awsconsole-prod"
uri: "https://console.aws.amazon.com/ec2/v2/home"
labels:
aws_account_id: "0987654321"
env: prod
- name: "awsconsole-third-party"
uri: "https://console.aws.amazon.com/ec2/v2/home"
labels:
aws_account_id: "1234554321"
aws:
external_id: "example-external-id"
When showing available IAM roles, Teleport will display only role ARNs that belong to the specific account.
For AWS accounts that require external IDs for accessing their resources, set
the external_id
field, which the Application Service uses when assuming the
AWS roles in these accounts.
Start the Teleport Application Service
If you deployed the Teleport Application Service on Kubernetes, it will have already started, and you can skip to Step 4.
- Grant the Application Service access to credentials that it can use to authenticate to
AWS.
- If you are running the Application Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
- If you are running the Application Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
- Otherwise, you must use environment variables
- Instance Metadata Service
- Kubernetes IRSA
- Environment Variables
Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.
The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.
Refer to IAM Roles for Service Accounts (IRSA) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role.
Teleport's built-in AWS client reads credentials from the following environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
When you start the Application Service, the service reads environment variables from a file at the path
/etc/default/teleport
. Obtain these credentials from your organization. Ensure that/etc/default/teleport
has the following content, replacing the values of each variable:AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>Have multiple sources of AWS credentials?
Teleport's AWS client loads credentials from different sources in the following order:
- Environment Variables
- Shared credentials file
- Shared configuration file (Teleport always enables shared configuration)
- EC2 Instance Metadata (credentials only)
While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Application Service with the
AWS_PROFILE
environment variable assigned to the name of your profile of choice.If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.
- Configure the Application Service to start automatically when the host boots up by
creating a systemd service for it. The instructions depend on how you installed
the Application Service.
You can check the status of the Application Service with
- Package Manager
- TAR Archive
On the host where you will run the Application Service, enable and start Teleport:
$ sudo systemctl enable teleport
$ sudo systemctl start teleportOn the host where you will run the Application Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport:
$ sudo teleport install systemd -o /etc/systemd/system/teleport.service
$ sudo systemctl enable teleport
$ sudo systemctl start teleportsystemctl status teleport
and view its logs withjournalctl -fu teleport
.
For non-standard AWS regions such as AWS GovCloud (US) regions and AWS China
regions, set the corresponding region in the AWS_REGION
environment variable
or in the AWS credentials file so that the Application Service can use the
correct STS endpoint.
Step 4/4. Access AWS resources
Now that you have configured the Teleport Application Service to proxy requests to AWS, users can access AWS resources through Teleport.
Access the AWS console
-
Visit the home page of the Teleport Web UI or click the Resources tab. If the Teleport Application Service is proxying the AWS Management Console as expected, the Web UI will display the name of the application you registered, which this guide assumes is
awsconsole
. (Enter the name of the application into the search box if there are too many resources to view all of them at once.) -
Click the Launch button for the AWS Console application, then click on the role you would like to assume when signing in to the AWS Console:
-
You will get redirected to the AWS Management Console, signed in with the selected role. You should see your Teleport user name as a federated login assigned to
ExampleReadOnlyRole
in the top-right corner of the AWS Console:
Access the AWS CLI
-
On your desktop, log into the AWS Management Console app you configured while following this guide:
$ tsh apps login --aws-role ExampleReadOnlyAccess awsconsole
Logged into AWS app "awsconsole".
Your IAM role:
arn:aws:iam::000000000000:role/ExampleReadOnlyAccess
Example AWS CLI command:
tsh aws s3 ls
Or start a local proxy:
tsh proxy aws --app awsconsoleThe
--aws-role
flag allows you to specify the AWS IAM role to assume when accessing AWS APIs. You can either provide a role name like--aws-role ExampleReadOnlyAccess
or a full role ARN likearn:aws:iam::0000000000:role/ExampleReadOnlyAccess
. -
Now you can use the
tsh aws
command like the nativeaws
command-line tool:$ tsh aws s3 ls
-
Run the following command to log out of the AWS application and remove credentials:
$ tsh apps logout awsconsole
Access applications using AWS SDKs
-
On your desktop, log into the AWS Management Console app you configured while following this guide:
$ tsh apps login --aws-role ExampleReadOnlyAccess awsconsole
-
Open a new terminal and use the following command to start a local HTTPS proxy server that forwards AWS API traffic to the Teleport Application Service. Leave the terminal open since it runs in the foreground:
$ tsh proxy aws -p 23456
Started AWS proxy on http://127.0.0.1:23456.
Use the following credentials and HTTPS proxy setting to connect to the proxy:
export AWS_ACCESS_KEY_ID=abcd1234-this-is-an-example
export AWS_SECRET_ACCESS_KEY=zyxw9876-this-is-an-example
export AWS_CA_BUNDLE=<ca-bundle-path>
export HTTPS_PROXY=http://127.0.0.1:23456 -
Copy the
export
commands from the terminal and paste them into a new terminal window. -
You can then run applications that use AWS SDKs to communicate with AWS APIs through the local proxy. For example, open a Python console and use the
boto3
client to fetch EC2 instances in your AWS account:>>> import boto3
>>> ec2 = boto3.resource('ec2')
>>> for instance in ec2.instances.all():
... print(instance.id)
...It is important to check how AWS credentials and HTTPS proxy setting can be configured for your application. For example, command line tools like
terraform
andeksctl
support setting AWS credentials and the HTTPS proxy using environment variables, as shown above.However, some AWS SDKs may require extra environment variables (e.g.
AWS_SDK_LOAD_CONFIG=true
for AWS SDK for Go v2) or require configuring the HTTPS proxy through code (e.g. AWS SDK for JavaScript). -
To log out of the AWS application and remove credentials:
$ tsh apps logout awsconsole
Use CloudTrail to see Teleport user activity
To view CloudTrail events for your federated sessions, navigate to the CloudTrail dashboard and go to "Event history".
Each Teleport federated login session uses a Teleport username as the federated username which you can search for to get the events history:
Troubleshooting
Read this section if you run into issues while following this guide.
Internal Server Error
or fails to connect in Web UI
When visiting the AWS Management Console from the Teleport Web UI, you may see
an InternalServer Error
message or other connection issues instead of the
AWS Management Console.
If this happens, check the Teleport Application Service logs:
- EC2
- EKS
Execute the following command on the EC2 instance that runs the Teleport Application Service:
$ journalctl -u teleport
$ kubectl -n teleport-agent logs statefulset/teleport-kube-agent
If the Teleport Application Service encounters an error sending a request to the AWS API, the logs will show the error message stack trace.
Within the logs you may see a connection failure such as a i/o timeout
regarding sts.amazonaws.com:443
. The Teleport Application Service must be able to
connect to https://sts.amazonaws.com
and https://signin.aws.amazon.com/federation
to create an authorized AWS console session.
The Application Service is not authorized to assume a role
If the Teleport Application Service fails to assume the ExampleReadOnlyAccess
role when a user attempts to access AWS, you will see an error message similar
to the following in the service's logs:
AccessDenied: User: arn:aws:sts::000000000000:assumed-role/ROLE_NAME is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::000000000000:role/ExampleReadOnlyAccess
The AccessDenied
message contains the principal assumed by the Teleport
Application Service in order to execute the sts:AssumeRole
action.
If the principal is the TeleportAWSAccess
role, check the trust policy of the
ExampleReadOnlyAccess
role to ensure that it contains that principal.
Otherwise, the Teleport Application Service is not assuming the
TeleportAWSAccess
role as expected. Follow the instructions to associate a
role with the Teleport Application
Service.
remote error: tls: bad certificate
error during SSM sessions
You may encounter the remote error: tls: bad certificate
error when starting
System Session Manager (SSM) sessions using the tsh aws ssm start-session
or
tsh aws ecs execute-command
commands.
The issue is that tsh
cannot properly proxy WebSocket connections sent by
SSM.
Please upgrade to the latest version of tsh
where workarounds have been
implemented for tsh aws ssm start-session
and tsh aws ecs execute-command
.
For more information on the tsh
workarounds, see the pull requests that
introduced them:
- https://github.com/gravitational/teleport/pull/30510
- https://github.com/gravitational/teleport/pull/33705
If you are using tsh proxy aws
or if your tsh
version does not contain the
above fixes, add the following domain to the NO_PROXY
environment variable
before running tsh
commands to ensure the WebSocket connections bypass tsh
:
export NO_PROXY=ssmmessages.us-west-1.amazonaws.com
Replace us-west-1
with the AWS region you are accessing.
Management Console expires in one hour
By default, your AWS Management Console session will expire when your Teleport
web session expires, with a maximum session duration of 12 hours. You can
adjust the duration of your Teleport session by modifying the max_session_ttl
parameter in your Teleport roles.
However, due to limitations in AWS IAM role chaining, Management Console sessions will be restricted to one hour if Teleport is running with temporary security credentials.
For example, if the Teleport Application Service is deployed on EKS with an IAM role for service accounts (IRSA), or if the Teleport Application Services assumes a web or SSO identity, the AWS Management Console session will be limited to an hour.
In such cases, it is recommended to deploy the Application Service on an EC2 instance and attach an IAM role to it.
No credential providers error
If you see the error NoCredentialProviders: no valid providers in chain
in Application Service logs then Teleport
is not detecting the required credentials to connect via AWS IAM permissions. Check whether
the credentials or security role has been applied in the machine running the Teleport Application Service.
When running on EKS, this error may occur if the Teleport Application Service cannot access IMDSv2 when the PUT requests hop limit on the worker node instance is set to 1. You can use the following commands to check the hop limit:
$ aws ec2 describe-instances --instance-ids <node-instance-id> | grep HttpPutResponseHopLimit
"HttpPutResponseHopLimit": 1,
See IMDSv2 support for EKS and EKS best practices for more details.
Next steps
Now that you know how to set up Teleport to protect access to the AWS Management Console and APIs, you can tailor your setup to the needs of your organization.
Refine your AWS IAM role mapping
The aws_role_arns
field supports template variables so they can be populated
dynamically when a user authenticates to Teleport.
For example, you can configure your identity provider to define a SAML attribute
or OIDC claim called aws_role_arns
, then use this field to list each user's
permitted AWS role ARNs on your IdP. If you define a Teleport role to mention
the {{external.aws_role_arns}}
variable, the Auth Service will fill in the
user's permitted ARNs based on data from the IdP:
aws_role_arns:
- {{external.aws_role_arns}}
See the Teleport Access Controls
Reference
for all of the variables and functions you can use in the aws_role_arns
field.
Register the AWS application dynamically
You can deploy a pool of Teleport agents to run the Teleport Application Service, then enroll an AWS application in your Teleport cluster as a dynamic resource. Read more about dynamically registering applications.
Choose an alternative agent join method
This guide shows you how to use the join token method to enroll the Teleport Application Service in your cluster. This is one of several available methods, and we recommend reading the Join Services to your Teleport Cluster guide to configure the most appropriate method for your environment.
Further reading
- Learn more about AWS federation in the AWS documentation.
- Read the AWS documentation to learn more about how trust policies work.