Teleport
Store Teleport Private Keys in AWS KMS
- Edge version
- Version 17.x
- Version 16.x
- Version 15.x
- Older Versions
This guide will show you how to set up your Teleport cluster to use the AWS Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your cluster.
Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Service instance's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use AWS KMS, all private key material for these CAs will be generated, stored, and used for signing inside of AWS KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave AWS KMS.
If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have already generated software private keys, a CA rotation must be performed. Read on to migrating an existing cluster to learn more.
Prerequisites
The features documented on this page are available in Teleport 15.0.0
and
higher.
- Teleport v16.4.10 Enterprise (self-hosted).
- To check that you can connect to your Teleport cluster, sign in with
tsh login
, then verify that you can runtctl
commands using your current credentials. For example:If you can connect to the cluster and run thetsh login --proxy=teleport.example.com --user=[email protected]tctl statusCluster teleport.example.com
Version 16.4.10
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
tctl status
command, you can use your current credentials to run subsequenttctl
commands from your workstation. If you host your own Teleport cluster, you can also runtctl
commands on the computer that hosts the Teleport Auth Service for full permissions. - An AWS account.
Teleport Cloud and Teleport Open Source do not currently support HSMs or Key Management Services.
Step 1/3. Configure AWS IAM permissions
Your Teleport Auth Service will need permissions to create, sign with, list, and destroy KMS keys in your AWS account.
Start by creating the following AWS IAM policy in your account.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListKeys",
"Effect": "Allow",
"Action": [
"kms:ListKeys"
],
"Resource": "*"
},
{
"Sid": "CreateKeys",
"Effect": "Allow",
"Action": [
"kms:CreateKey",
"kms:TagResource"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:RequestTag/TeleportCluster": "teleport.example.com",
"aws:ResourceTag/TeleportCluster": "teleport.example.com"
}
}
},
{
"Sid": "UseKeys",
"Effect": "Allow",
"Action": [
"kms:GetPublicKey",
"kms:ScheduleKeyDeletion",
"kms:DescribeKey",
"kms:ListResourceTags",
"kms:Sign"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/TeleportCluster": "teleport.example.com"
}
}
}
]
}
You can create the policy from the AWS console, or via the AWS CLI.
aws iam create-policy --policy-name TeleportKMS --policy-document file://policy.json{ "Policy": { "PolicyName": "TeleportKMS", "PolicyId": "XXXXXXXXXXXXXXXXXXXXX", "Arn": "arn:aws:iam::123456789012:policy/TeleportKMS", "Path": "/", "DefaultVersionId": "v1", "AttachmentCount": 0, "PermissionsBoundaryUsageCount": 0, "IsAttachable": true, "CreateDate": "2024-01-29T23:05:06+00:00", "UpdateDate": "2024-01-29T23:05:06+00:00" }}
Attach the policy to an IAM role that your Auth Service instances will authenticate to your AWS account as. You can attach the policy from the AWS console, or via the AWS CLI.
aws iam attach-role-policy --role-name <IAM role> --policy-arn <policy ARN>
Grant the Teleport Auth Service access to credentials that it can use to authenticate to AWS.
- If you are running the Teleport Auth Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
- If you are running the Teleport Auth Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
- Otherwise, you must use environment variables
Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.
The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.
Refer to IAM Roles for Service Accounts (IRSA) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role.
Teleport's built-in AWS client reads credentials from the following environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
When you start the Teleport Auth Service, the service reads environment variables from a
file at the path /etc/default/teleport
. Obtain these credentials from your
organization. Ensure that /etc/default/teleport
has the following content,
replacing the values of each variable:
AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>
Teleport's AWS client loads credentials from different sources in the following order:
- Environment Variables
- Shared credentials file
- Shared configuration file (Teleport always enables shared configuration)
- EC2 Instance Metadata (credentials only)
While you can provide AWS credentials via a shared credentials file or shared
configuration file, you will need to run the Teleport Auth Service with the AWS_PROFILE
environment variable assigned to the name of your profile of choice.
If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.
Be aware that anyone with access to this IAM role will be able create signatures as if they are the Teleport CA. It should be considered a highly privileged role and should be restricted as much as possible.
If you provide AWS credentials to the Teleport Auth Service via the Instance Metadata Service, be aware that any user with access to the instance could use those credentials. You may wish to use the following iptables rule to restrict IMDS access to the root user only:
$ iptables -A OUTPUT -m owner ! --uid-owner 0 -d 169.254.169.254 -j REJECT
Step 2/3. Configure your Auth Service to use KMS keys
CA key parameters are statically configured in the teleport.yaml
configuration
file of the Teleport Auth Service instances in your cluster.
Include the following ca_key_params
configuration in the auth_service
section.
# /etc/teleport.yaml
# ...
auth_service:
# ...
ca_key_params:
aws_kms:
account: "AWS account"
region: "AWS region"
If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in AWS KMS automatically and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to AWS KMS keys, read on to migrating an existing cluster.
Step 3/3. Make sure everything is working
After starting up your Auth Service with the aws_kms
configuration, you can
confirm that Teleport has generated AWS KMS keys in your account by viewing them
in the AWS Console.
You can also run the following tctl
commands to find the ARN of each key.
tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.tls[0].key | select(.) | @base64d'awskms:arn:aws:kms:us-west-2:123456789012:key/dd0f77e2-a1b7-4ecd-ba83-d29476c15fecawskms:arn:aws:kms:us-west-2:123456789012:key/fe8daacb-2457-4639-aa68-2bdc38a47a89awskms:arn:aws:kms:us-west-2:123456789012:key/fb48f161-55ba-47da-9d4e-24ed4ef8cce4awskms:arn:aws:kms:us-west-2:123456789012:key/7c2ae604-8b8c-425e-8d53-c4d3070cdb10awskms:arn:aws:kms:us-west-2:123456789012:key/d8a4196f-6864-421a-911f-16d69473135ctctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.ssh[0].private_key | select(.) | @base64d'awskms:arn:aws:kms:us-west-2:123456789012:key/dd8b4a02-5602-4d34-8773-a56e669db75fawskms:arn:aws:kms:us-west-2:123456789012:key/026e3c42-0c37-40d0-b9f4-bd1fa394b9a9awskms:arn:aws:kms:us-west-2:123456789012:key/ffa93061-0979-4ff1-b0af-d9a5dda6d894tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.jwt[0].private_key | select(.) | @base64d'awskms:arn:aws:kms:us-west-2:123456789012:key/d4e63a5d-ed81-46c4-ad72-3f57a07eb9d5awskms:arn:aws:kms:us-west-2:123456789012:key/5c99e6aa-deb1-45c0-865e-08847214b7ab
Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.
Migrating an existing cluster
If you have an existing Teleport cluster it will have already created software CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.
In order for the Teleport Auth Service to generate new keys in AWS KMS and have them trusted by the rest of the cluster, you will need to rotate all of your Teleport CAs.
teleport start
will print a warning during startup if any CA needs to be rotated.
Any users allowed the update
verb for cert_authority
resources in any of
their Teleport roles will also see a cluster alert reminding them to rotate the
CAs.
CA rotation can be performed manually or semi-automatically. See our admin guide
on Certificate rotation.
All CAs listed in the output of tctl status
must be rotated.