Skip to main content
Version: 16.x

AWS KMS

This guide will show you how to set up your Teleport cluster to use the AWS Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your cluster.

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Service instance's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use AWS KMS, all private key material for these CAs will be generated, stored, and used for signing inside of AWS KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave AWS KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have already generated software private keys, a CA rotation must be performed. Read on to migrating an existing cluster to learn more.

Prerequisites​

The features documented on this page are available in Teleport 15.0.0 and higher.

  • Teleport v16.4.7 Enterprise (self-hosted).
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example: 
    $ tsh login --proxy=teleport.example.com [email protected]
    $ tctl status
    # Cluster  teleport.example.com
    # Version  16.4.7
    # CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
  • An AWS account.
Compatibility Warning

Teleport Cloud and Teleport Open Source do not currently support HSMs or Key Management Services.

Step 1/3. Configure AWS IAM permissions​

Your Teleport Auth Service will need permissions to create, sign with, list, and destroy KMS keys in your AWS account.

Start by creating the following AWS IAM policy in your account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListKeys",
            "Effect": "Allow",
            "Action": [
                "kms:ListKeys"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CreateKeys",
            "Effect": "Allow",
            "Action": [
                "kms:CreateKey",
                "kms:TagResource"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:RequestTag/TeleportCluster": "teleport.example.com",
                    "aws:ResourceTag/TeleportCluster": "teleport.example.com"
                }
            }
        },
        {
            "Sid": "UseKeys",
            "Effect": "Allow",
            "Action": [
                "kms:GetPublicKey",
                "kms:ScheduleKeyDeletion",
                "kms:DescribeKey",
                "kms:ListResourceTags",
                "kms:Sign"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/TeleportCluster": "teleport.example.com"
                }
            }
        }
    ]
}

You can create the policy from the AWS console, or via the AWS CLI.

$ aws iam create-policy --policy-name TeleportKMS --policy-document file://policy.json
{
    "Policy": {
        "PolicyName": "TeleportKMS",
        "PolicyId": "XXXXXXXXXXXXXXXXXXXXX",
        "Arn": "arn:aws:iam::123456789012:policy/TeleportKMS",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2024-01-29T23:05:06+00:00",
        "UpdateDate": "2024-01-29T23:05:06+00:00"
    }
}

Attach the policy to an IAM role that your Auth Service instances will authenticate to your AWS account as. You can attach the policy from the AWS console, or via the AWS CLI.

$ aws iam attach-role-policy --role-name <IAM role> --policy-arn <policy ARN>

Grant the Teleport Auth Service access to credentials that it can use to authenticate to AWS.

  • If you are running the Teleport Auth Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
  • If you are running the Teleport Auth Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
  • Otherwise, you must use environment variables

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Have multiple sources of AWS credentials?

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Teleport Auth Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

warning

Be aware that anyone with access to this IAM role will be able create signatures as if they are the Teleport CA. It should be considered a highly privileged role and should be restricted as much as possible.

If you provide AWS credentials to the Teleport Auth Service via the Instance Metadata Service, be aware that any user with access to the instance could use those credentials. You may wish to use the following iptables rule to restrict IMDS access to the root user only:

$ iptables -A OUTPUT -m owner ! --uid-owner 0 -d 169.254.169.254 -j REJECT

Step 2/3. Configure your Auth Service to use KMS keys​

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Service instances in your cluster.

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
# ...
auth_service:
  # ...
  ca_key_params:
    aws_kms:
      account: "AWS account"
      region: "AWS region"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in AWS KMS automatically and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to AWS KMS keys, read on to migrating an existing cluster.

Step 3/3. Make sure everything is working​

After starting up your Auth Service with the aws_kms configuration, you can confirm that Teleport has generated AWS KMS keys in your account by viewing them in the AWS Console. You can also run the following tctl commands to find the ARN of each key.

$ tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.tls[0].key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/dd0f77e2-a1b7-4ecd-ba83-d29476c15fec
awskms:arn:aws:kms:us-west-2:123456789012:key/fe8daacb-2457-4639-aa68-2bdc38a47a89
awskms:arn:aws:kms:us-west-2:123456789012:key/fb48f161-55ba-47da-9d4e-24ed4ef8cce4
awskms:arn:aws:kms:us-west-2:123456789012:key/7c2ae604-8b8c-425e-8d53-c4d3070cdb10
awskms:arn:aws:kms:us-west-2:123456789012:key/d8a4196f-6864-421a-911f-16d69473135c
$ tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.ssh[0].private_key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/dd8b4a02-5602-4d34-8773-a56e669db75f
awskms:arn:aws:kms:us-west-2:123456789012:key/026e3c42-0c37-40d0-b9f4-bd1fa394b9a9
awskms:arn:aws:kms:us-west-2:123456789012:key/ffa93061-0979-4ff1-b0af-d9a5dda6d894
$ tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.jwt[0].private_key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/d4e63a5d-ed81-46c4-ad72-3f57a07eb9d5
awskms:arn:aws:kms:us-west-2:123456789012:key/5c99e6aa-deb1-45c0-865e-08847214b7ab

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

Migrating an existing cluster​

If you have an existing Teleport cluster it will have already created software CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

In order for the Teleport Auth Service to generate new keys in AWS KMS and have them trusted by the rest of the cluster, you will need to rotate all of your Teleport CAs.

teleport start will print a warning during startup if any CA needs to be rotated. Any users allowed the update verb for cert_authority resources in any of their Teleport roles will also see a cluster alert reminding them to rotate the CAs.

CA rotation can be performed manually or semi-automatically. See our admin guide on Certificate rotation. All CAs listed in the output of tctl status must be rotated.