Meet us at KubeCon + CloudNativeCon: Paris, France - March 19
Book Demo
Teleport logoTry For Free
Fork me on GitHub

Teleport

Store Teleport Private Keys in AWS KMS

  • Available for:
  • Enterprise

This guide will show you how to set up your Teleport cluster to use the AWS Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your cluster.

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Service instance's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use AWS KMS, all private key material for these CAs will be generated, stored, and used for signing inside of AWS KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave AWS KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have already generated software private keys, a CA rotation must be performed. Read on to migrating an existing cluster to learn more.

Prerequisites

The features documented on this page are available in Teleport 15.0.0 and higher.

  • Access to an Enterprise edition of Teleport running in your environment.

    For information about the differences between Teleport editions, see Comparing editions.

  • The Enterprise tctl admin tool and tsh client tool version >= 15.1.1.

    You can verify the tools you have installed by running the following commands:

    tctl version

    Teleport Enterprise v15.1.1 go1.21


    tsh version

    Teleport v15.1.1 go1.21

    You can download these tools by following the appropriate Installation instructions for your environment and Teleport edition.

  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines.

    For example:

    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status

    Cluster teleport.example.com

    Version 15.1.1

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

  • An AWS account.

Step 1/5. Configure AWS IAM permissions

Your Teleport Auth Service will need permissions to create, sign with, list, and destroy KMS keys in your AWS account.

Start by creating the following AWS IAM policy in your account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListKeys",
            "Effect": "Allow",
            "Action": [
                "kms:ListKeys"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CreateKeys",
            "Effect": "Allow",
            "Action": [
                "kms:CreateKey",
                "kms:TagResource"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:RequestTag/TeleportCluster": "teleport.example.com",
                    "aws:ResourceTag/TeleportCluster": "teleport.example.com"
                }
            }
        },
        {
            "Sid": "UseKeys",
            "Effect": "Allow",
            "Action": [
                "kms:GetPublicKey",
                "kms:ScheduleKeyDeletion",
                "kms:DescribeKey",
                "kms:ListResourceTags",
                "kms:Sign"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/TeleportCluster": "teleport.example.com"
                }
            }
        }
    ]
}

You can create the policy from the AWS console, or via the AWS CLI.

aws iam create-policy --policy-name TeleportKMS --policy-document file://policy.json
{ "Policy": { "PolicyName": "TeleportKMS", "PolicyId": "XXXXXXXXXXXXXXXXXXXXX", "Arn": "arn:aws:iam::123456789012:policy/TeleportKMS", "Path": "/", "DefaultVersionId": "v1", "AttachmentCount": 0, "PermissionsBoundaryUsageCount": 0, "IsAttachable": true, "CreateDate": "2024-01-29T23:05:06+00:00", "UpdateDate": "2024-01-29T23:05:06+00:00" }}

Attach the policy to an IAM role that your Auth Service instances will authenticate to your AWS account as. You can attach the policy from the AWS console, or via the AWS CLI.

aws iam attach-role-policy --role-name <IAM role> --policy-arn <policy ARN>

Grant the Teleport Auth Service access to credentials that it can use to authenticate to AWS. If you are running the Teleport Auth Service on an EC2 instance, you should use the EC2 Instance Metadata Service method. Otherwise, you must use environment variables:

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Teleport's built-in AWS client reads credentials from the following environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION

When you start the Teleport Auth Service, the service reads environment variables from a file at the path /etc/default/teleport. Obtain these credentials from your organization. Ensure that /etc/default/teleport has the following content, replacing the values of each variable:

AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Teleport Auth Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Step 2/5. Configure your Auth Service to use KMS keys

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Service instances in your cluster.

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
# ...
auth_service:
  # ...
  ca_key_params:
    aws_kms:
      account: "AWS account"
      region: "AWS region"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in AWS KMS automatically and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to AWS KMS keys, read on to migrating an existing cluster.

Step 3/3. Make sure everything is working

After starting up your Auth Service with the aws_kms configuration, you can confirm that Teleport has generated AWS KMS keys in your account by viewing them in the AWS Console. You can also run the following tctl commands to find the ARN of each key.

tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.tls[0].key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/dd0f77e2-a1b7-4ecd-ba83-d29476c15fecawskms:arn:aws:kms:us-west-2:123456789012:key/fe8daacb-2457-4639-aa68-2bdc38a47a89awskms:arn:aws:kms:us-west-2:123456789012:key/fb48f161-55ba-47da-9d4e-24ed4ef8cce4awskms:arn:aws:kms:us-west-2:123456789012:key/7c2ae604-8b8c-425e-8d53-c4d3070cdb10awskms:arn:aws:kms:us-west-2:123456789012:key/d8a4196f-6864-421a-911f-16d69473135c
tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.ssh[0].private_key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/dd8b4a02-5602-4d34-8773-a56e669db75fawskms:arn:aws:kms:us-west-2:123456789012:key/026e3c42-0c37-40d0-b9f4-bd1fa394b9a9awskms:arn:aws:kms:us-west-2:123456789012:key/ffa93061-0979-4ff1-b0af-d9a5dda6d894
tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.jwt[0].private_key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/d4e63a5d-ed81-46c4-ad72-3f57a07eb9d5awskms:arn:aws:kms:us-west-2:123456789012:key/5c99e6aa-deb1-45c0-865e-08847214b7ab

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

Migrating an existing cluster

If you have an existing Teleport cluster it will have already created software CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

In order for the Teleport Auth Service to generate new keys in AWS KMS and have them trusted by the rest of the cluster, you will need to rotate all of your Teleport CAs.

teleport start will print a warning during startup if any CA needs to be rotated. Any users allowed the update verb for cert_authority resources in any of their Teleport roles will also see a cluster alert reminding them to rotate the CAs.

CA rotation can be performed manually or semi-automatically. See our admin guide on Certificate rotation. All CAs listed in the output of tctl status must be rotated.