Securing Infrastructure Access at Scale in Large Enterprises
Dec 12
Virtual
Register Now
Support
LOG IN
Teleport logoTry For Free
Contact SalesTry for Free
Support
LOG IN
Fork me on GitHub

Teleport

Store Teleport Private Keys in AWS KMS

Version 16.x

This guide will show you how to set up your Teleport cluster to use the AWS Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your cluster.

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Service instance's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use AWS KMS, all private key material for these CAs will be generated, stored, and used for signing inside of AWS KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave AWS KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have already generated software private keys, a CA rotation must be performed. Read on to migrating an existing cluster to learn more.

Prerequisites

The features documented on this page are available in Teleport 15.0.0 and higher.

  • Teleport v16.4.8 Enterprise (self-hosted).
  • To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example: 
    tsh login --proxy=teleport.example.com --user=[email protected]
    tctl status
    Cluster  teleport.example.com
    Version  16.4.8
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
    If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
  • An AWS account.
Compatibility Warning

Teleport Cloud and Teleport Open Source do not currently support HSMs or Key Management Services.

Step 1/3. Configure AWS IAM permissions

Your Teleport Auth Service will need permissions to create, sign with, list, and destroy KMS keys in your AWS account.

Start by creating the following AWS IAM policy in your account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListKeys",
            "Effect": "Allow",
            "Action": [
                "kms:ListKeys"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CreateKeys",
            "Effect": "Allow",
            "Action": [
                "kms:CreateKey",
                "kms:TagResource"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:RequestTag/TeleportCluster": "teleport.example.com",
                    "aws:ResourceTag/TeleportCluster": "teleport.example.com"
                }
            }
        },
        {
            "Sid": "UseKeys",
            "Effect": "Allow",
            "Action": [
                "kms:GetPublicKey",
                "kms:ScheduleKeyDeletion",
                "kms:DescribeKey",
                "kms:ListResourceTags",
                "kms:Sign"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/TeleportCluster": "teleport.example.com"
                }
            }
        }
    ]
}

You can create the policy from the AWS console, or via the AWS CLI.

aws iam create-policy --policy-name TeleportKMS --policy-document file://policy.json
{    "Policy": {        "PolicyName": "TeleportKMS",        "PolicyId": "XXXXXXXXXXXXXXXXXXXXX",        "Arn": "arn:aws:iam::123456789012:policy/TeleportKMS",        "Path": "/",        "DefaultVersionId": "v1",        "AttachmentCount": 0,        "PermissionsBoundaryUsageCount": 0,        "IsAttachable": true,        "CreateDate": "2024-01-29T23:05:06+00:00",        "UpdateDate": "2024-01-29T23:05:06+00:00"    }}

Attach the policy to an IAM role that your Auth Service instances will authenticate to your AWS account as. You can attach the policy from the AWS console, or via the AWS CLI.

aws iam attach-role-policy --role-name <IAM role> --policy-arn <policy ARN>

Grant the Teleport Auth Service access to credentials that it can use to authenticate to AWS.

  • If you are running the Teleport Auth Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
  • If you are running the Teleport Auth Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
  • Otherwise, you must use environment variables

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Refer to IAM Roles for Service Accounts (IRSA) to set up an OIDC provider in AWS and configure an AWS IAM role that allows the pod's service account to assume the role.

Teleport's built-in AWS client reads credentials from the following environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION

When you start the Teleport Auth Service, the service reads environment variables from a file at the path /etc/default/teleport. Obtain these credentials from your organization. Ensure that /etc/default/teleport has the following content, replacing the values of each variable:

AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Teleport Auth Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Warning

Be aware that anyone with access to this IAM role will be able create signatures as if they are the Teleport CA. It should be considered a highly privileged role and should be restricted as much as possible.

If you provide AWS credentials to the Teleport Auth Service via the Instance Metadata Service, be aware that any user with access to the instance could use those credentials. You may wish to use the following iptables rule to restrict IMDS access to the root user only:

$ iptables -A OUTPUT -m owner ! --uid-owner 0 -d 169.254.169.254 -j REJECT

Step 2/3. Configure your Auth Service to use KMS keys

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Service instances in your cluster.

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
# ...
auth_service:
  # ...
  ca_key_params:
    aws_kms:
      account: "AWS account"
      region: "AWS region"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in AWS KMS automatically and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to AWS KMS keys, read on to migrating an existing cluster.

Step 3/3. Make sure everything is working

After starting up your Auth Service with the aws_kms configuration, you can confirm that Teleport has generated AWS KMS keys in your account by viewing them in the AWS Console. You can also run the following tctl commands to find the ARN of each key.

tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.tls[0].key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/dd0f77e2-a1b7-4ecd-ba83-d29476c15fecawskms:arn:aws:kms:us-west-2:123456789012:key/fe8daacb-2457-4639-aa68-2bdc38a47a89awskms:arn:aws:kms:us-west-2:123456789012:key/fb48f161-55ba-47da-9d4e-24ed4ef8cce4awskms:arn:aws:kms:us-west-2:123456789012:key/7c2ae604-8b8c-425e-8d53-c4d3070cdb10awskms:arn:aws:kms:us-west-2:123456789012:key/d8a4196f-6864-421a-911f-16d69473135c
tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.ssh[0].private_key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/dd8b4a02-5602-4d34-8773-a56e669db75fawskms:arn:aws:kms:us-west-2:123456789012:key/026e3c42-0c37-40d0-b9f4-bd1fa394b9a9awskms:arn:aws:kms:us-west-2:123456789012:key/ffa93061-0979-4ff1-b0af-d9a5dda6d894
tctl get cert_authorities --with-secrets --format json | jq -r '.[].spec.active_keys.jwt[0].private_key | select(.) | @base64d'
awskms:arn:aws:kms:us-west-2:123456789012:key/d4e63a5d-ed81-46c4-ad72-3f57a07eb9d5awskms:arn:aws:kms:us-west-2:123456789012:key/5c99e6aa-deb1-45c0-865e-08847214b7ab

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

Migrating an existing cluster

If you have an existing Teleport cluster it will have already created software CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

In order for the Teleport Auth Service to generate new keys in AWS KMS and have them trusted by the rest of the cluster, you will need to rotate all of your Teleport CAs.

teleport start will print a warning during startup if any CA needs to be rotated. Any users allowed the update verb for cert_authority resources in any of their Teleport roles will also see a cluster alert reminding them to rotate the CAs.

CA rotation can be performed manually or semi-automatically. See our admin guide on Certificate rotation. All CAs listed in the output of tctl status must be rotated.