Store Teleport Private Keys in Google Cloud KMS

This guide will show you how to set up your Teleport Cluster to use the Google Cloud Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your Teleport cluster.

How it works

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Service instance's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use Google Cloud KMS, all private key material for these CAs will be generated, stored, and used for signing inside of Google Cloud KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave Google Cloud KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have already generated software private keys, a CA rotation must be performed. Read on to migrating an existing cluster to learn more.

Prerequisites

• Teleport v16.4.12 Enterprise (self-hosted).
• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example:

  tsh login --proxy=teleport.example.com --user=[email protected]
  tctl status
  Cluster  teleport.example.com
  Version  16.4.12
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
• A Google Cloud account.

Step 1/5. Create a key ring in GCP

Each Teleport Auth Service instance will need to be configured to use a GCP key ring which will hold all keys generated and used by that Auth Service instance. If running a High-Availability Teleport cluster with two or more Auth Servers, every Auth Service instance can be configured to use the same key ring, or if desired each can be configured to use a unique key ring in a different region (for redundancy or to decrease latency).

It is recommended to create a dedicated key ring for use by Teleport to logically separate it from any other keys in your cloud account. Choose a supported KMS location for the key ring which is geographically near to your Teleport Auth Servers.

You can create a key ring from the Google Cloud Console or from the gcloud CLI tool. Follow this guide or run the following command if you have the gcloud CLI configured:

gcloud kms keyrings create "teleport-keyring" --location location

Step 2/5. Create a GCP service account

Teleport needs permissions to create, list, destroy, sign with, and view KMS keys in your key ring. Start by creating the following custom IAM role.

# teleport_kms_role.yaml
title: teleport_kms_role
description: 'Teleport permissions for using KMS keys'
stage: ALPHA
includedPermissions:
- cloudkms.cryptoKeys.create
- cloudkms.cryptoKeys.list
- cloudkms.cryptoKeyVersions.create
- cloudkms.cryptoKeyVersions.destroy
- cloudkms.cryptoKeyVersions.useToSign
- cloudkms.cryptoKeyVersions.viewPublicKey

gcloud iam roles create teleport_kms_role \    --project GCP-Project-ID \    --file teleport_kms_role.yaml \    --format yaml

Note the name field in the output which is the fully qualified name for the custom role and must be used in later steps.

export IAM_ROLE=<role name output from above>

If you don't already have a GCP service account for your Teleport Auth Service you can create one with the following command, otherwise use your existing service account.

gcloud iam service-accounts create teleport-auth-server \    --description="Service account for Teleport Auth Server" \    --display-name="Teleport Auth Server" \    --format=yaml

Note the email field in the output, this must be used as the identifier for the service account.

export SERVICE_ACCOUNT=<email output from above command>

Create the IAM policy binding to grant the role to the service account for this keyring.

gcloud kms keyrings add-iam-policy-binding teleport-keyring \    --location location \    --member "serviceAccount:${SERVICE_ACCOUNT}" \    --role "${IAM_ROLE}"

Step 3/5. Provide the service account credentials to the Auth Service

The Teleport Auth Service will use Application Default Credentials to make requests to the GCP KMS service. Provide credentials for the teleport-auth-server service account created in step 2 to the Application Default Credentials of the environment you are running your Teleport Auth Service in. Supported environments include GCE VMs, GKE pods, and others.

See the GCP docs for Application Default Credentials to learn how to provide them for your preferred environment.

gcloud kms keys list --location location --keyring "teleport-keyring"
Listed 0 items.
gcloud kms keys create --location location --keyring "teleport-keyring" \    --purpose asymmetric-signing \    --default-algorithm rsa-sign-pkcs1-4096-sha512 \    test-key
gcloud kms keys list --location location --keyring "teleport-keyring"
NAME                                                                                   PURPOSE          ALGORITHM                   PROTECTION_LEVELprojects/my-gcp-account/locations/global/keyRings/teleport-keyring/cryptoKeys/test-key ASYMMETRIC_SIGN  RSA_SIGN_PKCS1_4096_SHA512  SOFTWARE
echo hello > /tmp/hello.txt
gcloud kms asymmetric-sign --keyring "teleport-keyring" --location location \    --key "test-key" --version 1 \    --input-file /tmp/hello.txt --signature-file /tmp/hello.sig
gcloud kms keys versions destroy --keyring "teleport-keyring" --location location --key "test-key" 1

Step 4/5. Configure the Auth Service to use KMS keys

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Service instances in your cluster.

Find the fully qualified name of the KMS key ring you created in step 1 in the GCP Console or by running:

gcloud kms keyrings list --location location

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
auth_service:
  # ...
  ca_key_params:
    gcp_kms:
      keyring: "projects/<your-gcp-project>/locations/<location>/keyRings/<your-teleport-keyring>"
      protection_level: "SOFTWARE"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in GCP and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to GCP KMS keys, read on to migrating an existing cluster.

Step 5/5. Make sure everything is working

After starting up your Auth Service with the gcp_kms configuration, you can confirm that Teleport has generated keys in your keyring in the GCP Console or by running:

gcloud kms keys list --keyring "teleport-keyring" --location location

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

Migrating an existing cluster

If you have an existing Teleport cluster it will have already created software CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

In order for the Teleport Auth Service to generate new keys in GCP KMS and have them trusted by the rest of the cluster, you will need to rotate all of your Teleport CAs.

teleport start will print a warning during startup if any CA needs to be rotated. Any users allowed the update verb for cert_authority resources in any of their Teleport roles will also see a cluster alert reminding them to rotate the CAs.

CA rotation can be performed manually or semi-automatically, see our admin guide on Certificate rotation. All CAs listed in the output of tctl status must be rotated.