Fork me on GitHub

Teleport

Store Teleport Private Keys in Google Cloud KMS

Improve

This guide will show you how to set up your Teleport Cluster to use the Google Cloud Key Management Service (KMS) to store and handle the CA private key material used to sign all certificates issued by your Teleport cluster.

Teleport generates private key material for its internal Certificate Authorities (CAs) during the first Auth Server's initial startup. These CAs are used to sign all certificates issued to clients and hosts in the Teleport cluster. When configured to use Google Cloud KMS, all private key material for these CAs will be generated, stored, and used for signing inside of Google Cloud KMS. Instead of the actual private key, Teleport will only store the ID of the KMS key. In short, private key material will never leave Google Cloud KMS.

If launching a new Teleport cluster this will all be handled during initial startup with no specific interventions required after configuration. For existing Teleport clusters that already have private key material on disk, a CA rotation must be performed. This ensures that the private key material only ever exists in KMS when this feature is enabled. Read on to migrating an existing cluster to learn more.

This guide is intended for self-hosted Teleport Enterprise users.

Prerequisites

  • Teleport version 11.1.1 or newer Enterprise (self-hosted).

  • The tctl administration tool

    To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

    tsh login --proxy=teleport.example.com [email protected]
    tctl status

    Cluster teleport.example.com

    Version 11.3.1

    CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    You can run subsequent tctl commands in this guide on your local machine.

    For full privileges, you can also run tctl commands on your Auth Service host.

    To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

    tsh login --proxy=myinstance.teleport.sh [email protected]
    tctl status

    Cluster myinstance.teleport.sh

    Version 11.2.1

    CA pin sha256:sha-hash-here

    You must run subsequent tctl commands in this guide on your local machine.

  • A Google Cloud account.

Step 1/5. Create a key ring in GCP

Each Teleport Auth Server will need to be configured to use a GCP key ring which will hold all keys generated and used by that Auth Server. If running a High-Availability Teleport cluster with two or more Auth Servers, every Auth Server can be configured to use the same key ring, or if desired each can be configured to use a unique key ring in a different region (for redundancy or to decrease latency).

It is recommended to create a dedicated key ring for use by Teleport to logically separate it from any other keys in your cloud account. Choose a supported KMS location for the key ring which is geographically near to your Teleport Auth Servers.

You can create a key ring from the Google Cloud Console or from the gcloud CLI tool. Follow this guide or run the following command if you have the gcloud CLI configured:

gcloud kms keyrings create "
" --location

Step 2/5. Create a GCP service account

Teleport needs permissions to create, list, destroy, sign with, and view KMS keys in your key ring. Start by creating the following custom IAM role.

# teleport_kms_role.yaml
title: teleport_kms_role
description: 'Teleport permissions for using KMS keys'
stage: ALPHA
includedPermissions:
- cloudkms.cryptoKeys.create
- cloudkms.cryptoKeys.list
- cloudkms.cryptoKeyVersions.create
- cloudkms.cryptoKeyVersions.destroy
- cloudkms.cryptoKeyVersions.useToSign
- cloudkms.cryptoKeyVersions.viewPublicKey
gcloud iam roles create teleport_kms_role \ --project
\
--file teleport_kms_role.yaml \ --format yaml

Note the name field in the output which is the fully qualified name for the custom role and must be used in later steps.

export IAM_ROLE=<role name output from above>

If you don't already have a GCP service account for your Teleport Auth Server you can create one with the following command, otherwise use your existing service account.

gcloud iam service-accounts create teleport-auth-server \ --description="Service account for Teleport Auth Server" \ --display-name="Teleport Auth Server" \ --format=yaml

Note the email field in the output, this must be used as the identifier for the service account.

export SERVICE_ACCOUNT=<email output from above command>

Create the IAM policy binding to grant the role to the service account for this keyring.

gcloud kms keyrings add-iam-policy-binding
\
--location
\
--member "serviceAccount:${SERVICE_ACCOUNT}" \ --role "${IAM_ROLE}"

Step 3/5. Provide the service account credentials to your Auth Server

The Teleport Auth Server will use Application Default Credentials to make requests to the GCP KMS service. Provide credentials for the teleport-auth-server service account created in step 2 to the Application Default Credentials of the environment you are running your Teleport Auth Server in. Supported environments include GCE VMs, GKE pods, and others.

See the GCP docs for Application Default Credentials to learn how to provide them for your preferred environment.

To make sure the credentials have been configured correctly, you can run the gcloud CLI tool from your Teleport Auth Server's environment. Some example commands you could use to debug are listed here:

gcloud kms keys list --location
--keyring "
"

Listed 0 items.

gcloud kms keys create --location
--keyring "
" \
--purpose asymmetric-signing \ --default-algorithm rsa-sign-pkcs1-4096-sha512 \ test-key
gcloud kms keys list --location
--keyring "
"

NAME PURPOSE ALGORITHM PROTECTION_LEVEL

projects/my-gcp-account/locations/global/keyRings/teleport-keyring/cryptoKeys/test-key ASYMMETRIC_SIGN RSA_SIGN_PKCS1_4096_SHA512 SOFTWARE

echo hello > /tmp/hello.txt
gcloud kms asymmetric-sign --keyring "
" --location
\
--key "test-key" --version 1 \ --input-file /tmp/hello.txt --signature-file /tmp/hello.sig
gcloud kms keys versions destroy --keyring "
" --location
--key "test-key" 1

Step 4/5. Configure your Auth Server to use KMS keys

CA key parameters are statically configured in the teleport.yaml configuration file of the Teleport Auth Server(s) in your cluster.

Find the fully qualified name of the KMS key ring you created in step 1 in the GCP Console or by running:

gcloud kms keyrings list --location

Supported KMS protection levels are SOFTWARE and HSM. If you choose SOFTWARE, GCP KMS will performs all cryptographic operations in software (Teleport performs no cryptographic operations). If you choose HSM, GCP KMS will perform all cryptographic operations in a Hardware Security Module.

Both protection levels are considered secure by Google and Teleport, you should evaluate your own organization's requirements and security policies when making your decision.

One relevant difference is the usage quotas available to keys of each protection level. At the time of writing, software keys have a project-wide quota of 60k cryptographic operations per minute, while asymmetric HSM keys have a quota of 3k operations per minute. See the KMS docs for updated numbers. If your cluster will have many thousands of hosts or active users, the higher quota software keys have may help to avoid any potential throttling, especially during CA rotations where many new certificates must be signed.

Include the following ca_key_params configuration in the auth_service section.

# /etc/teleport.yaml
auth_service:
  # ...
  ca_key_params:
    gcp_kms:
      keyring: "projects/<your-gcp-project>/locations/<location>/keyRing/<your-teleport-keyring>"
      protection_level: "SOFTWARE"

If configuring this before the first start of a new Teleport cluster, the initial CA keys will be generated in GCP and no additional steps are necessary. If you wish to migrate an existing Teleport cluster from software keys to GCP KMS keys, read on to migrating an existing cluster.

Step 5/5. Make sure everything is working

After starting up your Auth Server with the gcp_kms configuration, you can confirm that Teleport has generated keys in your keyring in the GCP Console or by running

gcloud kms keys list --keyring "
" --location

Try logging in to the cluster with a Teleport user to make sure that new certificates can be signed without error.

Migrating an existing cluster

If you have an existing Teleport cluster it will have already created CA keys during its first start. Those existing CA keys will have been used to sign all existing user and host certificates, and will be trusted by all other services in your cluster.

When an Auth Server starts up with a gcp_kms keyring configured in its ca_key_params, it will refuse to sign any certificates with any existing software keys in the CA. This will prevent any new user logins or new hosts from joining your cluster if their requests are directed to that Auth Server and effectively cause downtime for that server until a CA rotation is completed.

If some downtime until you can complete a CA rotation is acceptable, the migration can be performed in three steps:

  1. Configure all Auth Servers ca_key_params to use your desired KMS keyring, as described in Step 4.
  2. Restart all Auth Servers.
  3. Perform a full CA rotation.

To avoid any downtime while migrating your cluster, do the following procedure instead:

  1. Start a new Auth Server with an identical backend configuration to your existing Auth Servers and with ca_key_params configured to use your KMS key ring. Make sure no requests are routed to this new, temporary, Auth Server by not adding it to your load balancer. You can run this anywhere with access to your existing backend and new KMS key ring, one option would be to run it locally on an existing Auth Server host (make sure to give it its own teleport.yaml and unique data_dir).
  2. Perform a full CA rotation. The temporary Auth Server will generate new KMS keys and include their names in the backend CA state.
  3. Stop/remove/delete the temporary Auth Server as it is no longer necessary.
  4. Configure all other existing Auth Servers with identical ca_key_params and reload/restart them, one by one. They will now use the KMS keys generated by the temporary Auth Server.
  5. Perform one more full CA rotation to evict all now-unused software keys from the CA backend state so that hosts will no longer trust them.