Sign In

Sign in to Teleport

Teleport Cloud LoginLogin to your Teleport AccountDashboard LoginLegacy Login & Teleport Enterprise Downloads
Get Started
Fork me on GitHub

Teleport

Teleport HSM Support

Version 13.x
Improve

This guide will show you how to set up the Teleport Auth Service to use a hardware security module (HSM) to store and handle private keys.

This guide is intended for Teleport Enterprise users.

Prerequisites

  • Teleport v13.0.3 Enterprise (self-hosted).
  • Make sure you can connect to Teleport. Log in to your cluster using tsh, then use tctl remotely: 
    tsh login --proxy=teleport.example.com [email protected]
    tctl status
    Cluster  teleport.example.com
    Version  13.0.3
    CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

    You can run subsequent tctl commands in this guide on your local machine.

    For full privileges, you can also run tctl commands on your Auth Service host.

  • An HSM reachable from your Teleport auth server.
  • The PKCS#11 module for your HSM.
Compatibility Warning

Teleport Cloud and Teleport Open Source do not currently support HSM.

While most PKCS#11 HSMs should be supported, the Teleport team tests with AWS CloudHSM, YubiHSM2, and SoftHSM2.

Step 1/5. Set up your HSM

You will need to set up your HSM and make sure that it is accessible from your Teleport Auth Server. You should create a unique HSM user or token for Teleport to use.

  1. Create a CloudHSM cluster in the VPC where you will run your Teleport Auth Server. https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-cluster.html

  2. Wait for the newly created cluster to enter the "Uninitialized" state.

  3. Add an HSM to the new cluster, using the AWS Console or the AWS CLI:

    aws cloudhsmv2 create-hsm --cluster-id cluster ID --availability-zone availability zone
    {
        "Hsm": {
            "AvailabilityZone": "ca-central-1a",
            "ClusterId": "cluster-6uysmebmutd",
            "SubnetId": "subnet-0c535b67a117f7186",
            "HsmId": "hsm-ppzzfxbleki",
            "State": "CREATE_IN_PROGRESS"
        }
    }

  4. Optionally verify the identity and authenticity of your new HSM https://docs.aws.amazon.com/cloudhsm/latest/userguide/verify-hsm-identity.html

  5. To initialize the cluster, you must download and sign a certificate signing request (CSR) that is generated by the cluster's first HSM. Download the CSR from the AWS Console or via the AWS CLI:

    aws cloudhsmv2 describe-clusters --filters clusterIds=cluster ID \      --output text \      --query 'Clusters[].Certificates.ClusterCsr' \      > ClusterCsr.csr

  6. Choose an appropriate RSA 2048 or RSA 4096 key and self-signed certificate to sign the HSM CSR. For a production cluster, AWS recommends a secured offsite and offline HSM. For a demonstration or test cluster, you can create a key and self-signed certificate with the following openssl commands:

    openssl genrsa -aes256 -out customerCA.key 2048
    openssl req -new -x509 -days 3652 -key customerCA.key -out customerCA.crt

  7. Sign the cluster CSR using the key and self-signed cert from the previous step. A demonstration openssl command to sign the CSR is:

    openssl x509 -req -days 3652 -in ClusterCsr.csr \      -CA customerCA.crt \      -CAkey customerCA.key \      -CAcreateserial \      -out CustomerHsmCertificate.crt

  8. Initialize the CloudHSM cluster by uploading the signed cert to the AWS Console or with the following AWS CLI command:

    aws cloudhsmv2 initialize-cluster --cluster-id cluster ID \      --signed-cert file://CustomerHsmCertificate.crt \      --trust-anchor file://customerCA.crt
    {
        "State": "INITIALIZE_IN_PROGRESS",
        "StateMessage": "Cluster is initializing. State will change to INITIALIZED upon completion."
    }

  9. A security group with the same name as your CloudHSM cluster will have been automatically created when you created the cluster. Attach this security group to the EC2 instance where you will run your Teleport Auth Server to allow traffic between the Auth Server and your HSM.

  10. On the Auth Server EC2 instance, install the CloudHSM CLI for the CloudHSM Client SDK 5. https://docs.aws.amazon.com/cloudhsm/latest/userguide/gs_cloudhsm_cli-install.html

    Bootstrap the CLI by configuring the IP address of the new HSM:

    sudo /opt/cloudhsm/bin/configure-cli -a HSM IP address

  11. Copy the self-signed certificate from step 6 (customerCA.crt) to the EC2 instance, save it at /opt/cloudhsm/etc/customerCA.crt.

  12. Activate the CloudHSM cluster with the CloudHSM CLI by creating an admin user with a new password:

    $ sudo /opt/cloudhsm/bin/cloudhsm-cli interactive
aws-cloudhsm > cluster activate
Enter password:
Confirm password:
{
  "error_code": 0,
  "data": "Cluster activation successful"
}

  13. Login to the CloudHSM CLI as the new admin user and create a Crypto User to be used by Teleport. Remember this new password because Teleport will use it later to authenticate to the PKCS#11 library.

    aws-cloudhsm > login --username admin --role admin
Enter password:
{
  "error_code": 0,
  "data": {
    "username": "admin",
    "role": "admin"
  }
}
aws-cloudhsm > user create --username teleport --role crypto-user
Enter password:
Confirm password:
{
  "error_code": 0,
  "data": {
    "username": "teleport",
    "role": "crypto-user"
  }
}
aws-cloudhsm > quit

  14. Install the PKCS#11 library for the Client SDK 5 on the same Auth Server EC2 instance https://docs.aws.amazon.com/cloudhsm/latest/userguide/pkcs11-library-install.html

    Bootstrap the PKCS#11 library by configuring the HSM IP address. If you only have one HSM in the cluster, you must include the --disable-key-availability-check flag.

    sudo /opt/cloudhsm/bin/configure-pkcs11 --disable-key-availability-check -a HSM IP address

  1. Install the YubiHSM2 SDK.

  2. Start yubihsm-connector with debug logging enabled. This is a background process that you will need to keep running to facilitate connections to your YubiHSM2.

    yubihsm-connector -d
    DEBU[0000] preflight complete                            cert= config= key= pid=73502 seccomp=false serial= syslog=false timeout=0s version=3.0.3
    DEBU[0000] takeoff                                       TLS=false listen="localhost:12345" pid=73502

  3. Use yubihsm-shell to create a new authentication key to be used by Teleport with the necessary capabilities.

    YubiHSM2 comes with a factory default authentication key at slot 1 with password password. You should replace and delete it as recommended by Yubico.

    When creating the authentication key to be used by Teleport, the password must have at least 8 characters. The example hunter22 is used here.

    $ yubihsm-shell
Using default connector URL: http://localhost:12345
yubihsm> connect
Session keepalive set up to run every 15 seconds
yubihsm> session open 1 password
Created session 0

# Create an Authenticate Key for Teleport
yubihsm> put authkey 0 0 "Teleport Auth Key" 1 generate-asymmetric-key:sign-pkcs:delete-asymmetric-key sign-pkcs:sign-pss:decrypt-pkcs:decrypt-oaep hunter22
Stored Authentication key 0x85cf

# Make sure you can open a session with the new authentication key and password
yubihsm> session open 0x85cf hunter22
Created session 1

# Delete the factory default authentication key
yubihsm> delete 0 1 authentication-key

    Take note of the slot number of the new authentication key. In the above example this is the hex value 0x85cf. This will need to be included in your Teleport configuration file in a later step.

  4. Create a yubihsm_pkcs11.conf file to configure the address and port that yubihsm-connector is listening on and enable debug logging:

    # /etc/yubishm_pkcs11.conf
connector = https://127.0.0.1:12345
debug

  5. Set the environment variable YUBIHSM_PKCS11_CONF to the path of your configuration file. This will be read by the PKCS#11 module and needs to be set in the Teleport auth server's environment.

    export YUBIHSM_PKCS11_CONF=/etc/yubihsm_pkcs11.conf

Step 2/5. Configure Teleport

To configure Teleport to use an HSM for all CA private key generation, storage, and signing, include the ca_key_params section in /etc/teleport.yaml on the auth server.

# /etc/teleport.yaml
teleport:
  ...

auth_service:
  enabled: true
  ...

  ca_key_params:
    pkcs11:
      # this is the default install location of the PKCS#11 module for the
      # CloudHSM Client SDK 5
      module_path: /opt/cloudhsm/lib/libcloudhsm_pkcs11.so

      # token_label should always be "hsm1" for the CloudHSM SDK 5
      # if you prefer to use SDK version 3, replace this with "cavium"
      token_label: "hsm1"

      # pin should be set to the username and password of the Crypto User
      # created earlier in the guide
      pin: "<CU_username>:<CU_password>"
      # pin_path can optionally be used to read the pin from a file
      # pin_path: /path/to/pin_file

# /etc/teleport.yaml
teleport:
  ...

auth_service:
  enabled: true
  ...

  ca_key_params:
    pkcs11:
      # this is the default installation path for Yubico's PKCS#11 module
      module_path: /usr/local/lib/pkcs11/yubihsm_pkcs11.dylib
      # slot_number should always be set to 0 for YubiHSM2
      slot_number: 0
      # pin should be the (hex) slot number of your authentication key,
      # concatenated with the password
      pin: "85cfhunter22"
      # pin_path can optionally be used to read the pin from a file
      # pin_path: /path/to/pin_file

Step 3/5. (Re)start Teleport Auth

If this is a new auth server which has not been started yet, starting a brand new cluster with an empty backend, HSM keys will be automatically generated at startup and no further action is required, skip to step 5. Otherwise, continue reading.

If you are connecting an HSM to an existing Teleport cluster, restart the auth server for the configuration changes to take effect. New CA keys will automatically be generated in the HSM. For these keys to be trusted by the rest of the cluster you will need to perform a CA rotation, see Step 4. The auth server will not perform any signing operations until the rotation has started. In an HA cluster you should add the HSM to the auth configuration one server at a time, and do not route any traffic to the auth server where the HSM is currently being added.

Step 4/5. Certificate Rotation with HSM

When adding a new HSM to an existing Teleport cluster, or adding a new HSM-connected Auth server to an HA Teleport cluster, you will need to rotate all Certificate Authorities in order for new certificates to by issued and trusted.

tctl status will print a warning if CA rotation is required:

tctl status
WARNING: One or more auth servers has a newly added or removed HSM or KMS configured. You should not route traffic to that server until a CA rotation has been completed.
Cluster      cluster-one
Version      13.0.0-alpha.1
host CA      never updated
user CA      never updated
db CA        never updated
openssh CA   never updated
jwt CA       never updated
saml_idp CA  never updated
CA pin       sha256:29afff5a1c1231375525b8a9b98ea7a44a760b7e65de3ca2e04d72db39b1672b
CA pin       sha256:e758c8f0f6cd95116d5da8171e0ff4adfa99dab3b1f171bfe854070884955524

teleport start will also print a warning during startup if any CA needs to be rotated. Until rotation is completed, the auth server will not sign any new certificates (except the Admin certificate used by tctl which will be signed by a temporary HSM key).

CA rotation can be performed manually or semi-automatically, see our admin guide on Certificate rotation. All CAs listed in the output of tctl status must be rotated.

Step 5/5. Confirm that Teleport is using your HSM

You are all set! Check the teleport logs for Creating new HSM key pair to confirm that the feature is working. You can also check that keys were created in your HSM using your HSM's admin tool.

Have a suggestion or can’t find something?
IMPROVE THE DOCS