Using HPKP to Bootstrap SSH Trust and Improve User Experience
The Problem: Bootstrapping Trust to Remediate Trust on First Use
The Solution: SSH Certificate Authority Pinning
Teleport cybersecurity blog posts and tech news
Want to give SSH CA pinning a try?

Home - Teleport Blog - How to Use Certificate Pinning to Improve UX - Feb 26, 2019

How to Use Certificate Pinning to Improve UX

by Russell Jones

Using HPKP to Bootstrap SSH Trust and Improve User Experience

Kubernetes kubeadm users might have noticed a familiar flag when adding nodes to a Teleport cluster starting in 3.1: ca-pin. This new ability to pin a Teleport SSH Certificate Authority is used not only to help bootstrap trust, but also used to remove the need for out-of-band distribution of cryptographic material to machines running the Teleport SSH & kubectl recording daemon.

$ tctl nodes add
The invite token: cf24c30b6d9f9c780c219cd5af000856
This token will expire in 30 minutes

Run this on the new node to join the cluster:

> teleport start \
   --roles=node \
   --token=cf24c30b6d9f9c780c219cd5af000856 \
   --ca-pin=sha256:79a6c30c7e7310b0a66778...f858ab846f7021f1429eb6 \
   --auth-server=10.12.0.6:3025

Figure 1: Output of tctl add nodes on my machine. Don't hack me.

The Problem: Bootstrapping Trust to Remediate Trust on First Use

To secure a connection between a client and server, one or both sides need to authenticate each other. For example, when using webmail the client (web browser) uses public key cryptography to authenticate the server (Gmail) by validating the x509 certificate presented by the server is signed by a trusted party known as a Certificate Authority (CA). The list of authorities to trust is usually bundled into the operating system or the web browser itself and is transparent to the user. It's sometimes called a trust store. For the other side of the connection, the server validates the client by asking the client to present some kind of secret token, typically a password, that was established ahead of a time (typically during registration).

When a Teleport Node asks a Teleport Auth Server to join a Teleport ssh or kubectl cluster, a similar exchange occurs. The big difference is that because each Teleport cluster is itself managing a private certificate authority, and furthermore the CA is itself often automatically rotated, the private Teleport CA is not in the joining node's operating system trust store. So how does a Teleport ssh daemon verify the identity of the Teleport server?

One way is to export the x509 certificate of the CA like in Figure (2) and place it at a well known location on disk. Then the client can read it in and use it to validate the certificate presented by the Auth Server. Sophisticated ops teams typically have infrastructure in-place to distribute internal certificate authorities to all nodes, but what about someone starting out with Teleport or maybe customers that don't have the more sophisticated infrastructure yet. How do they authenticate the connection to the Auth Server?

Figure 2: The certificate of the CA on my machine.

The Solution: SSH Certificate Authority Pinning

Inspiration for the solution came in the form of Kubernetes. Users of kubeadm are probably familiar with the

--discovery-token-ca-cert-hash flag, which is essentially what Teleport's --ca-pin flag is.

If you want to go to the source and read more about this, you can read the Kubernetes documentation, GitHub issue, or design document.

If you want a quick rundown, this is how Teleport's SSH certificate authority pinning works:

First the Teleport node connects to the Teleport Auth Server over an insecure connection and asks for the public key of the private CA. The Teleport node process parses the certificate, then calculates the SHA256 hash of the RawSubjectPublicKeyInfo field which consists of the public key and public key algorithm. If this hash value does not match the value in the

--ca-pin parameter, Teleport bails out because the connection may be compromised in some manner.

If the hash does match, Teleport attempts to build a fully secured connection the Auth Server using the CA obtained over the insecure connection to validate the certificate presented by the Auth Server. If Teleport's node agent can not validate the certificate chain, once again Teleport will bail out because the connection may be compromised in some manner. If Teleport can validate the chain, it knows it has connected to its legitimate Teleport Auth Server.

Figure 3: Visualizing a node joining a cluster.

We're pretty excited about this feature because not only does it hit an important security goal of mutual authentication, it's also downright user friendly.