How to use Let's Encrypt with an SSH Bastion
This blog post is about configuring the Teleport SSH proxy (aka, bastion) to use x509 certificates (AKA SSL or HTTPS) issued by Let’s Encrypt. Let’s be careful and not confuse x509 certificates with SSH certificates. Teleport itself is a clustered, certificate-based SSH server (i.e., the Teleport Auth Service acts as an SSH certificate authority (CA) for your organization.)
A Teleport user must request an SSH certificate before connecting to any Teleport nodes. Typically, the certificate automatically expires at the end of the day, so the process of issuing a user SSH certificate happens daily. A user can request their SSH certificate from a Teleport proxy server:
$ tsh login --proxy=proxy.example.com
The Teleport proxy server behind
proxy.example.com uses the
HTTPS protocol (on
port 3080 by default) to serve the SSH certificate back to the
tsh client. This
means that a Teleport proxy must use a valid x509 certificate to securely serve
via HTTPS. Below, we’ll go through the steps for using Let’s Encrypt to provide this x509 certificate.
From the Let’s Encrypt website:
Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. It is a service provided by the Internet Security Research Group (ISRG).
In this post we won’t go into a lot of detail about how Let’s Encrypt works but the most important highlights are:
- The process of validating a certificate domain is automated.
- A certificate issued by Let’s Encrypt is valid for just a couple of weeks and must be renewed frequently.
We’ll use domain validation based on HTTP ACME challenge,
which works by “provisioning an HTTP resource under a well-known URI”, in our
http://proxy.example.com. This means the port 80 on the Teleport
Proxy server machine must be available and accessible by Let’s Encrypt servers.
First, you need to install CertBot, the official Let’s Encrypt client, which is a command line utility we’ll use below to receive a x509 certificate for proxy.example.com domain. Using Debian 9, it’s as simple as typing:
$ sudo apt-get install certbot
You can follow one of the official tutorials on how to obtain a certificate from Let’s Encrypt but here’s how we recommend doing it:
sudo certbot certonly --standalone \ -d proxy.example.com \ -n \ --agree-tos \ --email=[email protected]
Here is some more detail:
||Use standalone domain validation. This will cause certbot to launch a web server listening on port 80 to respond to a ACME challenge. This is also why we have to run this command under sudo, otherwise certbot will not be able to bind to this port|
||Domain we’re requesting a certificate for. This must be the same name which will be used with
||Tells Let’s Encrypt that you agree with their Terms of Service.|
|Your email address|
By default, this command will create
/etc/letsencrypt/ directory and populate
it with all kinds of interesting information. We are mostly interested in
$ ls -lh /etc/letsencrypt/live/proxy.example.com lrwxrwx--- 1 root root 43 Jan 18 14:28 cert.pem lrwxrwx--- 1 root root 44 Jan 18 14:28 chain.pem lrwxrwx--- 1 root root 48 Jan 18 14:28 fullchain.pem lrwxrwx--- 1 root root 46 Jan 18 14:28 privkey.pem -rw-r--r-- 1 root root 543 Jan 18 14:22 README
We’ll need the private key and the “full chain” certificate to configure the Teleport Proxy.
/etc/teleport.yaml and make sure it contains the following section:
proxy_service: enabled: yes https_cert_file: /etc/letsencrypt/live/proxy.example.com/fullchain.pem https_key_file: /etc/letsencrypt/live/proxy.example.com/privkey.pem
Restart the Teleport proxy and it should be able to serve a valid HTTPS content
As we mentioned earlier, a certificate issued by Let’s Encrypt is valid only for a few weeks, which means it must be renewed periodically. You can run this command periodically (see letsencrypt rate limit guidelines) as root:
$ certbot certonly --standalone -d proxy.example.com -n --force-renewal
As you can see, it forces the renewal of the certificate. The Teleport proxy must be restarted after it runs. Another option is to run
it more frequently but you will have to drop
--force-renewal flag. CertBot is
smart enough not to touch a certificate if it’s still valid.
You can configure this command to run as a cron job or you can use systemd timers. The Certbot package from Debian 9, which we used for this post, comes with the pre-configured systemd timer which you can configure by editing these two files:
To recap, here’s what we have done to configure an SSH Bastion to serve HTTPS content using a certificate from Let’s Encrypt:
- Install certbot on the same server where a Teleport proxy is running.
- Make sure the port 80 is available and accessible by Let’s Encrypt servers.
- Use “standalone” domain verification method.
- Update Teleport configuration in
- Create a systemd timer which renews the certificate and restarts Teleport daemon.
- How to manage SSH keys?
- SSH using Github team membership via OAuth2 + 2FA
- How to record SSH sessions with OpenSSH servers