Deploying self-hosted Highly Available Teleport on an On-Prem data center.

This article is a how-to guide to deploy self-hosted Teleport in a highly available mode on an on-prem environment using HAproxy, etcd, and minIO. But please note teleport is cloud and platform agnostic so you can run teleport anywhere. We also do have a SaaS offering. If you are new to teleport here is a quick into to teleport

This setup we will use:

• Minio S3 for session recordings
• Etcd for the cluster state storage
• HA Proxy for Load Balancer

HAProxy Architecture Overview

To maintain teleport deployment to be highly available and fault tolerant we have to run multiple instances of teleport and this has to be load balanced so we are using HAProxy to do the load balancing. Again now the HAProxy has to be highly available to achieve High availability  HAProxy has to be set up on two different VM’s running in parallel one VM acts as the primary and the other one acts as a secondary. A floating IP is assigned to either one of the HAProxy VM based on the health.Keepalived daemon can be used to monitor the health of the HAProxy service and they will take care of failover to the secondary HAProxy instance and also changes the floating IP assignment to the secondary HAProxy instance.Etcd is the backend storage for teleport and this can be hosted in 2 or three VM and is connected to the auth server as a cluster.

Prerequisites

Here are a few requirements for this deployment,

1. Etcd cluster with 2 or 3 nodes, which are authenticated using TLS cert, key, and CA cert.

2. certificate management for teleport proxy service, here I am using Let's Encrypt.

3. 2 VM’s for Auth

4. 2 VM’s for Proxy

5. 2 instances of HAProxy and keepalive installed and configured. 6. Floating Ip for HAProxy.7. Minio or Ceph for session recordings.

Build the cluster in this order for a successful deployment.

etcd --> Auth servers → Add HAProxy backends --> Proxy servers → Add HAProxy backends

Deploy Auth Service

Note: Ensure you have the etcd cluster ready and those are accessible on port 2379 before this step. Now, let's start building the auth servers. Follow the instructions below,

1. Download the teleport binaries to the auth VM and install them as daemons.

- Here is the guide to installing teleport https://goteleport.com/docs/installation/#installation-instructions- Install the systemd service for teleport by running the following command `sudo teleport install systemd -o /etc/systemd/system/teleport.service`

2. Create the auth config file /etc/teleport.yaml

teleport:
  nodename: ip-10-0-0-28.ec2.internal
  advertise_ip: 10.0.0.28
  log:
    output: stderr
    severity: DEBUG
  data_dir: /var/lib/teleport

  storage:
    type: etcd
    peers: ["https://10.0.0.20:2379", "https://10.0.0.21:2379","https://10.0.0.22:2379"]
    prefix: /teleport/
    tls_key_file: /var/lib/etcd_client/tls.key
    tls_cert_file: /var/lib/etcd_client/tls.crt
    tls_ca_file: /var/lib/etcd_client/ca.crt
    audit_sessions_uri:'s3://example.com/path/to/bucket?region=us-east-1'#minio

ssh_service:
  enabled: no

proxy_service:
  enabled: no

auth_service:
  enabled: yes
  keep_alive_interval: 1m
  keep_alive_count_max: 3
  listen_addr: 0.0.0.0:3025
  public_addr: 10.0.0.28:3025
  proxy_listener_mode: multiplex
  authentication:
    second_factor: off
  cluster_name: ha-proxy-teleport

3. Start the teleport auth service using the command `sudo systemctl start teleport.service`

4. follow the same steps 1 to 3 on the other Auth service VM.

5. Verify those auth clusters are connected to the etcd state backend by running the command `sudo tctl auth ls` on both the auth servers you should see an output with two auth servers.

HAProxy config for Auth

1. On the HA proxy add the IP address of the two auth servers to the config and make sure it's pointed to port 3025

here is a frontend and backend example config

frontend http_front
   bind *:3025
   mode tcp
   default_backend auth

backend auth
   mode tcp
   server auth-1 <auth-server-1-ip>:3025
   server auth-2 <auth-server-2-ip>:3025

Proxy Service

1. Download the teleport binaries to the auth VM and install them as daemons.

- Here is the guide to installing teleport https://goteleport.com/docs/installation/#installation-instructions- Install the systemd service for teleport by running the following command `sudo teleport install systemd -o /etc/systemd/system/teleport.service`

2. Create the token for proxy servers by running the following commands on one of the auth server `sudo tctl tokens add --type=proxy,node`

3. Create a file `/var/lib/teleport/token` with a token in that.

4. Create the auth config file /etc/teleport.yaml

here is an example Proxy config file

version: v3
teleport:
  auth_token: /var/lib/teleport/token # auth_token generated in the previous step.
  ca_pin: ###### # ca_pin generated in the previous step.
  nodename: ip-10-0-0-14
  advertise_ip: 10.0.0.14
  cache:
    type: in-memory
  log:
    output: stderr
    severity: DEBUG
  data_dir: /var/lib/teleport
  storage:
    type: dir
    path: /var/lib/teleport/backend
  auth_server: 10.0.0.5:3025 # HAProxy IP address or the DNS name pointing to port 3025

auth_service:
  enabled: no

ssh_service:
  enabled: no

proxy_service:
  enabled: yes
  listen_addr: 0.0.0.0:3023
  tunnel_listen_addr: 0.0.0.0:3080
  web_listen_addr: 0.0.0.0:443
  public_addr: haproxy.teleport-onboarding.com:443
  ssh_public_addr: haproxy.teleport-onboarding.com:3023
  tunnel_public_addr: haproxy.teleport-onboarding.com:443
  https_keypairs:
    - cert_file: /etc/certs/fullchain.pem #cert file generated for the domain name of public_addr 
      key_file: /etc/certs/privkey.pem #key file generated for the domain name of public_addr

5. Start the teleport proxy service using the command `sudo systemctl start teleport.service`

6. Follow the same steps on the other proxy service VM.

You can verify the proxy servers connected to auth by running the command `sudo tctl proxy ls` on auth VM and it should list both the proxy servers.

HAProxy config for Proxy

1. On the HA proxy config add the IP address of the two proxy servers to the config and make sure it's pointed to port 3025 along with the auth server config, here is a frontend and backend example config

frontend http_front
   bind 0.0.0.0:443
   mode tcp
   default_backend proxy

backend proxy
   balance roundrobin
   mode tcp
   server proxy-1 <proxy-server-1-ip>:443
   server proxy-2 <proxy-server-2-ip>:443

Now you should be able to reach the web UI using the proxy_addr that you have set on the configurations.

Conclusion

This guide has shown us how to build a Teleport cluster using on-prem technologies HAProxy, etcd, and Minio. In this use case, this cluster can be built on an On-Prem data center to leverage teleport to unify and secure access for all your users. Signup for a free trial of Enterprise to start using teleport or you can also start free using our open-source Community Edition.