Scaling

Report an Issue

Is this page helpful?

How can we improve this page?

View as Markdown

This section explains the recommended configuration settings for large-scale self-hosted deployments of Teleport.

tip

Teleport Enterprise Cloud takes care of this setup for you so you can provide secure access to your infrastructure right away.

Get started with a free trial of Teleport Enterprise Cloud.

Hardware recommendations

Set up Teleport with a High Availability configuration.

Scenario	Max Recommended Count	Proxy Service	Auth Service	AWS Instance Types
Teleport SSH Nodes connected to Auth Service	10,000	2x 4 vCPUs, 8GB RAM	2x 8 vCPUs, 16GB RAM	m4.2xlarge
Teleport SSH Nodes connected to Auth Service	50,000	2x 4 vCPUs, 16GB RAM	2x 8 vCPUs, 16GB RAM	m4.2xlarge
Teleport SSH Nodes connected to Proxy Service through reverse tunnels	10,000	2x 4 vCPUs, 8GB RAM	2x 8 vCPUs, 16+GB RAM	m4.2xlarge

To host Teleport on Kubernetes see Hosting on Kubernetes for how to express these recommendations as teleport-cluster Helm chart values.

Auth Service and Proxy Service Configuration

Upgrade Teleport's connection limits from the default connection limit of 15000 to 65000.

# Teleport Auth Service and Proxy Service
teleport:
  connection_limits:
    max_connections: 65000

Agent configuration

Agents cache roles and other configuration locally in order to make access-control decisions quickly. By default agents are fairly aggressive in trying to re-initialize their caches if they lose connectivity to the Auth Service. In very large clusters, this can contribute to a "thundering herd" effect, where control plane elements experience excess load immediately after restart. Setting the max_backoff parameter to something in the 8-16 minute range can help mitigate this effect:

teleport:
  cache:
    enabled: true
    max_backoff: 12m

Kernel parameters

Tweak Teleport's systemd unit parameters to allow a higher amount of open files:

[Service]
LimitNOFILE=65536

Verify that Teleport's process has high enough file limits:

cat /proc/$(pidof teleport)/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            65536                65536                files

Hosting on Kubernetes

We recommend hosting self-hosted Teleport clusters on Kubernetes with the teleport-cluster Helm chart. See Deploy Teleport on Kubernetes to get started. The sizing guidance above still applies; the sections below show how to express it as chart values.

Auth Service and Proxy Service sizing

The chart runs the Auth Service and Proxy Service as separate deployments. To run more than one replica of each, you must first meet the high-availability prerequisites that the default standalone chart does not provide:

• The Auth Service needs a highly available backend. The default standalone mode stores cluster state in SQLite on a ReadWriteOnce volume, which cannot be shared safely across pods. Use a cloud chart mode (aws, gcp, or azure), which disables local persistence automatically, or configure an external backend such as DynamoDB, Firestore, or etcd. With an external backend you must also set persistence.enabled: false and store session recordings on highly available storage such as S3 or GCS. Otherwise every Auth Service replica references the same ReadWriteOnce volume, causing multi-attach or scheduling failures. See Storage backends and the High Availability guide.
• The Proxy Service needs a shared TLS certificate source. Unless you enable cert-manager (highAvailability.certManager), provide an existing secret (tls.existingSecretName), or use an ingress (ingress.enabled), the chart keeps the Proxy Service at a single replica regardless of the value below.

Once those prerequisites are in place, set the replica count with highAvailability.replicaCount, which corresponds to the 2x in the hardware recommendations:

highAvailability:
  replicaCount: 2

Set pod resources with the top-level resources value. To size the Auth Service and Proxy Service independently, override resources under the auth and proxy values, which are merged with and take precedence over the chart-scoped values:

# Applies to both Auth Service and Proxy Service pods unless overridden below.
resources:
  requests:
    cpu: "8"
    memory: "16Gi"
  limits:
    memory: "16Gi"

# Size the Proxy Service pods separately.
proxy:
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
    limits:
      memory: "8Gi"

Do not set CPU limits

Setting CPU limits on Teleport pods is an anti-pattern: Teleport becomes unstable once CPU throttling begins. Set CPU requests to reserve capacity, but leave CPU limits unset.

Connection limits and Teleport configuration

Apply Teleport configuration such as the connection limit described above through the auth.teleportConfig and proxy.teleportConfig values instead of a teleport.yaml file. These are merged with the chart-generated configuration:

auth:
  teleportConfig:
    teleport:
      connection_limits:
        max_connections: 65000
proxy:
  teleportConfig:
    teleport:
      connection_limits:
        max_connections: 65000

DynamoDB configuration

When using Teleport with DynamoDB, we recommend using on-demand provisioning. This allows DynamoDB to scale with cluster load.

For customers that can not use on-demand provisioning, we recommend at least 250 WCU and 100 RCU for 10k clusters.

etcd

When using Teleport with etcd, we recommend you do the following.

• For performance, use the fastest SSDs available and ensure low-latency network connectivity between etcd peers. See the etcd Hardware recommendations guide for more details.
• For debugging, ingest etcd's Prometheus metrics and visualize them over time using a dashboard. See the etcd Metrics guide for more details.

During an incident, we may ask you to run etcdctl, test that you can run the following command successfully.

etcdctl \    --write-out=table \    --cacert=/path/to/ca.cert \    --cert=/path/to/cert \    --key=/path/to/key.pem \    --endpoints=127.0.0.1:2379 \    endpoint status

Supported Load

The tests below were performed against a Teleport Cloud tenant which runs on instances with 8 vCPU and 32 GiB memory and has default limits of 4CPU and 4Gi memory.

Concurrent Logins

Resource Type	Login Command	Logins	Failure
SSH	tsh login	2000	Auth CPU Limits exceeded
Application	tsh app login	2000	Auth CPU Limits exceeded
Database	tsh db login	2000	Auth CPU Limits exceeded
Kubernetes	tsh kube login && tsh kube credentials	2000	Auth CPU Limits exceeded

Sessions Per Second

Resource Type	Sessions	Failure
SSH	1000	Auth CPU Limits exceeded
Application	2500	Proxy CPU Limits exceeded
Database	40	Proxy CPU Limits exceeded
Kubernetes	50	Proxy CPU Limits exceeded

Windows Desktop Service

Windows Desktop sessions can vary greatly in resource usage depending on the applications being used. The primary factor affecting resource usage per session is how often the screen is updated. For example, a session playing a video in full screen mode will consume significantly more resources than a session where the user is typing in a text editor.

We measured the resource usage of sessions playing fullscreen videos to get the worst-case estimate for resource requirements. We then inferred resource requirements for more standard use cases on the basis of those measurements.

Worst Case:

• 1/12 vCPU per concurrent session
• 42 MB RAM per concurrent session

Typical Case:

• 1/240 vCPU per concurrent session
• 2 MB RAM per concurrent session

From these estimates we calculated the following table of recommendations based on the expected maximum number of concurrent sessions:

Concurrent users	CPU (vCPU, low to high)	Memory (GB, low to high)
1	1	0.5
10	1	0.5 to 1
100	1 to 8	1 to 8
1000	4 to 96	4 to 64

To avoid service interruptions, we recommend leaning towards the higher end of the recommendations to start while monitoring your resource usage, and then scaling resources based on measured outcomes.

Note that you are not limited to a single windows_desktop_service, and can connect multiple to your cluster in order to spread resources out over multiple logical machines.