Troubleshooting Application Access

This section describes common issues that you might encounter in managing access to applications with Teleport and how to work around or resolve them.

Unable to access applications

The most common issues that prevent access to applications involve Cross-Site Request Forgery (CSRF) or Cross-Origin Resource Sharing (CORS) errors.

Cross-Site Request Forgery is a type of attack that uses an authenticated user's identity to perform operations without the user's knowledge. For example, a CSRF attack might transfer funds, change passwords, or make purchases by issuing a forged request with the user's credentials. Browsers and applications have checks to prevent these types of attacks. However, the checks can also block legitimate requests under certain conditions.

Symptom

If your browser can't create a secure cookie or can't authorize your login from a previously-created secure cookie, you might see the following error:

Invalid or missing CSRF token

This error can be caused by ad-blocking or script-blocking extensions or by the browser itself. For access to applications through Teleport, you might see this error with most often with applications—like Grafana and ArgoCD—that use WebSockets extensively and in browsers where cross-site scripting restrictions require traffic to be explicitly allowed.

Issues with Cross-Site Request Forgery (CSRF) or Cross-Origin Resource Sharing (CORS) usually result in a loss of application functionality, errors in the application itself indicating that traffic isn't being permitted, or application logs that indicate CORS or CSRF errors.

In most cases, you can fix these types of issues by adding explicit rewrite settings for the Origin and Host headers in the Teleport configuration for each application.

Solution 1: Application Service configuration file

To fix CSRF or CORS issues if you use statically configured apps in /etc/teleport.yaml:

1. Open the /etc/teleport.yaml file that contains the application configuration in a text editor.

2. Add a rewrite.headers section similar to the following grafana example:

app_service:
  enabled: true
  apps:
  - name: grafana
    uri: http://localhost:3000
    public_addr: grafana.teleport.example.com
    rewrite:
      headers:
      - "Origin: https://grafana.teleport.example.com" # Teleport application subdomain prepended with "https://"
      - "Host: grafana.teleport.example.com" # Teleport application subdomain itself

3. Save your changes and restart the Teleport service.

Solution 2: teleport-kube-agent values file

To fix CSRF or CORS issues if you deploy applications using Kubernetes and teleport-kube-agent:

1. Open the teleport/examples/chart/teleport-kube-agent/values.yaml file that contains the application configuration in a text editor.

2. Locate the apps section in the values.yaml file.

# Details of at least one app to be proxied. Example:
# apps:
#  - name: grafana
#    uri: http://localhost:3000
apps: []

3. Add a rewrite.headers section similar to the following grafana example:

apps:
- name: grafana
  uri: http://localhost:3000
  public_addr: grafana.teleport.example.com
  rewrite:
    headers:
    - "Origin: https://grafana.teleport.example.com" # Teleport application subdomain prepended with "https://"
    - "Host: grafana.teleport.example.com" # Teleport application subdomain itself

Solution 3: Dynamic app configuration

To fix CSRF or CORS issues if you deploy applications with dynamic configuration:

1. Edit your dynamic app configuration to include the rewrite.headers section:

kind: app
version: v3
metadata:
  name: grafana
  labels:
    env: dev
spec:
  uri: http://localhost:3000
  public_addr: grafana.teleport.example.com
  rewrite:
    headers:
    - name: "Origin"
      value: "https://grafana.teleport.example.com" # Teleport application subdomain prepended with "https://"
    - name: "Host"
      value: "grafana.teleport.example.com" # Teleport application subdomain itself

Solution 4: Kubernetes app autodiscovery

To fix CSRF or CORS issues if you deploy applications using Kubernetes autodiscovery:

1. Edit your Kubernetes Service configuration to include the rewrite.headers section:

apiVersion: v1
kind: Service
metadata:
  annotations:
    teleport.dev/app-rewrite: |
      headers:
      - name: "Origin"
        value: "https://grafana.teleport.example.com" # Teleport application subdomain prepended with "https://"
      - name: "Host"
        value: "grafana.teleport.example.com" # Teleport application subdomain itself

Untrusted certificate errors

By default, the certificates presented by the Teleport Proxy Service must be trusted and issued by a recognized certificate authority.

Symptom

If you have created self-signed certificates or use certificates signed by an unrecognized certificate authority, you might see an error similar to the following:

ERROR: "unable to verify HTTPS certificate chain in : \x1b[31mERROR: \x1b[0mWARNING:"

  The proxy you are connecting to has presented a certificate signed by a
  unknown authority. This is most likely due to either being presented
  with a self-signed certificate or the certificate was truly signed by an
  authority not known to the client.

Solution

If you have properly created a root certificate authority and a self-signed certificate, you can use the --insecure command-line option to allow clients to accept the certificate. For example, you can start Teleport with a self-signed certificate by running a command similar to the following:

sudo teleport start --config=/etc/teleport.yaml --insecure

If you have your own certificate authority that you would like to use to validate the certificate chain presented by the Teleport Proxy Service, you can manually set the SSL_CERT_FILE or SSL_CERT_DIR environment variable on the command line. For example:

sudo SSL_CERT_FILE=path-to-rootCA-pem teleport start --config=/etc/teleport.yaml

You should specify the SSL_CERT_FILE and SSL_CERT_DIR environment variables as command-line options because sudo doesn't inherit environment variables by default.

Request headers too large

By default, Teleport embeds the user's Teleport roles and traits in the JWT that is issued for applications. If your Teleport user has a large number of roles or traits, this can cause the JWT to become too large which will result in a failed request.

Symptom

When attempting to connect to an HTTP app behind Teleport, you see an error that states request header fields too large.

Solution

If the application that you're protecting with Teleport does not need to know about a user's Teleport roles or traits, you can configure Teleport to omit this information from the JWT. This will result in a smaller JWT that is less likely to exceed the limit.

This configuration is available under the jwt_claims property of the application's rewrite configuration. See Web Application Access for details.