Troubleshooting Application Access

This section describes common issues that you might encounter in managing access to applications with Teleport and how to work around or resolve them.

Unable to access applications

The most common issues that prevent access to applications involve Cross-Site Request Forgery (CSRF) or Cross-Origin Resource Sharing (CORS) errors.

Cross-Site Request Forgery is a type of attack that uses an authenticated user's identity to perform operations without the user's knowledge. For example, a CSRF attack might transfer funds, change passwords, or make purchases by issuing a forged request with the user's credentials. Browsers and applications have checks to prevent these types of attacks. However, the checks can also block legitimate requests under certain conditions.

Symptom

If your browser can't create a secure cookie or can't authorize your login from a previously-created secure cookie, you might see the following error:

Invalid or missing CSRF token

This error can be caused by ad-blocking or script-blocking extensions or by the browser itself. For access to applications through Teleport, you might see this error with most often with applications—like Grafana and ArgoCD—that use WebSockets extensively and in browsers where cross-site scripting restrictions require traffic to be explicitly allowed.

Issues with Cross-Site Request Forgery (CSRF) or Cross-Origin Resource Sharing (CORS) usually result in a loss of application functionality, errors in the application itself indicating that traffic isn't being permitted, or application logs that indicate CORS or CSRF errors.

Solution

In most cases, you can fix these types of issues by adding explicit rewrite settings for the Origin and Host headers in the Teleport configuration file for each application.

To fix CSRF or CORS issues:

1. Open the /etc/teleport.yaml file that contains the application configuration in a text editor.

2. Add a rewrite.headers section similar to the following grafana example:

   app_service:
     enabled: true
     apps:
     - name: grafana
       uri: http://localhost:3000
       public_addr: grafana.teleport.example.com
       rewrite:
         headers:
         - "Origin: https://grafana.teleport.example.com" # Teleport application subdomain prepended with "https://"
         - "Host: grafana.teleport.example.com" # Teleport application subdomain itself
   

3. Save your changes and restart the Teleport service.

To fix CSRF or CORS issues if you deploy applications using Kubernetes and teleport-kube-agent:

1. Open the teleport/examples/chart/teleport-kube-agent/values.yaml file that contains the application configuration in a text editor.

2. Locate the apps section in the values.yaml file.

   # Details of at least one app to be proxied. Example:
   # apps:
   #  - name: grafana
   #    uri: http://localhost:3000
   apps: []
   

3. Add a rewrite.headers section similar to the following grafana example:

   app_service:
     enabled: true
     apps:
     - name: grafana
       uri: http://localhost:3000
       public_addr: grafana.teleport.example.com
       rewrite:
         headers:
         - "Origin: https://grafana.teleport.example.com" # Teleport application subdomain prepended with "https://"
         - "Host: grafana.teleport.example.com" # Teleport application subdomain itself
   

Untrusted certificate errors

By default, the certificates presented by the Teleport Proxy Service must be trusted and issued by a recognized certificate authority.

Symptom

If you have created self-signed certificates or use certificates signed by an unrecognized certificate authority, you might see an error similar to the following:

ERROR: "unable to verify HTTPS certificate chain in : \x1b[31mERROR: \x1b[0mWARNING:"

  The proxy you are connecting to has presented a certificate signed by a
  unknown authority. This is most likely due to either being presented
  with a self-signed certificate or the certificate was truly signed by an
  authority not known to the client.

Solution

If you have properly created a root certificate authority and a self-signed certificate, you can use the --insecure command-line option to allow clients to accept the certificate. For example, you can start Teleport with a self-signed certificate by running a command similar to the following:

sudo teleport start --config=/etc/teleport.yaml --insecure

If you have your own certificate authority that you would like to use to validate the certificate chain presented by the Teleport Proxy Service, you can manually set the SSL_CERT_FILE or SSL_CERT_DIR environment variable on the command line. For example:

sudo SSL_CERT_FILE=path-to-rootCA-pem teleport start --config=/etc/teleport.yaml

You should specify the SSL_CERT_FILE and SSL_CERT_DIR environment variables as command-line options because sudo doesn't inherit environment variables by default.

Request headers too large

By default, Teleport embeds the user's Teleport roles and traits in the JWT that is issued for applications. If your Teleport user has a large number of roles or traits, this can cause the JWT to become too large which will result in a failed request.

Symptom

When attempting to connect to an HTTP app behind Teleport, you see an error that states request header fields too large.

Solution

If the application that you're protecting with Teleport does not need to know about a user's Teleport roles or traits, you can configure Teleport to omit this information from the JWT. This will result in a smaller JWT that is less likely to exceed the limit.

This configuration is available under the jwt_claims property of the application's rewrite configuration. See Web Application Access for details.