Fork me on GitHub

Teleport

Access Requests with Mattermost

Improve

This guide will explain how to set up Teleport with Mattermost, an open source messaging platform. Teleport's Mattermost notifies individuals of Access Requests. Users can then approve and deny Access Requests by following the message link, making it easier to implement security best practices without compromising productivity.

Prerequisites

  • A running Teleport cluster, including the Auth Service and Proxy Service. For details on how to set this up, see our Enterprise Getting Started guide.

  • The tctl admin tool and tsh client tool version >= 11.0.3, which you can download by visiting the customer portal.

    tctl version

    Teleport v11.0.3 go1.19

    tsh version

    Teleport v11.0.3 go1.19

  • A Teleport Cloud account, which includes a running Auth Service and Proxy Service. If you do not have a Teleport Cloud account, visit the sign up page to begin your free trial.

  • The tctl admin tool and tsh client tool version >= 10.3.8. To download these tools, visit the Downloads page.

    tctl version

    Teleport v10.3.8 go1.19

    tsh version

    Teleport v10.3.8 go1.19

  • A Mattermost account with admin privileges. This plugin has been tested with Mattermost v7.0.1.
  • Either a Linux host or Kubernetes cluster where you will run the Mattermost plugin.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 11.0.3

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 10.3.8

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Step 1/8. Define RBAC resources

Before you set up the Mattermost plugin, you will need to enable Role Access Requests in the Proxy or Auth Service.

For the purpose of this guide, we will define an editor-requester role, which can request the built-in editor role, and an editor-reviewer role that can review requests for the editor role.

Create a file called editor-request-rbac.yaml with the following content:

kind: role
version: v5
metadata:
  name: editor-reviewer
spec:
  allow:
    review_requests:
      roles: ['editor']
---
kind: role
version: v5
metadata:
  name: editor-requester
spec:
  allow:
    request:
      roles: ['editor']
      thresholds:
        - approve: 1
          deny: 1

Create the roles you defined:

tctl create -f editor-request-rbac.yaml

role 'editor-reviewer' has been created

role 'editor-requester' has been created

Allow yourself to review requests by users with the editor-requester role by assigning yourself the editor-reviewer role. First, retrieve your user definition:

TELEPORT_USER=$(tsh status --format=json | jq -r .active.username)
tctl get user/${TELEPORT_USER?} > user.yaml

Edit user.yaml to add the editor-reviewer role:

 spec:
   roles:
   - access
   - editor
+  - editor-reviewer

Update your user definition:

tctl create -f user.yaml

Log out of Teleport and log in again. You will now have the ability to review requests for the editor role.

Create a user called myuser who has the editor-requester role. This user cannot edit your cluster configuration unless they request the editor role:

tctl users add myuser --roles=editor-requester

tctl will print an invitation URL to your terminal. Visit the URL and log in as myuser for the first time, registering credentials as configured for your Teleport cluster.

Later in this guide, you will have myuser request the editor role so you can review the request using the Teleport plugin.

Step 2/8. Install the Teleport Mattermost plugin

We recommend installing Teleport plugins on the same host as the Teleport Proxy Service. This is an ideal location as plugins have a low memory footprint, and will require both public internet access and Teleport Auth Service access.

curl -L -O https://get.gravitational.com/teleport-access-mattermost-v11.0.3-linux-amd64-bin.tar.gz
tar -xzf teleport-access-mattermost-v11.0.3-linux-amd64-bin.tar.gz
cd teleport-access-mattermost
./install

To install from source you need git and go installed. If you do not have Go installed, visit the Go downloads page.

Checkout teleport-plugins

git clone https://github.com/gravitational/teleport-plugins.git
cd teleport-plugins/access/mattermost
make

Run ./install from teleport-mattermost or place the executable in the appropriate /usr/bin or /usr/local/bin on the server installation.

docker pull public.ecr.aws/gravitational/teleport-plugin-mattermost:11.0.3

To allow Helm to install charts that are hosted in the Teleport Helm repository, use helm repo add:

helm repo add teleport https://charts.releases.teleport.dev

To update the cache of charts from the remote repository, run helm repo update:

helm repo update

Step 3/8. Create a user and role for the plugin

Teleport's Access Request plugins authenticate to your Teleport cluster as a user with permissions to list and read Access Requests. This way, plugins can retrieve Access Requests from the Teleport Auth Service and present them to reviewers.

Define a user and role called access-plugin by adding the following content to a file called access-plugin.yaml:

kind: role
version: v5
metadata:
  name: access-plugin
spec:
  allow:
    rules:
      - resources: ['access_request']
        verbs: ['list', 'read']
      - resources: ['access_plugin_data']
        verbs: ['update']
---
kind: user
metadata:
  name: access-plugin
spec:
  roles: ['access-plugin']
version: v2

Create the user and role:

tctl create -f access-plugin.yaml

As with all Teleport users, the Teleport Auth Service authenticates the access-plugin user by issuing short-lived TLS credentials. In this case, we will need to request the credentials manually by impersonating the access-plugin role and user.

If you are using tctl from the Auth Service host, you will already have impersonation privileges.

To grant your user impersonation privileges for access-plugin, define a role called access-plugin-impersonator by pasting the following YAML document into a file called access-plugin-impersonator.yaml:

kind: role
version: v5
metadata:
  name: access-plugin-impersonator
spec:
  allow:
    impersonate:
      roles:
      - access-plugin
      users:
      - access-plugin

Create the access-plugin-impersonator role:

tctl create -f access-plugin-impersonator.yaml

Retrieve your user definition:

TELEPORT_USER=$(tsh status --format=json | jq -r .active.username)
tctl get users/${TELEPORT_USER?} > myuser.yaml

Edit myuser.yaml to include the role you just created:

  roles:
   - access
   - auditor
   - editor
+  - access-plugin-impersonator

Apply your changes:

tctl create -f myuser.yaml

Log out of your Teleport cluster and log in again. You will now be able to generate signed certificates for the access-plugin role and user.

Step 4/8. Export the access plugin identity

Like all Teleport users, access-plugin needs signed credentials in order to connect to your Teleport cluster. You will use the tctl auth sign command to request these credentials for your plugin.

The format of the credentials depends on whether you have set up your network to give the plugin direct access to the Teleport Auth Service, or if all Teleport clients and services connect to the Teleport Proxy Service instead.

Environment type

The following tctl auth sign command impersonates the access-plugin user, generates signed credentials, and writes an identity file to the local directory:

tctl auth sign --user=access-plugin --out=auth.pem

Teleport's Access Request plugins listen for new and updated Access Requests by connecting to the Teleport Auth Service's gRPC endpoint over TLS.

The identity file, auth.pem, includes both TLS and SSH credentials. Your Access Request plugin uses the SSH credentials to connect to the Proxy Service, which establishes a reverse tunnel connection to the Auth Service. The plugin uses this reverse tunnel, along with your TLS credentials, to connect to the Auth Service's gRPC endpoint.

You will refer to this file later when configuring the plugin.

If your network allows your plugin to access the Auth Service directly, e.g., you are running the plugin on the Auth Service host, the plugin uses TLS credentials to connect to the Auth Service's gRPC endpoint and listen for new and updated Access Requests.

You can generate TLS credentials with the following command:

tctl auth sign --format=tls --user=access-plugin --out=auth

This command should result in three PEM-encoded files: auth.crt, auth.key, and auth.cas (certificate, private key, and CA certs respectively). Later, you will configure the plugin to use these credentials to connect to the Auth Service directly.

The following tctl auth sign command impersonates the access-plugin user, generates signed credentials, and writes an identity file to the local directory:

tctl auth sign --user=access-plugin --out=auth

Then create a Kubernetes secret:

kubectl create secret generic teleport-mattermost-identity --from-file=auth_id=auth.pem

Teleport's Access Request plugins listen for new and updated Access Requests by connecting to the Teleport Auth Service's gRPC endpoint over TLS.

The identity file, auth.pem, includes both TLS and SSH credentials. Your Access Request plugin uses the SSH credentials to connect to the Proxy Service, which establishes a reverse tunnel connection to the Auth Service. The plugin uses this reverse tunnel, along with your TLS credentials, to connect to the Auth Service's gRPC endpoint.

You will refer to this file later when configuring the plugin.

The following tctl auth sign command impersonates the access-plugin user, generates signed credentials, and writes an identity file to the local directory:

tctl auth sign --user=access-plugin --out=auth

Then create a Kubernetes secret:

kubectl create secret generic teleport-mattermost-identity --from-file=auth_id=auth.pem

Teleport's Access Request plugins listen for new and updated Access Requests by connecting to the Teleport Auth Service's gRPC endpoint over TLS.

The identity file, auth.pem, includes both TLS and SSH credentials. Your Access Request plugin uses the SSH credentials to connect to the Proxy Service, which establishes a reverse tunnel connection to the Auth Service. The plugin uses this reverse tunnel, along with your TLS credentials, to connect to the Auth Service's gRPC endpoint.

The Helm chart only supports the file format.

You will refer to this file later when configuring the plugin.

Certificate Lifetime

By default, tctl auth sign produces certificates with a relatively short lifetime. For production deployments, you can use the --ttl flag to ensure a more practical certificate lifetime, e.g., --ttl=8760h to export a one-year certificate.

Step 5/8. Register a Mattermost bot

Now that you have generated the credentials your plugin needs to connect to your Teleport cluster, register your plugin with Mattermost so it can send Access Request messages to your workspace.

In Mattermost, click the menu button in the upper left of the UI, then click System Console → Integrations → Bot Accounts.

Set "Enable Bot Account Creation" to "true".

Enable Mattermost bots

This will allow you to create a new bot account for the Mattermost plugin.

Go back to your team. In the menu on the upper left of the UI, click Integrations → Bot Accounts → Add Bot Account.

Set the "Username", "Display Name", and "Description" fields according to how you would like the Mattermost plugin bot to appear in your workspace. Set "Role" to "Member".

You can download our avatar to set as your Bot Icon.

Set "post:all" to "Enabled".

Enable Mattermost Bots

Click "Create Bot Account". We will use the resulting OAuth 2.0 token when we configure the Mattermost plugin.

Step 6/8. Configure the Mattermost plugin

At this point, you have generated credentials that the Mattermost plugin will use to connect to Teleport and Mattermost. You will now configure the Mattermost plugin to use these credentials and post messages in the right channels for your workspace.

Environment type

The Mattermost plugin uses a config file in TOML format. On the host where you will run the Mattermost plugin, generate a boilerplate config by running the following commands:

teleport-mattermost configure > teleport-mattermost.toml
sudo mv teleport-mattermost.toml /etc

The Mattermost Helm Chart uses a YAML values file to configure the plugin. On the host where you have Helm installed, create a file called teleport-mattermost-helm.yaml based on the following example:

teleport:
  address: "example.com:3025"                              # Teleport Auth Server GRPC API address
  identitySecretName: teleport-plugin-mattermost-identity  # Identity file path

mattermost:
  url: https://mattermost.example.com/  # URL of the Mattermost instance
  token: mattermosttoken                # Mattermost token of the bot
  recipients:
    - "[email protected]"     # User
    - "team/example-channel"            # Channel

log:
  output: "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/mattermost.log"
  severity: "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

The Mattermost Helm Chart uses a YAML values file to configure the plugin. On the host where you have Helm installed, create a file called teleport-mattermost-helm.yaml based on the following example:

teleport:
  address: "myinstance.teleport.sh:443"                    # Teleport Cloud proxy HTTPS address
  identitySecretName: teleport-plugin-mattermost-identity  # Secret containing identity

mattermost:
  url: https://mattermost.example.com/  # URL of the Mattermost instance
  token: mattermosttoken                # Mattermost token of the bot
  recipients:
    - "[email protected]"     # User
    - "team/example-channel"            # Channel

log:
  output: "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/mattermost.log"
  severity: "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

Edit the configuration as explained below:

[teleport]

Environment type

addr: Include the hostname and HTTPS port of your Teleport Proxy Service (e.g., teleport.example.com:443). If you are configuring your plugin to connect directly to the Teleport Auth Service, use your Auth Service's gRPC endpoint (e.g., teleport.example.com:3025).

identity, client_key, client_crt, root_cas: The values you will use for these fields depend on whether the Mattermost plugin will connect to the Proxy Service or the Auth Service.

If you exported an identity file earlier, fill in the identity field with the path to the file and comment out the other fields.

If you exported a client key, client certificate, and root CAs earlier, fill in the client_key, client_crt, and root_cas fields with the paths to these files and leave identity commented out.

addr: Include the hostname and HTTPS port of your Teleport Cloud tenant (e.g., teleport.example.com:443).

identity, client_key, client_crt, root_cas: Fill in the identity field with the path to the identity file you exported earlier and comment out the other fields.

address: Include the hostname and HTTPS port of your Teleport Cloud tenant (e.g., teleport.example.com:443).

identitySecretName: Fill in the identitySecretName field with the name of the Kubernetes secret you created earlier.

address: Include the hostname and HTTPS port of your Teleport Cloud tenant (e.g., teleport.example.com:443).

identitySecretName: Fill in the identitySecretName field with the name of the Kubernetes secret you created earlier.

[mattermost]

url: Include the scheme (https://) and fully qualified domain name of your Mattermost deployment.

token: Find your Mattermost bot's OAuth 2.0 token. To do so, visit Mattermost. In the menu on the upper left of the UI, go to Integrations → Bot Accounts. Find the listing for the Teleport plugin and click "Create New Token". After you save the token, you will see a message with text in the format, "Access Token: TOKEN". Copy the token and paste it here.

recipients: This field configures the channels that the Mattermost plugin will notify when it receives an Access Request message. The value is an array of strings, where each element is either:

  • The email address of a Mattermost user to notify via a direct message when the plugin receives an Access Request event
  • A channel name in the format team/channel, where / separates the name of the team and the name of the channel

For example, this configuration will notify [email protected] and the Town Square channel in the myteam team of any Access Request events:

recipients = [
  "myteam/Town Square",
  "[email protected]"
]

url: Include the scheme (https://) and fully qualified domain name of your Mattermost deployment.

token: Find your Mattermost bot's OAuth 2.0 token. To do so, visit Mattermost. In the menu on the upper left of the UI, go to Integrations → Bot Accounts. Find the listing for the Teleport plugin and click "Create New Token". After you save the token, you will see a message with text in the format, "Access Token: TOKEN". Copy the token and paste it here.

recipients: This field configures the channels that the Mattermost plugin will notify when it receives an Access Request message. The value is an array of strings, where each element is either:

  • The email address of a Mattermost user to notify via a direct message when the plugin receives an Access Request event
  • A channel name in the format team/channel, where / separates the name of the team and the name of the channel

For example, this configuration will notify [email protected] and the Town Square channel in the myteam team of any Access Request events:

recipients:
  - "myteam/Town Square"
  - [email protected]

You will need to invite your Teleport plugin to any channel you add to the recipients list (aside from direct message channels). Visit Mattermost, navigate to each channel you want to invite the plugin to, and enter /invite @teleport (or the name of the bot you configured) into the message box.

Invite the bot

Users can also suggest reviewers when they create an Access Request, e.g.,:

tsh request create --roles=dbadmin [email protected],[email protected]

If an Access Request includes suggested reviewers, the Mattermost plugin will add these to the list of channels to notify. If a suggested reviewer is an email address, the plugin will look up the the direct message channel for that address and post a message in that channel.

If recipients is empty, and the user requesting elevated privileges has not suggested any reviewers, the plugin will skip forwarding the Access Request to Mattermost.

The final configuration should look similar to this:

Environment type

# example mattermost configuration TOML file
[teleport]
auth_server = "myinstance.teleport.sh:443"                   # Teleport Cloud proxy HTTPS address
identity = "/var/lib/teleport/plugins/mattermost/auth.pem"   # Identity file path

[mattermost]
url = "https://mattermost.example.com" # Mattermost Server URL
token = "api-token"                    # Mattermost Bot OAuth token
recipients = [
  "myteam/general",
  "[email protected]"
]

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/mattermost.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

# example mattermost configuration TOML file
[teleport]
auth_server = "example.com:3025"                             # Teleport Auth Server GRPC API address
client_key = "/var/lib/teleport/plugins/mattermost/auth.key" # Teleport GRPC client secret key
client_crt = "/var/lib/teleport/plugins/mattermost/auth.crt" # Teleport GRPC client certificate
root_cas = "/var/lib/teleport/plugins/mattermost/auth.cas"   # Teleport cluster CA certs

[mattermost]
url = "https://mattermost.example.com" # Mattermost Server URL
token = "api-token"                    # Mattermost Bot OAuth token
recipients = [
  "myteam/general",
  "[email protected]"
]

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/mattermost.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".
teleport:
  address: "myinstance.teleport.sh:443"                    # Teleport Cloud proxy HTTPS address
  identitySecretName: teleport-plugin-mattermost-identity  # Secret containing identity

mattermost:
  url: https://mattermost.example.com/  # URL of the Mattermost instance
  token: mattermosttoken                # Mattermost token of the bot
  recipients:
    - "[email protected]"     # User
    - "team/example-channel"            # Channel

log:
  output: "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/mattermost.log"
  severity: "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

teleport:
  address: "example.com:3025"                              # Teleport Auth Server GRPC API address
  identitySecretName: teleport-plugin-mattermost-identity  # Identity file path

mattermost:
  url: https://mattermost.example.com/  # URL of the Mattermost instance
  token: mattermosttoken                # Mattermost token of the bot
  recipients:
    - "[email protected]"     # User
    - "team/example-channel"            # Channel

log:
  output: "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/mattermost.log"
  severity: "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

Step 7/8. Test your Mattermost bot

After modifying your configuration, run the bot with the following command:

teleport-mattermost start -d

The -d flag provides debug information to make sure the bot can connect to Mattermost, e.g.:

DEBU   Checking Teleport server version mattermost/main.go:234
DEBU   Starting a request watcher... mattermost/main.go:296
DEBU   Starting Mattermost API health check... mattermost/main.go:186
DEBU   Starting secure HTTPS server on :8081 utils/http.go:146
DEBU   Watcher connected mattermost/main.go:260
DEBU   Mattermost API health check finished ok mattermost/main.go:19

After modifying your configuration, run the bot with the following command:

helm upgrade --install teleport-plugin-mattermost teleport/teleport-plugin-mattermost --values teleport-mattermost-helm.yaml

To inspect the plugin's logs, use the following command:

kubectl logs deploy/teleport-plugin-mattermost

Debug logs can be enabled by setting log.severity to DEBUG in teleport-mattermost-helm.yaml and executing the helm upgrade ... command above again. Then you can restart the plugin with the following command:

kubectl rollout restart deployment teleport-plugin-mattermost

Create an Access Request

A Teleport admin can create an Access Request for another user with tctl:

tctl request create myuser --roles=editor

Users can use tsh to create an Access Request and log in with approved roles:

tsh request new --roles=editor

Seeking request approval... (id: 8f77d2d1-2bbf-4031-a300-58926237a807)

Users can request access using the Web UI by visiting the "Access Requests" tab and clicking "New Request":

Creating an Access Request using the Web UI

The users and channels you configured earlier to review the request should receive a message from "Teleport" in Mattermost allowing them to visit a link in the Teleport Web UI and either approve or deny the request.

Resolve the request

Once you receive an Access Request message, click the link to visit Teleport and approve or deny the request:

Reviewing a request

You can also review an Access Request from the command line:

Replace REQUEST_ID with the id of the request

tctl request approve REQUEST_ID
tctl request deny REQUEST_ID

Replace REQUEST_ID with the id of the request

tsh request review --approve REQUEST_ID
tsh request review --deny REQUEST_ID
Auditing Access Requests

When the Mattermost plugin posts an Access Request notification to a channel, anyone with access to the channel can view the notification and follow the link. While users must be authorized via their Teleport roles to review Access Requests, you should still check the Teleport audit log to ensure that the right users are reviewing the right requests.

When auditing Access Request reviews, check for events with the type Access Request Reviewed in the Teleport Web UI and access_request.review if reviewing the audit log on the Auth Service host.

Step 8/8. Set up systemd

In production, we recommend starting the Teleport plugin daemon via an init system like systemd. Here's the recommended Teleport plugin service unit file for systemd:

[Unit]
Description=Teleport Mattermost Plugin
After=network.target

[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport-mattermost start --config=/etc/teleport-mattermost.toml
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-mattermost.pid

[Install]
WantedBy=multi-user.target

Save this as teleport-mattermost.service in either /usr/lib/systemd/system/ or another unit file load path supported by systemd.

Enable and start the plugin:

sudo systemctl enable teleport-mattermost
sudo systemctl start teleport-mattermost

Feedback

If you have any issues with this plugin, please create an issue on GitHub.