Teleport

SSH Login Approval using PagerDuty

Demo | Access Workflow Integration Using Pager Duty | Privileged Access Management | Teleport

Length: 03:09

This guide will talk through how to setup Teleport with PagerDuty. Teleport's PagerDuty integration allows you to treat Teleport access and permission requests as PagerDuty incidents, notify the appropriate team, and approve or deny the requests via PagerDuty special action.

Setup

Prerequisites

This guide assumes that you have:

A running Teleport Cluster
Admin privileges with access to tctl
A PagerDuty account, with access to creating a new API token
A node to run the plugin. We recommend running it alongside the Teleport Proxy for convenience

Teleport Cloud requires that plugins connect through the Proxy Service (mytenant.teleport.sh:443). Open Source and Enterprise installations can connect to the Auth Service (auth.example.com:3025) directly.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status
Cluster  teleport.example.com
Version  9.3.7
CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status
Cluster  myinstance.teleport.sh
Version  9.3.8
CA pin   sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Create a user and role for access

Using an existing Teleport cluster, create the following user and role resources with the command below, replacing YAML_PATH with the path to each resource spec.

$ tctl create -f YAML_PATH.yaml

Create a non-interactive bot user and role called access-plugin.

kind: user
metadata:
  name: access-plugin
spec:
  roles: ['access-plugin']
version: v2
---
kind: role
version: v4
metadata:
  name: access-plugin
spec:
  allow:
    rules:
      - resources: ['access_request']
        verbs: ['list', 'read']
      - resources: ['access_plugin_data']
        verbs: ['update']

Tip

If you're using other plugins, you might want to create different users and roles for different plugins

Export the access-plugin certificate

Teleport's plugins use the access-plugin role and user to approve access requests. We export the identity files to this plugin using tctl auth sign.

tctl auth sign --format=tls --user=access-plugin --out=auth --ttl=2190h
...

The above sequence should result in three PEM encoded files being generated: auth.crt, auth.key, and auth.cas (certificate, private key, and CA certs respectively).

tctl auth sign --user=access-plugin --out=auth.pem --ttl=2190h
...

The above sequence should result in one PEM encoded file: auth.pem.

Certificate Lifetime

By default, tctl auth sign produces certificates with a relatively short lifetime. For production deployments, the --ttl flag can be used to ensure a more practical certificate lifetime. --ttl=8760h exports a 1 year token

We'll reference these files later when configuring the plugins.

Setting up a PagerDuty API key

In your PagerDuty dashboard, go to Configuration → API Access → Create New API Key, add a key description, and save the key. We'll use the key in the plugin config file later.

Create a PagerDuty API key

Create a service account

Downloading and installing the plugin

We recommend installing Teleport plugins alongside the Teleport Proxy. This is an ideal location as plugins have a low memory footprint, and will require both public internet access and Teleport Auth Service access.

curl -L -O https://get.gravitational.com/teleport-access-pagerduty-v9.3.7-linux-amd64-bin.tar.gz
tar -xzf teleport-access-pagerduty-v9.3.7-linux-amd64-bin.tar.gz
cd teleport-access-pagerduty
./install

To install from source you need git and go installed. If you do not have Go installed, visit the Go downloads page.

Checkout teleport-plugins
git clone https://github.com/gravitational/teleport-plugins.git
cd teleport-plugins/access/pagerduty
make

Run ./install from teleport-pagerduty or place the executable in the appropriate /usr/bin or /usr/local/bin on the server installation.

docker pull quay.io/gravitational/teleport-plugin-pagerduty:9.3.7

Config file

Teleport's PagerDuty plugin has its own configuration file in TOML format. Before starting the plugin for the first time, you'll need to generate and edit that config file.

teleport-pagerduty configure > teleport-pagerduty.toml
sudo mv teleport-pagerduty.toml /etc

Editing the config file

After generating the config, edit it as follows:

# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "example.com:3025"                            # Teleport Auth Server GRPC API address
client_key = "/var/lib/teleport/plugins/pagerduty/auth.key" # Teleport GRPC client secret key
client_crt = "/var/lib/teleport/plugins/pagerduty/auth.crt" # Teleport GRPC client certificate
root_cas = "/var/lib/teleport/plugins/pagerduty/auth.cas"   # Teleport cluster CA certs

[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id

[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key"  # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate

[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls

[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "myinstance.teleport.sh:443"                  # Teleport Cloud proxy HTTPS address
identity = "/var/lib/teleport/plugins/pagerduty/auth.pem"   # Identity file path

[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id

[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key"  # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate

[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls

[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

Testing the plugin

With the config above, you should be able to run the plugin invoking teleport-pagerduty start -d. The will provide some debug information to make sure the bot can connect to PagerDuty.

teleport-pagerduty start -d
DEBU   DEBUG logging enabled logrus/exported.go:117
INFO   Starting Teleport Access PagerDuty extension 0.1.0-dev.1: pagerduty/main.go:124
DEBU   Checking Teleport server version pagerduty/main.go:226
DEBU   Starting a request watcher... pagerduty/main.go:288
DEBU   Starting PagerDuty API health check... pagerduty/main.go:170
DEBU   Starting secure HTTPS server on :8081 utils/http.go:146
DEBU   Watcher connected pagerduty/main.go:252
DEBU   PagerDuty API health check finished ok pagerduty/main.go:176
DEBU   Setting up the webhook extensions pagerduty/main.go:178

By default, teleport-pagerduty will assume its config is in /etc/teleport-pagerduty.toml, but you can override it with --config option.

Set up systemd

In production, we recommend starting the Teleport plugin daemon via an init system like systemd. Here's the recommended Teleport plugin service unit file for systemd:

[Unit]
Description=Teleport Pagerduty Plugin
After=network.target
[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport-pagerduty start --config=/etc/teleport-pagerduty.toml
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-pagerduty.pid
[Install]
WantedBy=multi-user.target

Save this as teleport-pagerduty.service.

Example PagerDuty request

On-call auto-approval

The PagerDuty plugin has an option to auto-approve access requests. This feature will map an external SSO identity to a PagerDuty on-call email address. If the user requesting matches the person on call the request will be automatically approved.

This feature can be enabled via the PagerDuty config file using auto_approve = true.

# Example teleport-pagerduty.toml snippet
#...
[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id
auto_approve = true           # Auto approve

Audit log

The plugin will let anyone with access to the PagerDuty account so it's important to review Teleport's audit log.

Feedback

If you have any issues with this plugin please create an issue here.

Have a suggestion or can’t find something?

IMPROVE THE DOCS