SSH Login Approval using PagerDuty

Teleport PagerDuty Plugin Setup

This guide will talk through how to setup Teleport with PagerDuty. Teleport to PagerDuty integration allows you to treat Teleport access and permission requests as PagerDuty incidents — notifying the appropriate team, and approve or deny the requests via PagerDuty special action.

Setup

Prerequisites

This guide assumes that you have:

• A running Teleport Cluster
• Admin privileges with access to tctl
• A PagerDuty account, with access to creating a new API token
• A node to run the plugin, we recommend running it alongside the Teleport Proxy for convenience

Create User and Role for access

Log into Teleport Authentication Server, this is where you normally run tctl. Create a new user and role that only has API access to the access_request API. The below script will create a yaml resource file for a new user and role.

# This command will create two Teleport Yaml resources, a new Teleport user and a
# Role for that users that can only approve / list requests.
$ cat > rscs.yaml <<EOF
kind: user
metadata:
  name: access-plugin-pagerduty
spec:
  roles: ['access-plugin-pagerduty']
version: v2
---
kind: role
metadata:
  name: access-plugin-pagerduty
spec:
  allow:
    rules:
      - resources: ['access_request']
        verbs: ['list','read','update']
    # teleport currently refuses to issue certs for a user with 0 logins,
    # this restriction may be lifted in future versions.
    logins: ['access-plugin-pagerduty']
version: v4
EOF

# Run this to create the user and role in Teleport.
$ tctl create -f rscs.yaml

Export access-plugin Certificate

Teleport Plugin use the access-plugin-pagerduty role and user to perform the approval. We export the identity files, using tctl auth sign.

tctl auth sign --format=tls --user=access-plugin-pagerduty --out=auth --ttl=2190h
...

The above sequence should result in three PEM encoded files being generated: auth.crt, auth.key, and auth.cas (certificate, private key, and CA certs respectively). We'll reference the auth.crt, auth.key, and auth.cas files later when configuring the plugins.

Setting up PagerDuty API key

In your PagerDuty dashboard, go to Configuration → API Access → Create New API Key, add a key description, and save the key. We'll use the key in the plugin config file later.

Create PagerDuty API Key

Create Service Account

Downloading and installing the plugin

We recommend installing the Teleport Plugins alongside the Teleport Proxy. This is an ideal location as plugins have a low memory footprint, and will require both public internet access and Teleport Auth access. We currently only provide linux-amd64 binaries, you can also compile these plugins from source.

$ wget https://get.gravitational.com/teleport-access-pagerduty-v8.0.7-linux-amd64-bin.tar.gz
$ tar -xzf teleport-access-pagerduty-v8.0.7-linux-amd64-bin.tar.gz
$ cd teleport-access-pagerduty/
$ ./install
$ which teleport-pagerduty
/usr/local/bin/teleport-pagerduty

Run ./install in from 'teleport-pagerduty' or place the executable in the appropriate /usr/bin or /usr/local/bin on the server installation.

Config file

Teleport PagerDuty plugin has its own configuration file in TOML format. Before starting the plugin for the first time, you'll need to generate and edit that config file.

teleport-pagerduty configure > teleport-pagerduty.toml
sudo mv teleport-pagerduty.toml /etc

Editing the config file

After generating the config, edit it as follows:

# Example PagerDuty config file
# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "example.com:3025"                            # Teleport Auth Server GRPC API address
client_key = "/var/lib/teleport/plugins/pagerduty/auth.key" # Teleport GRPC client secret key
client_crt = "/var/lib/teleport/plugins/pagerduty/auth.crt" # Teleport GRPC client certificate
root_cas = "/var/lib/teleport/plugins/pagerduty/auth.cas"   # Teleport cluster CA certs

[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id

[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key"  # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate

[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls

[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

Testing the Plugin

With the config above, you should be able to run the plugin invoking teleport-pagerduty start -d. The will provide some debug information to make sure the bot can connect to PagerDuty.

teleport-pagerduty start -d
DEBU   DEBUG logging enabled logrus/exported.go:117
INFO   Starting Teleport Access PagerDuty extension 0.1.0-dev.1: pagerduty/main.go:124
DEBU   Checking Teleport server version pagerduty/main.go:226
DEBU   Starting a request watcher... pagerduty/main.go:288
DEBU   Starting PagerDuty API health check... pagerduty/main.go:170
DEBU   Starting secure HTTPS server on :8081 utils/http.go:146
DEBU   Watcher connected pagerduty/main.go:252
DEBU   PagerDuty API health check finished ok pagerduty/main.go:176
DEBU   Setting up the webhook extensions pagerduty/main.go:178

By default, teleport-pagerduty will assume its config is in /etc/teleport-pagerduty.toml, but you can override it with --config option.

Setup with SystemD

In production, we recommend starting teleport plugin daemon via an init system like systemd . Here's the recommended Teleport Plugin service unit file for systemd:

[Unit]
Description=Teleport Pagerduty Plugin
After=network.target

[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport-pagerduty start --config=/etc/teleport-pagerduty.toml
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-pagerduty.pid

[Install]
WantedBy=multi-user.target

Save this as teleport-pagerduty.service.

Example PagerDuty Request

On-call Auto Approval

The PagerDuty plugin has an option to auto-approve access requests. This feature will map external SSO identity to PagerDuty on call email addresses. If the user requesting matches the person on call the request will be automatically approved.

This feature can be enabled via the PagerDuty config file using auto_approve = true.

# Example teleport-pagerduty.toml snippet
#...
[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id
auto_approve = true           # Auto approve

Audit Log

The plugin will let anyone with access to the PagerDuty account so it's important to review Teleport's audit log.

Feedback

If you have any issues with this plugin please create an issue here.