Demo | Access Workflow Integration Using Pager Duty | Privileged Access Management | Teleport
Length: 03:09
This guide will talk through how to setup Teleport with PagerDuty. Teleport's PagerDuty integration allows you to treat Teleport access and permission requests as PagerDuty incidents, notify the appropriate team, and approve or deny the requests via PagerDuty special action.
Setup
Prerequisites
This guide assumes that you have:
- A running Teleport Cluster
- Admin privileges with access to
tctl
- A PagerDuty account, with access to creating a new API token
- A node to run the plugin. We recommend running it alongside the Teleport Proxy for convenience
Teleport Cloud requires that plugins connect through the Proxy Service (mytenant.teleport.sh:443
). Open Source and Enterprise installations can connect to the Auth Service (auth.example.com:3025
) directly.
To connect to Teleport, log in to your cluster using tsh
, then use tctl
remotely:
tsh login --proxy=teleport.example.com [email protected]tctl statusCluster teleport.example.com
Version 9.3.7
CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
You can run subsequent tctl
commands in this guide on your local machine.
For full privileges, you can also run tctl
commands on your Auth Service host.
To connect to Teleport, log in to your cluster using tsh
, then use tctl
remotely:
tsh login --proxy=myinstance.teleport.sh [email protected]tctl statusCluster myinstance.teleport.sh
Version 9.3.8
CA pin sha256:sha-hash-here
You must run subsequent tctl
commands in this guide on your local machine.
Create a user and role for access
Using an existing Teleport cluster, create the following user
and role
resources with the command below, replacing YAML_PATH
with the path to each resource spec.
$ tctl create -f YAML_PATH.yaml
Create a non-interactive bot user and role called access-plugin
.
kind: user
metadata:
name: access-plugin
spec:
roles: ['access-plugin']
version: v2
---
kind: role
version: v4
metadata:
name: access-plugin
spec:
allow:
rules:
- resources: ['access_request']
verbs: ['list', 'read']
- resources: ['access_plugin_data']
verbs: ['update']
If you're using other plugins, you might want to create different users and roles for different plugins
Export the access-plugin certificate
Teleport's plugins use the access-plugin
role and user to approve access requests. We export the identity files to this plugin using tctl auth sign
.
tctl auth sign --format=tls --user=access-plugin --out=auth --ttl=2190h...
The above sequence should result in three PEM encoded files being generated: auth.crt
, auth.key
, and auth.cas
(certificate, private key, and CA certs respectively).
tctl auth sign --user=access-plugin --out=auth.pem --ttl=2190h...
The above sequence should result in one PEM encoded file: auth.pem
.
By default, tctl auth sign
produces certificates with a relatively short lifetime. For production deployments, the --ttl
flag can be used to ensure a more practical certificate lifetime. --ttl=8760h
exports a 1 year token
We'll reference these files later when configuring the plugins.
Setting up a PagerDuty API key
In your PagerDuty dashboard, go to Configuration → API Access → Create New API Key, add a key description, and save the key. We'll use the key in the plugin config file later.
Create a PagerDuty API key
Create a service account
Downloading and installing the plugin
We recommend installing Teleport plugins alongside the Teleport Proxy. This is an ideal location as plugins have a low memory footprint, and will require both public internet access and Teleport Auth Service access.
curl -L -O https://get.gravitational.com/teleport-access-pagerduty-v9.3.7-linux-amd64-bin.tar.gztar -xzf teleport-access-pagerduty-v9.3.7-linux-amd64-bin.tar.gzcd teleport-access-pagerduty./install
To install from source you need git
and go
installed. If you do not have Go installed, visit the Go downloads page.
Checkout teleport-plugins
git clone https://github.com/gravitational/teleport-plugins.gitcd teleport-plugins/access/pagerdutymake
Run ./install
from teleport-pagerduty
or place the executable in the appropriate /usr/bin
or /usr/local/bin
on the server installation.
docker pull quay.io/gravitational/teleport-plugin-pagerduty:9.3.7
Config file
Teleport's PagerDuty plugin has its own configuration file in TOML format. Before starting the plugin for the first time, you'll need to generate and edit that config file.
teleport-pagerduty configure > teleport-pagerduty.tomlsudo mv teleport-pagerduty.toml /etc
Editing the config file
After generating the config, edit it as follows:
# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "example.com:3025" # Teleport Auth Server GRPC API address
client_key = "/var/lib/teleport/plugins/pagerduty/auth.key" # Teleport GRPC client secret key
client_crt = "/var/lib/teleport/plugins/pagerduty/auth.crt" # Teleport GRPC client certificate
root_cas = "/var/lib/teleport/plugins/pagerduty/auth.cas" # Teleport cluster CA certs
[pagerduty]
api_key = "key" # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7" # PagerDuty service id
[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key" # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate
[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls
[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password
[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".
# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "myinstance.teleport.sh:443" # Teleport Cloud proxy HTTPS address
identity = "/var/lib/teleport/plugins/pagerduty/auth.pem" # Identity file path
[pagerduty]
api_key = "key" # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7" # PagerDuty service id
[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key" # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate
[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls
[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password
[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".
Testing the plugin
With the config above, you should be able to run the plugin invoking
teleport-pagerduty start -d
. The will provide some debug information to make sure
the bot can connect to PagerDuty.
teleport-pagerduty start -dDEBU DEBUG logging enabled logrus/exported.go:117
INFO Starting Teleport Access PagerDuty extension 0.1.0-dev.1: pagerduty/main.go:124
DEBU Checking Teleport server version pagerduty/main.go:226
DEBU Starting a request watcher... pagerduty/main.go:288
DEBU Starting PagerDuty API health check... pagerduty/main.go:170
DEBU Starting secure HTTPS server on :8081 utils/http.go:146
DEBU Watcher connected pagerduty/main.go:252
DEBU PagerDuty API health check finished ok pagerduty/main.go:176
DEBU Setting up the webhook extensions pagerduty/main.go:178
By default, teleport-pagerduty
will assume its config is in /etc/teleport-pagerduty.toml
, but you can override it with --config
option.
Set up systemd
In production, we recommend starting the Teleport plugin daemon via an init system like systemd. Here's the recommended Teleport plugin service unit file for systemd:
[Unit]
Description=Teleport Pagerduty Plugin
After=network.target
[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/teleport-pagerduty start --config=/etc/teleport-pagerduty.toml
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/teleport-pagerduty.pid
[Install]
WantedBy=multi-user.target
Save this as teleport-pagerduty.service
.
Example PagerDuty request
On-call auto-approval
The PagerDuty plugin has an option to auto-approve access requests. This feature will map an external SSO identity to a PagerDuty on-call email address. If the user requesting matches the person on call the request will be automatically approved.
This feature can be enabled via the PagerDuty config file using auto_approve = true
.
# Example teleport-pagerduty.toml snippet
#...
[pagerduty]
api_key = "key" # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7" # PagerDuty service id
auto_approve = true # Auto approve
Audit log
The plugin will let anyone with access to the PagerDuty account so it's important to review Teleport's audit log.
Feedback
If you have any issues with this plugin please create an issue here.