Fork me on GitHub

Teleport

SSH Login Approval using PagerDuty

Improve
Demo | Access Workflow Integration Using Pager Duty | Privileged Access Management | Teleport

Demo | Access Workflow Integration Using Pager Duty | Privileged Access Management | Teleport

Length: 03:09

This guide will talk through how to setup Teleport with PagerDuty. Teleport's PagerDuty integration allows you to treat Teleport access and permission requests as PagerDuty incidents, notify the appropriate team, and approve or deny the requests via PagerDuty special action.

Setup

Prerequisites

This guide assumes that you have:

  • A running Teleport Cluster
  • Admin privileges with access to tctl
  • A PagerDuty account, with access to creating a new API token
  • A node to run the plugin. We recommend running it alongside the Teleport Proxy for convenience

Teleport Cloud requires that plugins connect through the Proxy Service (mytenant.teleport.sh:443). Open Source and Enterprise installations can connect to the Auth Service (auth.example.com:3025) directly.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=teleport.example.com [email protected]
tctl status

Cluster teleport.example.com

Version 10.1.2

CA pin sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

You can run subsequent tctl commands in this guide on your local machine.

For full privileges, you can also run tctl commands on your Auth Service host.

To connect to Teleport, log in to your cluster using tsh, then use tctl remotely:

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status

Cluster myinstance.teleport.sh

Version 9.3.10

CA pin sha256:sha-hash-here

You must run subsequent tctl commands in this guide on your local machine.

Create a user and role for access

Teleport's Access Request plugins authenticate to your Teleport cluster as a user with permissions to list, read, and update Access Requests. This way, plugins can retrieve Access Requests from the Teleport Auth Service, present them to reviewers, and modify them after a review.

Define a user and role called access-plugin by adding the following content to a file called access-plugin.yaml:

kind: role
version: v5
metadata:
  name: access-plugin
spec:
  allow:
    rules:
      - resources: ['access_request']
        verbs: ['list', 'read', 'update']
      - resources: ['access_plugin_data']
        verbs: ['update']
---
kind: user
metadata:
  name: access-plugin
spec:
  roles: ['access-plugin']
version: v2

Create the user and role:

tctl create -f access-plugin.yaml

As with all Teleport users, the Teleport Auth Service authenticates the access-plugin user by issuing short-lived TLS credentials. In this case, we will need to request the credentials manually by impersonating the access-plugin role and user.

If you are using tctl from the Auth Service host, you will already have impersonation privileges.

To grant your user impersonation privileges for access-plugin, define a role called access-plugin-impersonator by pasting the following YAML document into a file called access-plugin-impersonator.yaml:

kind: role
version: v5
metadata:
  name: access-plugin-impersonator
spec:
  allow:
    impersonate:
      roles:
      - access-plugin
      users:
      - access-plugin

Create the access-plugin-impersonator role:

tctl create -f access-plugin-impersonator.yaml

Retrieve your user definition:

TELEPORT_USER=$(tsh status --format=json | jq -r .active.username)
tctl get users/${TELEPORT_USER?} > myuser.yaml

Edit myuser.yaml to include the role you just created:

  roles:
   - access
   - auditor
   - editor
+  - access-plugin-impersonator

Apply your changes:

tctl create -f myuser.yaml

Log out of your Teleport cluster and log in again. You will now be able to generate signed certificates for the access-plugin role and user.

Export the access-plugin certificate

Like all Teleport users, access-plugin needs signed credentials in order to connect to your Teleport cluster. You will use the tctl auth sign command to request these credentials for your plugin.

The format of the credentials depends on whether you have set up your network to give the plugin direct access to the Teleport Auth Service, or if all Teleport clients and services connect to the Teleport Proxy Service instead.

The following tctl auth sign command impersonates the access-plugin user, generates signed credentials, and writes an identity file to the local directory:

tctl auth sign --user=access-plugin --out=auth.pem

Teleport's Access Request plugins listen for new and updated Access Requests by connecting to the Teleport Auth Service's gRPC endpoint over TLS.

The identity file, auth.pem, includes both TLS and SSH credentials. Your Access Request plugin uses the SSH credentials to connect to the Proxy Service, which establishes a reverse tunnel connection to the Auth Service. The plugin uses this reverse tunnel, along with your TLS credentials, to connect to the Auth Service's gRPC endpoint.

You will refer to this file later when configuring the plugin.

If your network allows your plugin to access the Auth Service directly, e.g., you are running the plugin on the Auth Service host, the plugin uses TLS credentials to connect to the Auth Service's gRPC endpoint and listen for new and updated Access Requests.

You can generate TLS credentials with the following command:

tctl auth sign --format=tls --user=access-plugin --out=auth

This command should result in three PEM-encoded files: auth.crt, auth.key, and auth.cas (certificate, private key, and CA certs respectively). Later, you will configure the plugin to use these credentials to connect to the Auth Service directly.

Certificate Lifetime

By default, tctl auth sign produces certificates with a relatively short lifetime. For production deployments, you can use the --ttl flag to ensure a more practical certificate lifetime, e.g., --ttl=8760h to export a one-year certificate.

We'll reference these files later when configuring the plugins.

Setting up a PagerDuty API key

In your PagerDuty dashboard, go to Configuration → API Access → Create New API Key, add a key description, and save the key. We'll use the key in the plugin config file later.

Create a PagerDuty API key Create a service account

Create a service account Create a service account

Downloading and installing the plugin

We recommend installing Teleport plugins alongside the Teleport Proxy. This is an ideal location as plugins have a low memory footprint, and will require both public internet access and Teleport Auth Service access.

curl -L -O https://get.gravitational.com/teleport-access-pagerduty-v10.1.2-linux-amd64-bin.tar.gz
tar -xzf teleport-access-pagerduty-v10.1.2-linux-amd64-bin.tar.gz
cd teleport-access-pagerduty
./install

To install from source you need git and go installed. If you do not have Go installed, visit the Go downloads page.

Checkout teleport-plugins

git clone https://github.com/gravitational/teleport-plugins.git
cd teleport-plugins/access/pagerduty
make

Run ./install from teleport-pagerduty or place the executable in the appropriate /usr/bin or /usr/local/bin on the server installation.

docker pull quay.io/gravitational/teleport-plugin-pagerduty:10.1.2

Config file

Teleport's PagerDuty plugin has its own configuration file in TOML format. Before starting the plugin for the first time, you'll need to generate and edit that config file.

teleport-pagerduty configure > teleport-pagerduty.toml
sudo mv teleport-pagerduty.toml /etc

Editing the config file

After generating the config, edit it as follows:

# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "example.com:3025"                            # Teleport Auth Server GRPC API address
client_key = "/var/lib/teleport/plugins/pagerduty/auth.key" # Teleport GRPC client secret key
client_crt = "/var/lib/teleport/plugins/pagerduty/auth.crt" # Teleport GRPC client certificate
root_cas = "/var/lib/teleport/plugins/pagerduty/auth.cas"   # Teleport cluster CA certs

[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id

[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key"  # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate

[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls

[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".
# example teleport-pagerduty configuration TOML file
[teleport]
auth_server = "myinstance.teleport.sh:443"                  # Teleport Cloud proxy HTTPS address
identity = "/var/lib/teleport/plugins/pagerduty/auth.pem"   # Identity file path

[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id

[http]
public_addr = "example.com" # URL on which callback server is accessible externally, e.g. [https://]teleport-pagerduty.example.com
# listen_addr = ":8081" # Network address in format [addr]:port on which callback server listens, e.g. 0.0.0.0:443
https_key_file = "/var/lib/teleport/plugins/pagerduty/server.key"  # TLS private key
https_cert_file = "/var/lib/teleport/plugins/pagerduty/server.crt" # TLS certificate

[http.tls]
verify_client_cert = true # The preferred way to authenticate webhooks on Pagerduty. See more: https://developer.pagerduty.com/docs/webhooks/webhooks-mutual-tls

[http.basic_auth]
user = "user"
password = "password" # If you prefer to use basic auth for Pagerduty Webhooks authentication, use this section to store user and password

[log]
output = "stderr" # Logger output. Could be "stdout", "stderr" or "/var/lib/teleport/pagerduty.log"
severity = "INFO" # Logger severity. Could be "INFO", "ERROR", "DEBUG" or "WARN".

Testing the plugin

With the config above, you should be able to run the plugin invoking teleport-pagerduty start -d. The will provide some debug information to make sure the bot can connect to PagerDuty.

teleport-pagerduty start -d

DEBU DEBUG logging enabled logrus/exported.go:117

INFO Starting Teleport Access PagerDuty extension 0.1.0-dev.1: pagerduty/main.go:124

DEBU Checking Teleport server version pagerduty/main.go:226

DEBU Starting a request watcher... pagerduty/main.go:288

DEBU Starting PagerDuty API health check... pagerduty/main.go:170

DEBU Starting secure HTTPS server on :8081 utils/http.go:146

DEBU Watcher connected pagerduty/main.go:252

DEBU PagerDuty API health check finished ok pagerduty/main.go:176

DEBU Setting up the webhook extensions pagerduty/main.go:178

By default, teleport-pagerduty will assume its config is in /etc/teleport-pagerduty.toml, but you can override it with --config option.

Set up systemd

In production, we recommend starting the Teleport plugin daemon via an init system like systemd. Here's the recommended Teleport plugin service unit file for systemd:

[Unit]

Description=Teleport Pagerduty Plugin

After=network.target

[Service]

Type=simple

Restart=on-failure

ExecStart=/usr/local/bin/teleport-pagerduty start --config=/etc/teleport-pagerduty.toml

ExecReload=/bin/kill -HUP $MAINPID

PIDFile=/run/teleport-pagerduty.pid

[Install]

WantedBy=multi-user.target

Save this as teleport-pagerduty.service.

Example PagerDuty request

On-call auto-approval

The PagerDuty plugin has an option to auto-approve Access Requests. This feature will map an external SSO identity to a PagerDuty on-call email address. If the user requesting matches the person on call the request will be automatically approved.

This feature can be enabled via the PagerDuty config file using auto_approve = true.

# Example teleport-pagerduty.toml snippet
#...
[pagerduty]
api_key = "key"               # PagerDuty API Key
user_email = "[email protected]" # PagerDuty bot user email (Could be admin email)
service_id = "PIJ90N7"        # PagerDuty service id
auto_approve = true           # Auto approve

Audit log

The plugin will let anyone with access to the PagerDuty account so it's important to review Teleport's audit log.

Feedback

If you have any issues with this plugin please create an issue here.