This blog is part three in a series about identity-based access and management of AWS resources. In Part I, we covered how to use OSS Teleport to access Amazon EC2 instances running in private subnets. Part II explained implementing identity-based access via SSO integration with Okta. In Part III, we will guide you through the steps to configure privilege escalation for just-in-time access requests.
Just-in-time privilege escalation with Teleport RBAC and Slack
The just-in-time privilege elevation mechanism limits the time for which privileged access is enabled on a critical system. Admins, users, applications and scripts can only access sensitive information when necessary and only for the length of time necessary to accomplish their tasks.
ChatOps is a term that refers to conversation-driven operations based on instant messaging and notification platforms such as Slack, Jira, and PagerDuty. When DevOps workflows need a human in the loop, ChatOps becomes a valuable approach. It notifies users and teams about potential outages and incidents and even prompts them to approve or deny requests.
By combining just-in-time privilege elevation with ChatOps, administrators are notified allowing them to interactively approve timebound privilege escalation requests.
With Teleport Enterprise, customers can leverage RBAC, access requests and Slack integration to implement highly efficient workflows for temporary privilege escalation.
Let's consider a scenario where an AWS enterprise customer has outsourced DevOps to a third-party managed service provider. The external service providers are managed by a full-time employee (team leader) who is authorized to approve or deny privileged access requests. To avoid unwanted exposure resulting in credential exploitation and misconfiguration, the organization wants to implement just-in-time privilege elevation that should expire after a specified period of time.
For example, a contractor from the service provider’s team will have to request root access to perform DBA operations on a database server once a week. Since these operations can be completed within an hour, the team leader approves the escalation request that’s valid only for an hour. The privilege elevation request is sent to the team leader via a Slack channel through which he can approve or deny the request. After the time window expires, the contractor no longer has root access to the DB server.
We will go through the steps necessary to implement this scenario. The prerequisites are:
- AWS Account
- AWS CLI
- DNS Service
- Slack Account
- Teleport Enterprise (signup for a free trial here)
Since this tutorial builds upon the previous scenario of setting up Teleport as an AWS bastion host, please take a look at the steps involved in provisioning the AWS infrastructure including the VPC, security groups, Internet gateway, NAT gateway and routing tables. You also need to provision Amazon EC2 instances and install Teleport Enterprise software. For a detailed walkthrough, refer to the second tutorial of this series.
Before getting started, ensure that you have the below topology:
This tutorial also assumes that you have created a local user account, tele-admin, in Teleport with editor and access roles. Refer to the previous blog for the details on adding the local account to Teleport.
Step 1 - Creating roles and users in Teleport Enterprise
Let's start by creating Linux users, Teleport users and Teleport roles needed for this scenario.
SSH into the EC2 instance representing the database server and create two Linux users - a regular user and a user with sudo privileges.
$ tsh ssh [email protected] $ sudo adduser john $ sudo adduser dba $ sudo usermod -aG sudo dba
The account, john, will be used by the contractor to access the database server. Occasionally, the same user may have to SSH as dba - a user with elevated privileges - to perform maintenance activities on the server.
Assuming that you have signed into Teleport proxy as user tele-admin, create three roles in Teleport — contractor, dba and team-lead.
$ vi contractor.yaml kind: role version: v4 metadata: name: contractor spec: allow: logins: ['john'] node_labels: name: dbserver request: roles: ['dba'] deny: node_labels: name: appserver $ tctl create -f contractor.yaml
The above definition creates a role that explicitly allows access to the dbserver as user john, which we created in the EC2 instance (dbserver). It also lets the user request for privilege escalation by elevating the role to a dba.
Now, let's create the dba role.
$ vi dba.yaml kind: role version: v4 metadata: name: dba spec: allow: logins: ['dba'] node_labels: name: dbserver deny: node_labels: name: appserver options: max_session_ttl: 1h $ tctl create -f dba.yaml
This role is similar to the contractor role except that it allows the user to SSH as a dba user. The session is configured to expire in one hour through the setting,
Finally, let's create the team-lead role that's critical to implementing just-in-time privilege escalation.
$ vi team-lead.yaml kind: role version: v4 metadata: name: team-lead spec: allow: impersonate: users: ['*'] roles: ['*'] review_requests: roles: - 'dba' rules: - resources: - access_request verbs: - list - read - update - delete $ tctl create -f team-lead.yaml
The above role will be assigned to a full-time employee who has permission to approve privilege escalation requests. They can list the requests and then decide to approve or reject them.
With the roles in place, let's go ahead and create users in Teleport Enterprise that are assigned to respective roles.
First, let's create a user, dave, who assumes the role of the team lead and Teleport admin.
$ tctl users add --roles=access,auditor,editor,team-lead dave --logins=ubuntu
Next, let's create another user — john — who is a contractor but can occasionally become a DBA.
$ tctl users add --roles=access,contractor john --logins=john
Test if john can SSH into the dbserver with his username and also as a dba.
$ tsh login --proxy=tele.j-access.in --auth=local --user=john
$ tsh ssh [email protected]
While john is able to SSH into the dbserver, he doesn’t have sudo privileges.
$ tsh ssh [email protected]
When john tries to SSH as dba, the Teleport auth server restricts his access due to the enforced RBAC.
Let’s get a quick summary of Linux users, Teleport users and the associated roles.
|Teleport User||Persona||Teleport Roles||Linux User|
Note that John will become a DBA for an hour only when Dave approves his request.
Step 2 - Implementing privilege escalation workflow
Since John is allowed to assume the role of a DBA temporarily, he has to raise a request.
Run the below command to initiate the request:
$ tsh login --proxy=tele.j-access.in --auth=local --user=john --request-roles=dba
Open another terminal window, and login into Teleport as user dave, the team leader.
$ tsh login --proxy=tele.j-access.in --auth=local --user=dave
Check the pending requests for privilege escalation with the below command:
$ tctl request ls
Dave can now approve the request raised by John to become a DBA for one hour.
tctl request approve b5637bea-f5d4-48e0-a8a0-4e00f70eca8b
As soon as the request is approved, John sees the below output on his terminal:
Notice that the dba role is added to the approved roles that’s valid for 57m0s.
Now, let’s check if John can really become a DBA.
Indeed, John is the DBA with sudo privileges. He can now perform the server maintenance tasks. After the time window expires, he has to request Dave again for a new session with escalated privilege.
When John raises a request, Dave needs to be notified that there is a pending request waiting for his approval. For sending the alert, let’s configure Slack which sends an alert to Dave when his attention is needed.
Step 3 - Configuring Slack to approve or reject access requests
First, we should create a user and a role required by the Slack plugin.
$ vi access-plugin.yaml kind: user metadata: name: access-plugin spec: roles: ['access-plugin'] version: v2 --- kind: role version: v4 metadata: name: access-plugin spec: allow: rules: - resources: ['access_request'] verbs: ['list', 'read'] - resources: ['access_plugin_data'] verbs: ['update'] $ tctl create -f access-plugin.yaml
Teleport's plugins use the access-plugin role and user to approve access requests. We export the identity files to this plugin.
Sign into Teleport as Dave to impersonate the access-plugin user to export the identity files.
$ tsh login --proxy=tele.j-access.in --auth=local --user=dave $ tctl auth sign --format=tls --user=access-plugin --out=auth --ttl=2190h
The above command dumps three files — auth.key, auth.cas, and auth.crt. We will need these files to complete one of the later steps of the tutorial.
Now is the time to configure the Slack application. Visit https://api.slack.com/apps to create a new Slack app.
Give the app a name and choose the workplace where you want to get the notification.
Under the OAuth and Permissions section of the left navigation bar, choose scopes and add chat:write, incoming-webhook, users:read, users:read.email oAuth scopes. This will let the plugin post messages to the Slack channel.
Access the Bot User oAuth token and copy it for later use.
Install the app in your workspace by allowing the bot to access the channel.
Finally, invite the Teleport bot into the teams channel in your Slack app.
With the Slack bot in place, let’s connect that to Teleport.
SSH into Teleport proxy and run the below commands that download and install the Slack plugin.
$ curl -L https://get.gravitational.com/teleport-access-slack-v9.0.0-linux-amd64-bin.tar.gz $ tar -xzf teleport-access-slack-v9.0.0-linux-amd64-bin.tar.gz $ cd teleport-access-slack $ ./install Copy the auth files — `auth.key`, `auth.cas` and `auth.crt` — generated earlier to the `/var/lib/teleport/plugins/slack/` directory. Create the file by name teleport-slack.toml under the /etc/ directory with the below content: ```yaml [teleport] addr = "tele.j-access.in:3025" # Teleport Auth Server GRPC API address client_key = "/var/lib/teleport/plugins/slack/auth.key" # Teleport GRPC client secret key client_crt = "/var/lib/teleport/plugins/slack/auth.crt" # Teleport GRPC client certificate root_cas = "/var/lib/teleport/plugins/slack/auth.cas" # Teleport cluster CA certs [slack] token = "SLACK_oAUTH_TOKEN" # Slack Bot OAuth token [role_to_recipients] "*" = "team" [log] output = "stderr" severity = "INFO"
Launch another terminal and SSH into Teleport proxy to start the Slack plugin.
$ teleport-slack start
In your workstation, login to Teleport proxy as user john requesting for privileged access
$ tsh login --proxy=tele.j-access.in --auth=local --user=john --request-roles=dba
Immediately, you will get a Slack notification with the request.
Click the link and login to the web UI as Dave. You should see a request waiting for his approval.
As soon as it is approved, John’s terminal shows that the login request is approved.
This scenario demonstrated integrating Teleport RBAC with Slack to perform a just-in-time privilege escalation based on ChatOps. It highlights the need to configure time-bound SSH sessions to avoid credential exploitation and possible misconfiguration of servers. The integration with Slack brings an intuitive alerting and approval process to the workflow. Teleport's RBAC combined with Slack delivers a simple yet powerful mechanism of just-in-time privilege escalation. Learn more on how Teleport supports advanced authorization use cases using dual authorization, and moderated sessions.
Try Teleport cloud for free today — https://goteleport.com/signup/
Passkeys for Infrastructure
By Ben Arent
SFTP: a More Secure Successor to SCP
By Andrew LeFevre
SELinux, Dragons and Other Scary Things
By Jakub Nyckowski