Amazon RDS Just-in-Time (JIT) Access With Teleport and Slack

This blog is part three in a series about secure access to Amazon RDS. In Part 1, we covered how to use OSS Teleport as an identity-aware access proxy to access Amazon RDS instances running in private subnets. Part 2 explained implementing single sign-on (SSO) for Amazon RDS access using Okta and Teleport. In Part 3, we will guide you through the steps to configure privilege escalation for just-in-time access requests for Amazon RDS access.

Part 3: Just-in-time (JIT) privilege escalation for Amazon RDS with Teleport

The concept of just-in-time privilege elevation provides access to critical systems for a limited period of time. Access to sensitive information will only be granted to administrators, users, applications and scripts when it is absolutely necessary and only for the length of time required.

The term ChatOps refers to the use of instant messaging platforms such as Slack, Jira and PagerDuty for alerting or notifying users of changes to the system. When DevOps workflows need human involvement from time to time, ChatOps becomes a valuable tool. It makes users and teams aware of potential outages and incidents and even prompts them to approve or deny requests.

By combining just-in-time privilege elevation with ChatOps, administrators are notified allowing them to interactively approve time-bound privilege escalation requests.

With Teleport Enterprise, customers can leverage RBAC, access requests, and Slack integration to implement highly efficient workflows for temporary privilege escalation to access databases that are a part of Amazon RDS.

Let's consider a scenario where an AWS enterprise customer has outsourced DevOps and DBA operations to a third-party managed service provider. The external service providers are managed by a full-time employee (team leader) who is authorized to approve or deny privileged access requests. To avoid unwanted exposure resulting in credential exploitation and misconfiguration, the organization wants to implement just-in-time privilege elevation that should expire after a specified period of time.

For example, a contractor from the service provider’s team will have to request access to a production database to perform maintenance operations once a week. Since these operations can be completed within an hour, the team leader approves the escalation request that’s valid only for an hour. The privilege elevation request is sent to the team leader via a Slack channel through which the team leader can approve or deny the request. After the time window expires, the contractor no longer has access to the database available within the Amazon RDS server.

This tutorial builds on the first part of this series where we configured the Teleport proxy, auth server, DB agent running in Amazon EC2 instances, and an Amazon RDS instance based on the PostgreSQL engine.

Before getting started, ensure that you have the below topology:

The EC2 instance shows the Teleport proxy and auth server running in the same host.

The above EC2 instance is running the Teleport DB agent configured for the autodiscovery of Amazon RDS instances.

Finally, we have an Amazon RDS instance called salesdb running in a private subnet of the VPC that can be accessed only through Teleport.

This tutorial also assumes that you have created a local user account, tele-admin, in Teleport with editor and access roles. Refer to Part 2 for the details on adding the local account to Teleport.

Step 1 - Preparing the Amazon RDS database

Start a client session by accessing the salesdb PostgreSQL database created through Amazon RDS.

Let’s create a couple of databases and a user with access to those. We will then create a table and a user-defined function to query the rows of the table.

> CREATE USER user2 WITH PASSWORD 'secret99';
> GRANT rds_iam TO user2;

There are three databases — inventory, forecast and suppliers — already created within the salesdb instance. Let’s grant access to user2 for those databases.

> GRANT CONNECT ON DATABASE inventory TO user2;
> GRANT CONNECT ON DATABASE forecast TO user2;
> GRANT CONNECT ON DATABASE suppliers TO user2;

Let’s switch to the suppliers database and create a table called vendors.

\c suppliers;

> CREATE TABLE vendors (
    id int generated by default as identity,
    name varchar(100) not null,
    city varchar(100) not null,
    country varchar(100) not null
);

> INSERT INTO vendors (NAME,CITY,COUNTRY) VALUES ('Acme Inc', 'Delhi', 'India');
> INSERT INTO vendors (NAME,CITY,COUNTRY) VALUES ('Widgets Inc', 'San Jose', 'USA');
> INSERT INTO vendors (NAME,CITY,COUNTRY) VALUES ('Nuts & Bolts Inc', 'London', 'UK');
> INSERT INTO vendors (NAME,CITY,COUNTRY) VALUES ('Spanners R Us', 'New York', 'USA');
> GRANT ALL ON TABLE vendors TO user2;

The table has been given full access to user user2.

We will now create a function that queries the vendors from a specific country and grant access to user2. We will also set user2 as the owner of the function.

CREATE OR REPLACE FUNCTION vendors_america()
RETURNS SETOF vendors AS $$
BEGIN
  RETURN QUERY SELECT * FROM vendors WHERE country='USA';
END
$$ LANGUAGE plpgsql;
GRANT ALL ON FUNCTION vendors_america() TO user2;
ALTER FUNCTION vendors_america() OWNER TO user2;

This step aims to ensure that user2 has enough privileges to access all the databases with special permissions to query the vendors table in the suppliers database.

In a typical scenario, john always accesses the database as user1, which is restricted only to the inventory database.

When John, the contractor DBA wants to access the supplier database, John requests time-bound access to it by assuming a new Teleport role, which is approved by Dave, the team lead.

Step 2 - Creating roles and users in Teleport Enterprise

Let’s go ahead and create the roles that represent the contractor, DBA, and team lead personas.

kind: role
version: v5
metadata:
  name: contractor
spec:
  allow:
    request:
      roles:
      - dba    
    db_labels:
      '*': '*'    
    db_users: ['user1']
    db_names: ['inventory']

$ tctl create -f contractor.yaml

The above role explicitly provides access to the inventory database available only when logged in as user1. This role is the most restrictive of all the three.

The next role, DBA, provides unrestricted access to all the databases but only for an hour. The max_session_ttl enforces the time-bound access for any user belonging to this role. John, the contractor, will occasionally assume the role of DBA to access production databases.

kind: role
version: v5
metadata:
  name: dba
spec:
  allow:
    db_labels:
      '*': '*'    
    db_users: ["*"]
    db_names: ["*"]
  options:
    max_session_ttl: 1h    

$ tctl create -f dba.yaml

Finally, let’s create the team lead role that has permission to approve privilege escalation requests. Dave, the full-time employee and the team leader of the project, will assume this role.

kind: role
version: v5
metadata:
  name: team-lead
spec:
  allow:
    impersonate: 
       users: ['*']
       roles: ['*']     
    review_requests:
      roles:
      - 'dba'
    rules:
    - resources:
      - access_request
      verbs:
      - list
      - read
      - update
      - delete

$ tctl create -f team-lead.yaml

Verify the roles with the below command:

$ tctl get roles --format=text

With the roles in place, let’s go ahead and create users that map to the personas defined for this scenario.

First, let’s create the user john, who is added to the contractor role.

$ tctl users add --roles=access,contractor john

Then, we will create user dave, who assumes the role of the team lead. Dave is also added to the preset roles of access, auditor and editor, which give him all the permissions available to the Teleport administrator.

$ tctl users add --roles=access,auditor,editor,team-lead dave

Let’s login as John and explore the assigned access boundary.

tsh login --proxy=tele.j-access.in --auth=local --user=john 

Notice that John assumes the roles of an accessor and contractor.

Let’s access the inventory database by logging into PostgreSQL as database user, user1.

Once logged into the database, let’s try to switch to the forecast database.

John has no access to any database except inventory, which is explicitly defined in the Teleport contractor role.

Let’s get a quick summary of PostgreSQL users, Teleport users and the associated roles.

Step 3 - Implementing privilege escalation workflow

Now is the time for John to request just-in-time privilege escalation to access the vendors database. For this, John initiates a new session by passing the –-request-roles switch to the tsh command-line tool.

$ tsh login --proxy=tele.j-access.in --user=john --request-roles=dba

The request for privilege escalation is now pending. When Dave, the team lead, approves it, John will temporarily assume the role of DBA that provides him unrestricted access to all the databases.

Let’s log in as Dave to approve the request.

$ tsh login --proxy=tele.j-access.in --auth=local --user=dave

Dave can now check the pending requests waiting for approval.

$ tctl request ls

Dave goes ahead and approves the request with the below command:

$ tctl request approve d49219a6-6be7-4dd7-afc5-3f48e90f495c

As soon as it is approved, John completes the login process. John has approximately an hour to complete the maintenance tasks.

Let’s see if John can access databases other than inventory.

As we can see from the above screenshot, John is able to switch between all the available databases. John is also able to query the tables, access functions and stored procedures.

After the time window defined in the policy expires, John has to send a request to Dave for a new session with escalated privilege.

When John raises a request, Dave needs to be notified that there is a pending request waiting for approval. For sending the alert, let’s configure Slack which alerts Dave when access is requested by the users.

Step 4 - Configuring Slack to approve or reject access requests

First, we should create a user and a role required by the Slack plugin.

kind: user
metadata:
  name: access-plugin
spec:
  roles: ['access-plugin']
version: v2
---
kind: role
version: v4
metadata:
  name: access-plugin
spec:
  allow:
    rules:
      - resources: ['access_request']
        verbs: ['list', 'read']
      - resources: ['access_plugin_data']
        verbs: ['update']

$ tctl create -f access-plugin.yaml

Teleport's plugins use the access-plugin role and user to approve access requests. We export the identity files to this plugin.

Sign into Teleport as Dave to impersonate the access-plugin user to export the identity files.

$ tsh login --proxy=tele.j-access.in --auth=local --user=dave
$ tctl auth sign --format=tls --user=access-plugin --out=auth --ttl=2190h

The above command dumps three files — auth.key, auth.cas and auth.crt. We will need these files to complete one of the later steps of the tutorial.

Now is the time to configure the Slack application. Visit https://api.slack.com/apps to create a new Slack app.

Give the app a name and choose the workplace where you want to get the notification.

Under the OAuth and Permissions section of the left navigation bar, choose scopes and add chat:write, incoming-webhook, users:read, users:read.email OAuth scopes. This will let the plugin post messages to the Slack channel.

Access the Bot User OAuth token and copy it for later use.

Install the app in your workspace by allowing the bot to access the channel.

Finally, invite the Teleport bot into the teams channels in your Slack app.

With the Slack bot in place, let’s connect that to Teleport.

SSH into Teleport proxy and run the below commands that download and install the Slack plugin.

$ curl -L https://get.gravitational.com/teleport-access-slack-v9.0.0-linux-amd64-bin.tar.gz
$ tar -xzf teleport-access-slack-v9.0.0-linux-amd64-bin.tar.gz
$ cd teleport-access-slack
$ ./install

Note: For latest Teleport version, always check the Teleport download page.

Copy the auth files — auth.key, auth.cas and auth.crt — generated earlier to the /var/lib/teleport/plugins/slack/ directory.

Create the file by name teleport-slack.toml under the /etc/ directory with the below content:

[teleport]
addr = "tele.j-access.in:3025"           # Teleport Auth Server GRPC API address
client_key = "/var/lib/teleport/plugins/slack/auth.key" # Teleport GRPC client secret key
client_crt = "/var/lib/teleport/plugins/slack/auth.crt" # Teleport GRPC client certificate
root_cas = "/var/lib/teleport/plugins/slack/auth.cas"   # Teleport cluster CA certs

[slack]
token = "SLACK_oAUTH_TOKEN"             # Slack Bot OAuth token

[role_to_recipients]
"*" = "team"

[log]
output = "stderr" 
severity = "INFO" 

Launch another terminal and SSH into Teleport proxy to start the Slack plugin.

$ teleport-slack start

In your workstation, login to Teleport proxy as user john requesting privileged access.

$ tsh login --proxy=tele.j-access.in --auth=local --user=john --request-roles=dba

Immediately, you will get a Slack notification with the request.

Click the link and login to the web UI as Dave. You should see a request waiting for approval.

As soon as it is approved, John’s terminal shows that the login request is approved.

Conclusion

This scenario demonstrated integrating Teleport RBAC with Slack to perform just-in-time privilege escalation based on ChatOps. It highlights the need to configure time-bound SSH sessions to avoid credential exploitation and possible misconfiguration of database servers. The integration with Slack brings an intuitive alerting and approval process to the workflow. Teleport's RBAC combined with Slack delivers a simple yet powerful mechanism of just-in-time privilege escalation.

Learn more on how Teleport supports advanced authorization use cases using dual authorization, impersonation and moderated sessions.

Sign up for Teleport Cloud today!

Teleport Cloud offers a fully managed Teleport cluster so that you can quickly get started with Teleport and start protecting access to database servers instantly. We also recommend joining our Slack channel, where the Teleport community members hang out to discuss Teleport in production.