Amazon EC2 SSH Session Recording and Auditing with Teleport

This blog is Part IV in a series about identity-based access management of AWS resources. In Part I, we covered how to use OSS Teleport to access Amazon EC2 instances running in private subnets. Part II explained implementing identity-based access via SSO integration with Okta. Part III covered the steps to configure privilege escalation for just-in-time access requests. In Part IV, we will guide you through the steps to configure SSH session recording and auditing.

EC2 SSH session recording with Teleport

Auditing and reviewing SSH sessions of Linux servers is critical for achieving compliance. While managed services such as Amazon CloudWatch and Amazon CloudTrail provide auditing capabilities, they are mostly confined to the Amazon EC2 control-plane events. Once a user logs in to a Linux-based Amazon EC2 instance, customers don't have visibility into the session.

Teleport Audit, one of the key pillars of the platform, delivers the below capabilities:

• Unified resource catalog - A dynamic list of infrastructure resources, including servers, Kubernetes clusters databases, applications, and more
• Audit log - Acts as a single source of truth for all the security events generated by Teleport
• Live session view - A dynamic list of live sessions across all protocols and environments
• Session recording - An interactive mechanism to play back sessions recorded and stored in a centralized location

Teleport session recording is available either in proxy mode or node recording mode. The proxy mode makes it possible to record SSH sessions in a bastion host setup and can record SSH sessions based on OpenSSH and agent forwarding. It is helpful when gradually transitioning large server fleets to Teleport. The default node recording mode is more secure and is designed to be extensible to support enhanced session recording with eBPF.

In the previous tutorial, we demonstrated how to perform privilege escalation with Teleport RBAC and Slack. A contractor requests a timebound SSH session with sudo privileges, approved by a team leader.

We will extend the scenario to cover the audit and review of the SSH session initiated by the contractor. We will also walk through the steps to use Amazon DynamoDB and Amazon S3 storage for persisting Teleport events and session recordings in AWS infrastructure.

Note: This scenario builds upon the previous articles in this series that include running AWS bastion host, and setting up just-in-time access. In order to follow this tutorial, you will need provisioned AWS infrastructure (Amazon EC2 instances, a VPC, security groups, internet gateway, NAT gateway and routing tables) and have Teleport Enterprise installed. You should have a Teleport proxy configured in the public subnet with two Teleport nodes running in the private subnet. The configuration should also have a user with Teleport roles access, auditor, editor, and a user with a role that includes permissions to request controlled privilege escalation.

Tip: The auditor role is a preset role that allows reading cluster events, audit logs, and playing back session records. The team leader persona below assumes the role of an auditor since he needs to access the session recordings and audit logs.

Step 1 - Accessing and playing back SSH session videos

Start by requesting an SSH session for John with elevated privileges and letting Dave approve it.

$ tsh login --proxy=tele.j-access.in --auth=local --user=john --request-roles=dba

Now, John accesses the database server as a "DBA" and quickly becomes the server's "root" user.

One of his tasks is to install Docker on the server to run a few containers. So, John proceeds with the installation of Docker in the database server.

Apart from installing Docker, John deliberately disables the AppArmor Linux security module, designed to implement mandatory access control (MAC) policies. This step, which is not expected to be performed on production servers, leads to a security risk.

After a few minutes, John logs out of the database server, and eventually, his privileges as a root user expire because of the RBAC policy.

During the regular maintenance check, an automated script reports that the mandatory AppArmor module is missing in the database server. Dave now wants to access the recording of John's last SSH session to identify the problem.

Dave can log in to the Web UI with credentials and accesses the session recordings available under the Activity tab of the left navigation bar.

Clicking the play button opens a new tab with the video that plays back the entire session.

Dave can also play the session video within the terminal. For this, Dave copies the session Id from the web UI and runs the below command:

tsh play --proxy=tele.j-access.in 0b724167-82e7-4c4b-95d2-a79d2c899170

It’s also possible to print the session events in JSON to the terminal by adding the –format=json switch.

tsh play --proxy=tele.j-access.in --format=json 0b724167-82e7-4c4b-95d2-a79d2c899170

Apart from the session recordings, Teleport also maintains a detailed event log on the auth server. If you have access to the Teleport auth server, SSH into it to access the event log.

sudo tail -f /var/lib/teleport/log/events.log

For example, the below events are generated and logged when the user John logs in to Teleport:

By default, Teleport stores the event logs and session recordings in /var/lib/teleport/log directory of the auth server. For high availability and durability, it is possible to move them to AWS Cloud. The event logs can be stored in Amazon DynamoDB while storing the session recordings in Amazon S3 buckets.

Step 2 - Storing SSH audit logs and session recordings in AWS

We will start by creating the IAM roles assumed by the Amazon EC2 instance running the Teleport auth server.

In AWS IAM Console, create a new role for the EC2 instance running the Teleport auth server.

Search for S3FullAccess and AmazonDynamoDBFullAccess policies and select both of them.

Call the role "TeleportAudit" and click on create.

Let's assign this role to the EC2 instance running the Teleport auth server.

Select the EC2 instance and choose to modify the IAM role under the security options.

Assign the TeleportAudit role and save the settings.

Now, the Teleport auth server running within the EC2 instance can talk to Amazon DynamoDB and Amazon S3 to store the logs and session recordings.

Create an S3 bucket in the same region where the Teleport service is running. For this tutorial, we call it j-access.in-teleport-logs.

It's time to point Teleport auth service to DynamoDB and S3. We will do this by adding the below settings to the /etc/teleport.yaml file:

 storage:
    region: ap-south-1
    audit_sessions_uri: "s3://j-access.in-teleport-logs/records"
    audit_events_uri:  ['dynamodb://teleport-events', 'file:///var/lib/teleport/audit/events', 'stdout://']

The lines starting with audit_sessions_uri and audit_events_uri are responsible for redirecting the content to AWS.

Restart the Teleport service on the auth server and wait for the events to flow.

If you check the records directory of the S3 bucket, you will find the *.tar files corresponding to the session recordings.

The same is the case with the event log. Explore the DynamoDB table, "teleport-events", to see the event log.

When you access the event logs and session recordings from the command line or the web UI, Teleport fetches them from DynamoDB and S3 bucket respectively.

Conclusion

In this tutorial, we explored Teleport audit concepts and the ability to move the event logs and session recordings to AWS Cloud. First, we created roles based on the preset and custom role definitions that provided access to the event logs and session recordings. Next, we extended the configuration to store the logs and recordings in Amazon DynamoDB and Amazon S3, which makes the data highly available and durable. Session recording with Teleport is more granular as compared to AWS Systems Manager.

Security and compliance with Teleport: Single sign-on (SSO), just-in-time (JIT) access, and session recording are the basic foundation for access control security compliance. Sign up for Teleport Cloud today!