Database Access with AWS RDS Proxy

This guide will help you to set up Teleport to access your AWS RDS proxies.

Prerequisites

• AWS account with RDS Proxy instances and permissions to create and attach IAM policies.
• Any RDS Proxy instances intended for connection through Teleport must have TLS enabled.
• A host, e.g., an EC2 instance, where you will run the Teleport Database Service.

tsh login --proxy=teleport.example.com [email protected]
tctl status
Cluster  teleport.example.com
Version  12.1.1
CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

tsh login --proxy=myinstance.teleport.sh [email protected]
tctl status
Cluster  myinstance.teleport.sh
Version  12.1.2
CA pin   sha256:sha-hash-here

Step 1/5. Create a Database Service configuration

The Database Service requires a valid auth token to connect to the cluster. Generate one by running the following command against your Teleport Auth Service and save it in /tmp/token on the node that will run the Database Service:

tctl tokens add --type=db

Install Teleport on the host where you will run the Teleport Database Service:

Use the appropriate commands for your environment to install your package.

Download Teleport's PGP public key
sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc
Source variables about OS version
source /etc/os-release
Add the Teleport APT repository for v12. You'll need to update this
file for each major release of Teleport.
Note: if using a fork of Debian or Ubuntu you may need to use '$ID_LIKE'
and the codename your distro was forked from instead of '$ID' and '$VERSION_CODENAME'.
Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L42-L67
echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v12" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null
sudo apt-get update
sudo apt-get install teleport

Source variables about OS version
source /etc/os-release
Add the Teleport YUM repository for v12. You'll need to update this
file for each major release of Teleport.
Note: if using a fork of RHEL/CentOS or Amazon Linux you may need to use '$ID_LIKE'
and the codename your distro was forked from instead of '$ID'
Supported versions are listed here: https://github.com/gravitational/teleport/blob/master/build.assets/tooling/cmd/build-os-package-repos/runners.go#L133-L153
sudo yum-config-manager --add-repo $(rpm --eval "https://yum.releases.teleport.dev/$ID/$VERSION_ID/Teleport/%{_arch}/stable/v12/teleport.repo")
sudo yum install teleport
Tip: Add /usr/local/bin to path used by sudo (so 'sudo tctl users add' will work as per the docs)
echo "Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin" > /etc/sudoers.d/secure_path
Optional:  Use DNF on newer distributions
$ sudo dnf config-manager --add-repo https://rpm.releases.teleport.dev/teleport.repo
$ sudo dnf install teleport

curl https://get.gravitational.com/teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
shasum -a 256 teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
cd teleport
sudo ./install

dpkg -i ~/Downloads/teleport-ent_12.1.1_$SYSTEM-ARCH.deb
Selecting previously unselected package teleport-ent.
(Reading database ... 30810 files and directories currently installed.)
Preparing to unpack teleport-ent_12.1.1_$SYSTEM_ARCH.deb ...
Unpacking teleport-ent 12.1.1 ...
Setting up teleport-ent 12.1.1 ...

rpm -i ~/Downloads/teleport-ent-12.1.1.$SYSTEM-ARCH.rpm
warning: teleport-ent-12.1.1.$SYSTEM-ARCH.rpm: Header V4 RSA/SHA512 Signature, key ID 6282c411: NOKEY

curl https://get.gravitational.com/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
shasum -a 256 teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-bin.tar.gz
cd teleport-ent
sudo ./install

curl https://get.gravitational.com/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz.sha256
<checksum> <filename>
curl -O https://cdn.teleport.dev/teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz
shasum -a 256 teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz
Verify that the checksums match
tar -xvf teleport-ent-v12.1.1-linux-$SYSTEM-ARCH-fips-bin.tar.gz
cd teleport-ent
sudo ./install

Grant the Teleport Database Service access to credentials that it can use to authenticate to AWS. If you are running the Teleport Database Service on an EC2 instance, you should use the EC2 Instance Metadata Service method. Otherwise, you must use environment variables:

AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>

Create the Database Service configuration. Replace teleport.example.com with your Teleport Proxy address or Teleport Cloud tenant (e.g. mytenant.teleport.sh), and replace us-west-1 with your preferred region:

teleport db configure create \   -o file \   --proxy=teleport.example.com:443 \   --token=/tmp/token \   --rdsproxy-discovery=us-west-1

The command will generate a Database Service configuration with RDS Proxy instances auto-discovery enabled on the us-west-1 region and place it at the /etc/teleport.yaml location.

Step 2/5. Create an IAM policy for Teleport

Teleport needs AWS IAM permissions to be able to discover and register RDS Proxy instances.

Teleport can bootstrap IAM permissions for the Database Service based on its configuration using the teleport db configure bootstrap command. You can use this command in automatic or manual mode:

• In automatic mode, Teleport will attempt to create appropriate IAM policies and attach them to the specified IAM identity (user or role). This requires IAM permissions to create and attach IAM policies.
• In manual mode, Teleport will print required IAM policies. You can then create and attach them manually using the AWS management console.

Run one of the following commands on your Database Service node:

teleport db configure bootstrap -c /etc/teleport.yaml --attach-to-user TeleportUser

teleport db configure bootstrap -c /etc/teleport.yaml --attach-to-role TeleportRole

teleport db configure bootstrap -c /etc/teleport.yaml --manual

See the full bootstrap command reference.

Step 3/5. Start the Database Service

Configure the Teleport Database Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Teleport Database Service.

sudo systemctl start teleport

sudo teleport install systemd -o /etc/systemd/system/teleport.service
sudo systemctl enable teleport
sudo systemctl start teleport

The Database Service will discover all RDS Proxy instances according to the configuration and register them in the cluster. In addition to the primary endpoints of the RDS Proxy instances, their custom endpoints will also be registered.

Step 4/5. Configure database user credentials

The Database Service connects to an RDS Proxy instance using IAM authentication. In addition, the RDS Proxy instance must also be able to connect to the RDS DB instance or Aurora DB cluster using pre-configured database user credentials.

First, use Secrets Manager to store sets of user name and password credentials. You create a separate Secrets Manager secret for each database user account that RDS Proxy connects to on the RDS DB instance or Aurora DB cluster. The password associated with the secret must match the database password for that user in the target database.

For example, the following AWS CLI command creates a Secrets Manager secret for database user alice:

aws secretsmanager create-secret \  --name rdsproxy_alice --description "database user alice" \  --secret-string '{"username":"alice","password":"password_for_alice"}'

Next, create an IAM role with a policy that can access these secrets, for example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": [
                "arn:aws:secretsmanager:us-west-1:account_id:secret:rdsproxy_alice",
                "arn:aws:secretsmanager:us-west-1:account_id:secret:rdsproxy_anotheruser"
            ]
        },
        {
            "Action": "kms:Decrypt",
            "Resource": "arn:aws:kms:us-west-1:account_id:key/key_id",
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "secretsmanager.us-east-2.amazonaws.com"
                }
            }
        }
    ]
}

The IAM role should have the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "rds.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Lastly, modify the RDS Proxy authentication settings to use the created IAM role and Secrets Manager secrets. Make sure IAM authentication is set to Required:

aws rds modify-db-proxy --db-proxy-name my-rds-proxy --auth AuthScheme=SECRETS,SecretArn=arn-of-teleport-access-user,\IAMAuth=REQUIRED AuthScheme=SECRETS,SecretArn=arn-of-non-teleport-user,IAMAuth=DISABLED ...

Step 5/5. Connect

Once the Database Service has started and joined the cluster, log in to see the registered databases:

tsh login --proxy=teleport.example.com --user=alice
tsh db ls
Name                         Description                     Labels
---------------------------- ------------------------------- -------
rds-proxy                    RDS Proxy in us-west-1          ...
rds-proxy-my-reader-endpoint RDS Proxy endpoint in us-west-1 ...

To retrieve credentials for a database and connect to it:

tsh db connect --db-user=alice --db-name=dev rds-proxy

To log out of the database and remove credentials:

tsh db logout rds-proxy

Troubleshooting

Certificate error

If your tsh db connect error includes the following text, you likely have an RDS database created before July 28, 2020, which presents an X.509 certificate that is incompatible with Teleport:

x509: certificate relies on legacy Common Name field, use SANs instead

AWS provides instructions to rotate your SSL/TLS certificate.

No credential providers error

If you see the error NoCredentialProviders: no valid providers in chain in Database Service logs then Teleport is not detecting the required credentials to connect via AWS IAM permissions. Check whether the credentials or security role has been applied in the machine running the Teleport Database Service.

Timeout errors

The Teleport Database Service needs connectivity to your database endpoints. That may require enabling inbound traffic on the database from the Database Service on the same VPC or routing rules from another VPC. Using the nc program you can verify connections to databases:

nc -zv postgres-instance-1.sadas.us-east-1.rds.amazonaws.com 5432
Connection to postgres-instance-1.sadas.us-east-1.rds.amazonaws.com (172.31.24.172) 5432 port [tcp/postgresql] succeeded!

Maximum policy size exceeded errors

Due to IAM and STS character limits, you may encounter one of the following errors in the Database Service logs when large numbers of databases are registered:

• LimitExceeded: Maximum policy size of 2048 bytes exceeded for user <iam-user>
• LimitExceeded: Maximum policy size of 10240 bytes exceeded for role <iam-role>

For reference, a user policy can maintain permissions for approximately 6 Redshift databases, or 20 RDS databases due to the IAM policy character limits. A role policy can maintain permissions for approximately 30 Redshift databases, or 100 RDS databases.

One way to work around this is to manually manage the IAM policy used for database connections, and use a wildcard "*" for "Resource" to reduce the policy size:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "rds-db:connect",
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "redshift:GetClusterCredentials",
            "Resource": "*"
        }
    ]
}

Another option is to deploy Database Access in a Highly Available (HA) configuration where databases can be sharded to separate Database Services with different IAM roles.

If the limit is exceeded for a user policy, it is also recommended to use an IAM role for the Database Service instead.

Next steps

• Learn how to restrict access to certain users and databases.

• View the High Availability (HA) guide.

• Take a look at the YAML configuration reference.

• See the full CLI reference.

• Learn more on Setting up database credentials in AWS Secrets Manager and Setting up AWS Identity and Access Management (IAM) policies for RDS Proxy