Skip to main content
Version: 17.x (unreleased)

AWS Database Discovery

Teleport can be configured to discover AWS-hosted databases automatically and register them with your Teleport cluster.

In this guide, we will show you how to set-up AWS database auto-discovery.

How it works​

Teleport database auto-discovery involves two components:

  1. The Teleport Discovery Service that watches for new databases or changes to previously discovered databases. It dynamically registers each discovered database as a db resource in your Teleport cluster. It does not need connectivity to the databases it discovers.
  2. The Teleport Database Service that monitors the dynamic db resources registered by the Discovery Service. It proxies communications between users and the database.

Prerequisites​

  • A running Teleport cluster version 17.0.0-dev or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

  • The tctl admin tool and tsh client tool.

    Visit Installation for instructions on downloading tctl and tsh.

  • An AWS account with permissions to create and attach IAM policies.
  • A host to run the Teleport Discovery Service.
  • A host to run the Teleport Database Service.
  • One or more databases hosted on AWS.

Step 1/8. Install Teleport​

Install Teleport on the host(s) that will run the Teleport Discovery Service and Teleport Database Service.

The Database Service needs network connectivity to databases, whereas the Discovery Service does not.

Install Teleport on your Linux server:

  1. Assign edition to one of the following, depending on your Teleport edition:

    EditionValue
    Teleport Enterprise Cloudcloud
    Teleport Enterprise (Self-Hosted)enterprise
    Teleport Community Editionoss

  2. Get the version of Teleport to install. If you have automatic agent updates enabled in your cluster, query the latest Teleport version that is compatible with the updater:

    $ TELEPORT_DOMAIN=example.teleport.com
    $ TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/automaticupgrades/channel/default/version | sed 's/v//')"

    Otherwise, get the version of your Teleport cluster:

    $ TELEPORT_DOMAIN=example.teleport.com
    $ TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/ping | jq -r '.server_version')"

  3. Install Teleport on your Linux server:

    $ curl https://cdn.teleport.dev/install-v15.4.11.sh | bash -s ${TELEPORT_VERSION} edition

    The installation script detects the package manager on your Linux server and uses it to install Teleport binaries. To customize your installation, learn about the Teleport package repositories in the installation guide.

Step 2/8. Discovery Service IAM permissions​

Grant the Discovery Service access to credentials that it can use to authenticate to AWS.

  • If you are running the Discovery Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
  • If you are running the Discovery Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
  • Otherwise, you must use environment variables

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Have multiple sources of AWS credentials?

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Discovery Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Grant AWS IAM permissions​

Attach the following AWS IAM permissions to the Discovery Service AWS IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DocumentDBDiscovery",
            "Effect": "Allow",
            "Action": "rds:DescribeDBClusters",
            "Resource": "*"
        }
    ]
}
StatementPurpose
DocumentDBDiscoveryDiscover Amazon DocumentDB Clusters.

Step 3/8. Deploy the Discovery Service​

Create a Teleport config file​

Create a teleport.yaml config file on the host that will run the Discovery Service. You can leave the discovery_group aws-example value as-is or change it to something you find more descriptive.

version: v3
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  proxy_server: "teleport.example.com:443"
auth_service:
  enabled: false
proxy_service:
  enabled: false
ssh_service:
  enabled: false
discovery_service:
  enabled: true
  discovery_group: "aws-example"

This config file enables the discovery_service and configures it to join the Teleport cluster. It also sets the Discovery Service's discovery_group. We will configure the discovery_group aws-example dynamically in a later step, so that we can control the Discovery Service's configuration without restarting the Discovery Service.

warning

Discovery Service exposes a configuration parameter - discovery_service.discovery_group - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services.

When running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud resources or a different value if they are watching different cloud resources.

It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.

  • 2 Discovery Services configured with discovery_group: "prod" polling data from Production account.
  • 2 Discovery Services configured with discovery_group: "staging" polling data from Staging account.

Generate a join token​

The Discovery Service requires a valid join token to join your Teleport cluster. Run the following tctl command and save the token output in /tmp/token on the server that will run the Discovery Service:

$ tctl tokens add --type=discovery --format=text
abcd123-insecure-do-not-use-this
Alternative methods

For users with a lot of infrastructure in AWS, or who might create or recreate many instances, consider alternative methods for joining new EC2 instances running Teleport:

Start the Discovery Service​

Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Discovery Service.

On the host where you will run the Discovery Service, enable and start Teleport:

$ sudo systemctl enable teleport
$ sudo systemctl start teleport

You can check the status of the Discovery Service with systemctl status teleport and view its logs with journalctl -fu teleport.

Step 4/8. Discovery group config​

Create a file aws-example-discovery-config.yaml and save it on a host where you can use tctl.

version: v1
kind: "discovery_config"
metadata:
  name: "example"
spec:
  # Only Discovery services in this discovery_group will use the matchers from
  # this dynamic config.
  discovery_group: aws-example
  aws:
    # Database types. Valid options are:
    # 'docdb' - discovers and registers Amazon DocumentDB databases.
    # 'elasticache' - discovers Amazon ElastiCache Redis databases.
    # 'memorydb' - discovers Amazon MemoryDB Redis databases.
    # 'opensearch' - discovers Amazon OpenSearch Redis databases.
    # 'rds' - discovers Amazon RDS and Aurora databases.
    # 'rdsproxy' - discovers Amazon RDS Proxy databases.
    # 'redshift' - discovers Amazon Redshift databases.
    # 'redshift-serverless' - discovers Amazon Redshift Serverless databases.
  - types: ["docdb"]
    regions: ["us-east-1"]
    # AWS tags to match where "*" is a wildcard.
    # You can use "*": "*" to match all database resource tags.
    tags:
      "env": "prod" # this example matches only databases tagged with env=prod.
    # Optionally assume an AWS IAM role before calling the AWS API to discover
    # databases.
    assume_role:
      role_arn: "" # "arn:aws:iam::123456789012:role/example-role"
      # Optional AWS external ID that the Database Service will use to assume
      # a role in an external AWS account.
      external_id: "" # "example-external-id"

Create the discovery_config:

$ tctl create aws-example-discovery-config.yaml

The Discovery Service we configured earlier is in the same discovery_group as this discovery_config and will begin using the discovery_config to discover AWS databases. Once it discovers databases, the Discovery Service will register them as db resources in your Teleport cluster.

info

A Teleport db resource represents the specification of a database that a Teleport Database Service can then use to provide access to the database. When a Database Service instance matches the db resource via label selectors, it will begin to heartbeat the database by regularly creating short-lived db_server resources in your Teleport cluster. Tools like tsh db ls and tctl db ls will only display db_server resources, i.e. databases that a Database Service instance is providing access to.

Step 5/8. List registered databases​

Before we set-up the Database Service to provide access to discovered databases, we should check that the Discovery Service is actually discovering databases.

You can list dynamically registered databases with tctl. The Discovery Service adds the label teleport.dev/origin: cloud to every database it registers with your Teleport cluster.

Verify that the Discovery Service has registered db resources for databases that you expect it to have discovered:

$ tctl get db

Or check for a specific database:

$ tctl get db/<database-name>

Refer to Discovery Service troubleshooting if you do not see db resources corresponding to databases that you think should be discovered.

discovered database names

Each discovered database's name will have additional identifying information appended to it to ensure uniqueness. That additional info may include:

  • endpoint type (e.g. "reader" endpoint)
  • matcher type
  • AWS region
  • AWS account ID.

For example, if an RDS Aurora database named "my-postgres" is discovered in AWS account "123456789012" in region us-east-1, it would be named "my-postgres-rds-aurora-us-east-1-123456789012" in Teleport.

A discovered database also has a shorter display name that consists of only the AWS database name and the endpoint type, for example "my-postgres" or "my-postgres-reader". Either the full name or the display name can be used for tctl and tsh commands, but if the display name is ambiguous, then you will have to use the full name.

You can override the database name by applying the TeleportDatabaseName AWS tag to the AWS database resource - this is used as the db name verbatim, i.e. additional identifying information will not be appended to it.

Step 6/8. Database Service IAM permissions​

Grant the Database Service access to credentials that it can use to authenticate to AWS.

  • If you are running the Database Service on an EC2 instance, you may use the EC2 Instance Metadata Service method
  • If you are running the Database Service in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
  • Otherwise, you must use environment variables

Teleport will detect when it is running on an EC2 instance and use the Instance Metadata Service to fetch credentials.

The EC2 instance should be configured to use an EC2 instance profile. For more information, see: Using Instance Profiles.

Have multiple sources of AWS credentials?

Teleport's AWS client loads credentials from different sources in the following order:

  • Environment Variables
  • Shared credentials file
  • Shared configuration file (Teleport always enables shared configuration)
  • EC2 Instance Metadata (credentials only)

While you can provide AWS credentials via a shared credentials file or shared configuration file, you will need to run the Database Service with the AWS_PROFILE environment variable assigned to the name of your profile of choice.

If you have a specific use case that the instructions above do not account for, consult the documentation for the AWS SDK for Go for a detailed description of credential loading behavior.

Create an AWS IAM role for the Database Service and attach the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DocumentDBConnectAsIAMRole",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:aws:iam::aws-account-id:role/DatabaseUserRole"
            ]
        },
        {
            "Sid": "DocumentDBCheckDomainURL",
            "Effect": "Allow",
            "Action": "rds:DescribeDBClusters",
            "Resource": "*"
        }
    ]
}
StatementPurpose
DocumentDBConnectAsIAMRoleAssume an IAM role to connect to a DocumentDB cluster with IAM authentication.
DocumentDBCheckDomainURLValidate a domain's URL if it was auto-discovered by the Discovery Service.

Step 7/8. Deploy the Database Service​

Configure database connectivity​

Unlike the Discovery Service, the Database Service must have network connectivity to databases to provide access to them for your Teleport cluster. You will need to ensure that several network reachability requirements are met for the Database Service:

  1. The Database Service has a network route to database(s)
  2. The Database Service has a network route to your Teleport cluster
  3. The Database Service security group allows outbound traffic to database(s)
  4. The Database Service security group allows outbound traffic to your Teleport cluster
  5. The database(s) security group(s) allow inbound traffic from the Database Service

In the highly likely case that your databases are deployed in private subnets with strict security group(s) attached to them, you will typically need to deploy a Database Service instance in the same VPC, possibly in the same subnet(s), and with a security group attached to it that the database(s) allow inbound traffic from. The Teleport Database Service will probably need a route to the public internet via an AWS NAT gateway or internet gateway in order to reach your Teleport cluster.

This is not an exhaustive list of network requirements or suggestions, as that will depend on your specific networking setup.

Create a Teleport config file​

Create a teleport.yaml config file on the host that will run the Discovery Service:

version: v3
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  proxy_server: "teleport.example.com:443"
auth_service:
  enabled: false
proxy_service:
  enabled: false
ssh_service:
  enabled: false
db_service:
  enabled: true
  resources:
    - labels:
        "account-id": "*"
        "region": "us-east-1"
        "teleport.dev/cloud": "AWS"
        "teleport.dev/origin": "cloud"

This config file enables the db_service and configures it to join the Teleport cluster. The section db_service.resources is a list of label selectors. The Database Service will match db resources that have these labels and begin to heartbeat the databases by regularly creating short-lived db_server resources in your Teleport cluster.

In this case, it will match auto-discovered AWS databases in the us-east-1 region from any AWS account ("*" is a wildcard and it can be used as a label key and/or value). You can make it match more specific databases by adjusting the label selectors.

tip

The AWS tags attached to AWS databases are imported as Teleport db labels in addition to some other identifying metadata. Refer to Database Labels Reference for more information about available database labels.

Generate a join token​

The Database Service requires a valid join token to join your Teleport cluster. Run the following tctl command and save the token output in /tmp/token on the server that will run the Database Service:

$ tctl tokens add --type=db --format=text
abcd123-insecure-do-not-use-this
Alternative methods

For users with a lot of infrastructure in AWS, or who might create or recreate many instances, consider alternative methods for joining new EC2 instances running Teleport:

Start the Database Service​

Configure the Database Service to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Database Service.

On the host where you will run the Database Service, enable and start Teleport:

$ sudo systemctl enable teleport
$ sudo systemctl start teleport

You can check the status of the Database Service with systemctl status teleport and view its logs with journalctl -fu teleport.

Step 8/8. List database servers​

To confirm that the Database Service is proxying discovered databases, run the following tctl command:

## adjust these comma-separated filtering labels as needed
$ tctl db ls teleport.dev/origin=cloud,teleport.dev/cloud=AWS,region=us-east-1,account-id="*"

If you do not see the databases that you expected, then refer to Database Service troubleshooting below.

note

This guide shows you how to set-up AWS database auto-discovery with a Discovery Service and Database Service, but does not cover database user provisioning.

Additional Teleport RBAC configuration and possibly IAM configuration may also be required to connect to the discovered databases via Teleport.

Refer to the appropriate guide in Enroll AWS Databases for information about database user provisioning and configuration.

Next​

Troubleshooting​

Discovery Service troubleshooting​

First, check if any databases have been discovered. To do this, you can use the tctl get db command and check if the expected databases have already been registered with your Teleport cluster.

If some databases do not appear in the list, check if the Discovery Service selector labels match the missing database tags or look into the Discovery Service logs for permission errors.

Check that the Discovery Service is running with credentials for the correct AWS account. It can discover resources in another AWS account, but it must be configured to assume a role in the other AWS account if that's the case.

Check if there is more than one Discovery Services running:

$ tctl inventory status --connected

If you are running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud databases or a different value if they are watching different cloud databases. If this is not configured correctly, a typical symptom is db resources being intermittently deleted from your Teleport cluster's registry.

Database Service troubleshooting​

Databases do not appear in tctl db ls​

If the tctl get db command returns the discovered databases you expect, but the tctl db ls command does not include them, check that you have set the db_service.resources section correctly, for example:

db_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod"

If the section is correctly configured, but databases still do not appear, check that you have the correct permissions to list databases in Teleport. You should have a Teleport role that matches the database labels and allows the "read" and "list" verbs for db and db_server objects. Here's an example that grants those permissions for every database in your cluster:

kind: role
version: v6
metadata:
  name: view-all-databases
spec:
  allow:
    db_labels:
      '*': '*'
    rules:
      - resources: [db_server, db]
        verbs: [read, list]

Errors when connecting to a database​

note

This section assumes you have already provisioned a database user and configured Teleport RBAC for that database user by following a specific guide in Enroll AWS Databases.

If there are connection errors when you try to connect to a database, then first check if there are multiple db_server heartbeat resources for the target database: tctl get db_server/yourDatabaseName. If there are, it means that multiple Teleport Database Service instances are proxying the database - this is an HA setup that will complicate troubleshooting. Teleport will choose one of those Database Service instances at random to proxy the connection and if one of them can't reach the database endpoint or lacks permissions, then you will see random connection errors.

Even if connection errors are consistent, you should scale down or reconfigure your Teleport Database Service instances such that only one matches the target db while you are troubleshooting errors. Verify that there is only one db_server with tctl get db_server/yourDatabaseName and then try the connection again.

Check the Teleport Database Service logs with DEBUG level logging enabled and look for network or permissions errors.

Refer to Troubleshooting Database Access for more general troubleshooting steps.

Additionally, a guide specific to the type of database in Enroll AWS Databases. may have more specific troubleshooting information.