{"token_count": 5537} # Terraform Database Auto-Discovery This guide shows you how to use Terraform to configure Teleport to discover Amazon databases and deploy a Teleport Database Service on Amazon Elastic Container Service (ECS) to provide access to those databases. Teleport Terraform modules currently support the following Amazon databases: - RDS Aurora - RDS MySQL - RDS Postgres ## How it works The `teleport-discovery-aws` Terraform module creates AWS IAM resources, an [AWS OIDC integration](https://goteleport.com/docs/ver/19.x/enroll-resources/application-access/cloud-apis/awsoidc-integration.md), and a Teleport `discovery_config` that the **Teleport Discovery Service** uses to discover databases in AWS. Teleport Cloud runs the Discovery Service for you. For a self-hosted Teleport cluster, you run a Discovery Service with a `discovery_group` that matches the group configured by the module. The `teleport-db-agent-aws` Terraform module deploys a **Teleport Database Service** as an ECS service. The module creates an ECS cluster and service, task IAM roles, CloudWatch log group, security group, and Teleport IAM join token. By default, the Database Service selects discovered databases in the same AWS account, region, and VPC. The Discovery Service only needs to be able to reach the Teleport Proxy and the AWS API - it does not need network access to the databases. The Database Service must be able to reach: 1. The Teleport cluster's public proxy endpoint 2. The AWS database endpoints ## Prerequisites - A running Teleport cluster accessible at a hostname with a valid TLS certificate. If you want to get started with Teleport, [sign up](https://goteleport.com/signup) for a free trial or [set up a demo environment](https://goteleport.com/docs/ver/19.x/get-started/deploy-community.md). - The `tctl` and `tsh` clients. Installing `tctl` and `tsh` clients 1. Determine the version of your Teleport cluster. The `tctl` and `tsh` clients must be at most one major version behind your Teleport cluster version. Send a GET request to the Proxy Service at `/v1/webapi/find` and use a JSON query tool to obtain your cluster version. Replace teleport.example.com:443 with the web address of your Teleport Proxy Service: **Mac/Linux** ``` $ TELEPORT_DOMAIN=teleport.example.com:443 $ TELEPORT_VERSION="$(curl -s https://$TELEPORT_DOMAIN/v1/webapi/find | jq -r '.server_version')" ``` **Windows - Powershell** ``` $ $TELEPORT_DOMAIN = "teleport.example.com:443" $ $TELEPORT_VERSION = (Invoke-RestMethod -Uri "https://${TELEPORT_DOMAIN}/v1/webapi/find").server_version ``` 2. Follow the instructions for your platform to install `tctl` and `tsh` clients: **Mac** Download the signed macOS .pkg installer for Teleport, which includes the `tctl` and `tsh` clients: ``` $ curl -O https://cdn.teleport.dev/teleport-${TELEPORT_VERSION?}.pkg ``` In Finder double-click the `pkg` file to begin installation. --- DANGER Using Homebrew to install Teleport is not supported. The Teleport package in Homebrew is not maintained by Teleport and we can't guarantee its reliability or security. --- **Windows - Powershell** ``` $ curl.exe -O https://cdn.teleport.dev/teleport-v$TELEPORT_VERSION-windows-amd64-bin.zip Unzip the archive and move the `tctl` and `tsh` clients to your %PATH% NOTE: Do not place the `tctl` and `tsh` clients in the System32 directory, as this can cause issues when using WinSCP. Use %SystemRoot% (C:\Windows) or %USERPROFILE% (C:\Users\) instead. ``` **Linux** All of the Teleport binaries in Linux installations include the `tctl` and `tsh` clients. For more options (including RPM/DEB packages and downloads for i386/ARM/ARM64) see our [installation page](https://goteleport.com/docs/ver/19.x/installation/single-machine.md). ``` $ curl -O https://cdn.teleport.dev/teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ tar -xzf teleport-v${TELEPORT_VERSION?}-linux-amd64-bin.tar.gz $ cd teleport $ sudo ./install Teleport binaries have been copied to /usr/local/bin ``` Connecting with TLS routing disabled This guide's commands assume your Teleport cluster uses TLS routing (`proxy_listener_mode: multiplex`), where the `tctl` and `tsh` clients reach every Teleport service through the Proxy Service's web address on port `443`. If you're not sure whether this applies to your cluster, check with whoever manages it. If your cluster uses separate listener ports instead, adjust ports as follows: - **`tsh` commands** (e.g., `tsh login --proxy=...`): continue using the Proxy Service web address on port `3080` (or `443` if behind a load balancer). Do not change these to port `3025`. - **Direct `tctl` or Auth Service API commands**: use port `3025` for the Auth Service gRPC listener: ``` $ tctl status --auth-server=teleport.example.com:3025 ``` * Terraform `1.5.7` or later. * An AWS account containing at least one RDS or Aurora database. * A VPC and at least one subnet in which to deploy the Database Service. * AWS credentials for the AWS Terraform provider. * Teleport credentials for the Teleport Terraform provider. For a local Terraform run, follow the [Teleport Terraform provider local demo](https://goteleport.com/docs/ver/19.x/configuration/terraform-provider/local.md). This guide deploys the Database Service to the same AWS account, region, and VPC as the databases. To access databases in other AWS accounts, configure `database_service_resources` and the required cross-account IAM roles instead of using the module's default resource matcher. ## Step 1/6. Prepare the Discovery Service **Teleport Cloud** Teleport Cloud runs a Discovery Service in the `cloud-discovery-group` group. No Discovery Service installation is required. **Self-hosted** Install Teleport on a host that can reach your Teleport Proxy Service. The host needs network access to the AWS API, but it does not need network access to any databases. To install Teleport binaries on your Linux server, the recommended installation method is the cluster install script. This script is served by your Teleport cluster's Proxy Service and automatically selects the correct version, edition, and installation mode to match your cluster. 1. Remove any existing Teleport binaries on your system: ``` $ sudo rm -f /usr/local/bin/{tsh,teleport,tctl,tbot,fdpass-teleport,teleport-update} ``` 2. Assign teleport.example.com:443 to your Teleport cluster hostname and port, but not the scheme (https\://). 3. Run your cluster's install script: ``` $ curl "https://teleport.example.com:443/scripts/install.sh" | sudo bash ``` Create a join token for the Discovery Service: The Discovery Service requires a valid join token to join your Teleport cluster. Run the following `tctl` command and save the token output in `/tmp/token` on the server that will run the Discovery Service: ``` $ tctl tokens add --type=discovery --format=text abcd123-insecure-do-not-use-this ``` Create `/etc/teleport.yaml` on the host. Assign teleport.example.com:443 to the public address of your Teleport Proxy Service: ``` version: v3 teleport: join_params: token_name: "/tmp/token" method: token proxy_server: "teleport.example.com:443" auth_service: enabled: false proxy_service: enabled: false ssh_service: enabled: false discovery_service: enabled: true discovery_group: "aws-databases" ``` Start the Discovery Service. The instructions depend on how you installed the Discovery Service and whether your system supports systemd: **Package Manager** Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. On the host where you will run the Discovery Service, enable and start Teleport: ``` $ sudo systemctl enable teleport $ sudo systemctl start teleport ``` You can check the status of the Discovery Service with `systemctl status teleport` and view its logs with `journalctl -fu teleport`. **TAR Archive** Configure the Discovery Service to start automatically when the host boots up by creating a systemd service for it. On the host where you will run the Discovery Service, create a systemd service configuration for Teleport, enable the Teleport service, and start Teleport: ``` $ sudo teleport install systemd -o /etc/systemd/system/teleport.service $ sudo systemctl enable teleport $ sudo systemctl start teleport ``` You can check the status of the Discovery Service with `systemctl status teleport` and view its logs with `journalctl -fu teleport`. **No systemd** On the host where you will run the Discovery Service, start Teleport: ``` $ sudo teleport start --config=/etc/teleport.yaml ``` Teleport runs in the foreground and outputs logs for the services it is running. ## Step 2/6. Configure the Terraform providers Configure the [AWS Terraform provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs) with the region containing the VPC and databases. Add the AWS and Teleport providers to your Terraform configuration: ``` terraform { required_version = ">= 1.5.7" required_providers { aws = { source = "hashicorp/aws" version = "~> 6.0" } teleport = { source = "terraform.releases.teleport.dev/gravitational/teleport" version = "~> 18.0" } } } provider "aws" { region = "us-east-1" } provider "teleport" {} ``` Configure the Teleport Terraform provider with credentials for your cluster. For a local Terraform run, log in to Teleport and export credentials for the provider from the same shell in which you will run Terraform: ``` $ tsh login $ eval "$(tctl terraform env)" ``` For CI/CD and other remote execution environments, follow [Using the Teleport Terraform Provider](https://goteleport.com/docs/ver/19.x/configuration/terraform-provider.md) to configure Machine ID or workload identity credentials. ## Step 3/6. Configure the Terraform modules Add both modules and a security group ingress rule to your Terraform configuration. **Teleport Cloud** ``` module "aws_discovery" { source = "terraform.releases.teleport.dev/teleport/discovery/aws" version = "~> 18.0" teleport_proxy_public_addr = "example.teleport.sh:443" teleport_discovery_group_name = "cloud-discovery-group" aws_matchers = [{ types = ["rds"] regions = ["us-east-1"] tags = { "*" = ["*"] } }] } module "teleport_db_agent" { source = "terraform.releases.teleport.dev/teleport/db-agent/aws" version = "~> 18.0" teleport_proxy_public_addr = "example.teleport.sh:443" vpc_id = "vpc-0123456789abcdef0" ecs_service_subnets = [ "subnet-0123456789abcdef0", "subnet-abcdef01234567890", ] allow_database_modification = true database_types_for_default_iam_policy = ["rds"] } resource "aws_vpc_security_group_ingress_rule" "teleport_db_agent" { security_group_id = "sg-0123456789abcdef0" referenced_security_group_id = module.teleport_db_agent.security_group_id ip_protocol = "-1" } ``` **Self-hosted** ``` module "aws_discovery" { source = "terraform.releases.teleport.dev/teleport/discovery/aws" version = "~> 19.0" teleport_proxy_public_addr = "teleport.example.com:443" teleport_discovery_group_name = "aws-databases" aws_matchers = [{ types = ["rds"] regions = ["us-east-1"] tags = { "*" = ["*"] } }] } module "teleport_db_agent" { source = "terraform.releases.teleport.dev/teleport/db-agent/aws" version = "~> 19.0" teleport_proxy_public_addr = "teleport.example.com:443" vpc_id = "vpc-0123456789abcdef0" ecs_service_subnets = [ "subnet-0123456789abcdef0", "subnet-abcdef01234567890", ] allow_database_modification = true database_types_for_default_iam_policy = ["rds"] } resource "aws_vpc_security_group_ingress_rule" "teleport_db_agent" { security_group_id = "sg-0123456789abcdef0" referenced_security_group_id = module.teleport_db_agent.security_group_id ip_protocol = "-1" } ``` Replace the VPC, subnet, security group, and region values with values for your AWS environment. If the databases use different security groups, create one `aws_vpc_security_group_ingress_rule` resource for each security group. The discovery module's `aws_matchers` input controls which databases are registered with Teleport. The example discovers all RDS and Aurora databases in the configured region. Replace the wildcard `tags` selector to restrict discovery to databases with specific AWS tags. The database-agent module's default resource matcher selects registered databases with `account-id`, `region`, and `vpc-id` labels matching the AWS account, region, and VPC in which the ECS service runs. Setting `database_types_for_default_iam_policy = ["rds"]` adds permissions to describe RDS databases and connect with IAM authentication. Set `allow_database_modification = true` to also allow the Database Service to enable IAM database authentication. The default policy grants these RDS actions against all applicable resources. To restrict the permissions for a production deployment, pass a policy document to `ecs_task_role_inline_policy`. Statements with the `RDSAutoEnableIAMAuth`, `RDSConnect`, and `RDSFetchMetadata` SIDs override the corresponding default statements. The discovery module uses an AWS OIDC integration by default. The public proxy address of a self-hosted cluster must be reachable over HTTPS so AWS can use it as an OIDC identity provider. If the proxy is not publicly reachable, configure `discovery_service_iam_credential_source` to use an AWS IAM role available to the Discovery Service instead. AWS permits only one IAM OIDC provider for a given provider URL in an account. If the account already has an IAM OIDC provider for this Teleport cluster, set `create_aws_iam_openid_connect_provider = false` in the discovery module. ## Step 4/6. Apply the Terraform configuration Initialize Terraform and review the plan: ``` $ terraform init $ terraform apply ``` The discovery module should plan the following resources: - An AWS IAM role and policy for RDS discovery. - An AWS IAM OIDC provider and Teleport AWS OIDC integration. - A Teleport `discovery_config` for the selected RDS databases. The database-agent module should plan the following resources: - An ECS cluster, task definition, and service. - ECS task and execution IAM roles. - A CloudWatch log group. - A security group that permits outbound traffic. - A Teleport IAM join token restricted to the ECS task role. The plan should also include the inbound rule for the RDS security group. Review the plan carefully, then approve it. The ECS service runs two Database Service tasks by default. The tasks join the Teleport cluster with their AWS IAM identity, so the module does not store a secret join token in the ECS task definition. ## Step 5/6. Verify database discovery The Discovery Service registers matched databases as Teleport `db` resources. List the registered databases: ``` $ tctl get db ``` The Discovery Service adds the `teleport.dev/origin: cloud` label to each database that it registers. Verify that the output contains RDS databases from the configured AWS account and region. If expected databases are missing, check the discovery integration in the Teleport Web UI under **Zero Trust Access > Integrations**, then refer to the [Discovery Service troubleshooting](#discovery-service-troubleshooting) section. ## Step 6/6. Verify the Database Service After the ECS tasks start and match the registered databases, they create short-lived Teleport `db_server` resources. List the databases served by the Database Service: ``` $ tctl db ls teleport.dev/origin=cloud,teleport.dev/cloud=AWS,region=us-east-1 ``` If the command lists the expected RDS databases, auto-discovery and the Database Service are configured correctly. If `tctl get db` lists a database but `tctl db ls` does not, open the [Amazon ECS console](https://console.aws.amazon.com/ecs/v2/clusters) and inspect the task logs in the CloudWatch log group. Also verify the VPC route, RDS security group rule, resource labels, and ECS task IAM permissions. Refer to [Database Service troubleshooting](#database-service-troubleshooting) for additional checks. --- NOTE This guide configures RDS auto-discovery and deploys the Database Service. It does not provision database users or configure Teleport RBAC for those users. Follow the appropriate guide in [Enroll AWS Databases](https://goteleport.com/docs/ver/19.x/enroll-resources/database-access/enrollment/aws.md) before connecting to a database. --- ## Next steps - Learn how the Database Service uses [Dynamic Registration](https://goteleport.com/docs/ver/19.x/enroll-resources/database-access/guides/dynamic-registration.md). - [Connect to a database](https://goteleport.com/docs/ver/19.x/enroll-resources/database-access/guides.md). - Configure access to AWS databases in [external AWS accounts](https://goteleport.com/docs/ver/19.x/enroll-resources/database-access/enrollment/aws/aws-cross-account.md). ## Troubleshooting ### Inspect AWS discovery status Run `tctl discovery status` to check the state of dynamic Discovery Service configurations, when each Discovery Service instance last reported, and the latest AWS enrollment totals: ``` $ tctl discovery status --cloud=aws ``` The following example shows a healthy configuration that discovered and enrolled two resources: ``` Discovery config example-discovery: Discovery group: production Status: healthy Last run: 1 minute ago Service (00000000-0000-0000-0000-000000000000): Poll interval: 5 minutes Last update: 1 minute ago example-integration: AWS RDS: Previous sync: 1 minute ago (took 12s) Result: 2 found, 2 enrolled, 0 failed ``` A persistent `error` or `not reporting yet` status, a message that no Discovery Service instances are running, or a nonzero `failed` count indicates a discovery or enrollment problem. The command reports dynamic `discovery_config` resources. It does not include static discovery matchers configured in `teleport.yaml`. See [`tctl discovery status`](https://goteleport.com/docs/ver/19.x/reference/cli/tctl.md#tctl-discovery-status) for the full flag reference. ### Discovery Service troubleshooting First, check if any databases have been discovered. To do this, you can use the `tctl get db` command and check if the expected databases have already been registered with your Teleport cluster. If some databases do not appear in the list, check if the Discovery Service selector matches the missing databases' provider labels or tags, or check the Discovery Service logs for permission errors. Check that the Discovery Service is running with credentials for the intended AWS environment. To discover resources in another account or project, grant its principal access to the target account or project. Check if there is more than one Discovery Service instance running: ``` $ tctl inventory status --connected ``` If you are running multiple Discovery Service instances, you must ensure that each service is configured with the same `discovery_group` value if they are watching the same cloud databases or a different value if they are watching different cloud databases. If this is not configured correctly, a typical symptom is `db` resources being intermittently deleted from your Teleport cluster's registry. ### Database Service troubleshooting #### Databases do not appear in `tctl db ls` If the `tctl get db` command returns the discovered databases you expect, but the `tctl db ls` command does not include them, check that you have set the `db_service.resources` section correctly, for example: ``` db_service: enabled: true resources: - labels: "env": "prod" ``` If the section is correctly configured, but databases still do not appear, check that you have the correct permissions to list databases in Teleport. You should have a Teleport role that matches the database labels and allows the "read" and "list" verbs for `db` and `db_server` objects. Here's an example that grants those permissions for every database in your cluster: ``` kind: role version: v6 metadata: name: view-all-databases spec: allow: db_labels: '*': '*' rules: - resources: [db_server, db] verbs: [read, list] ``` #### Errors when connecting to a database --- NOTE This section assumes you have already provisioned a database user and configured Teleport RBAC for that database user by following the appropriate [database enrollment guide](https://goteleport.com/docs/ver/19.x/enroll-resources/database-access/enrollment.md). --- If there are connection errors when you try to connect to a database, then first check if there are multiple `db_server` heartbeat resources for the target database: `tctl get db_server/yourDatabaseName`. If there are, it means that multiple Teleport Database Service instances are proxying the database - this is an HA setup that will complicate troubleshooting. Teleport will choose one of those Database Service instances at random to proxy the connection and if one of them can't reach the database endpoint or lacks permissions, then you will see random connection errors. Even if connection errors are consistent, you should scale down or reconfigure your Teleport Database Service instances such that only one matches the target `db` while you are troubleshooting errors. Verify that there is only one `db_server` with `tctl get db_server/yourDatabaseName` and then try the connection again. Check the Teleport Database Service logs with DEBUG level logging enabled and look for network or permissions errors. Refer to the [Database Service troubleshooting guide](https://goteleport.com/docs/ver/19.x/enroll-resources/database-access/troubleshooting.md) for more general troubleshooting steps. The enrollment guide for your database may provide more specific troubleshooting information.