Skip to main content
Version: 17.x (unreleased)

Kubernetes Clusters Discovery

Kubernetes Clusters Discovery allows Kubernetes clusters hosted on cloud providers to be discovered and enrolled automatically.

While discovering a new Kubernetes cluster, Teleport does not install any component on the cluster. Instead, it requires direct access to the cluster's API and minimal access permissions.

Supported clouds​

  • AWS: Discovery for AWS EKS clusters.
  • Azure: Discovery for Azure AKS clusters.
  • Google Cloud: Discovery for Google Kubernetes Engine clusters.

How Kubernetes Clusters Discovery works​

Kubernetes Clusters Discovery consists of two steps:

Polling cloud APIs​

The Teleport Discovery Service is responsible for scanning the configured cloud providers and identifying if any Kubernetes clusters match a set of filtering labels. When the process identifies a new Kubernetes cluster, it creates a dynamic resource within Teleport. This resource includes information imported from the cloud provider such as:

  • Name: Cluster name
  • Labels
    • Cluster tags.
    • Cluster location.
    • Identification of which cloud account the cluster belongs to — AWS Account ID / Azure Subscription ID.
tip

You can import the cluster under a different name into Teleport's registry. To achieve this, you must attach the following tag to the resources — EKS, AKS, GKE — in your cloud provider:

  • key: TeleportKubernetesName
  • value: desired name

The Discovery Service will check if the cluster includes the tag and use its value as the resource name in Teleport.

You should use this feature whenever there are clusters in different regions/cloud providers with the same name to prevent them from colliding in Teleport.

In addition to detecting new Kubernetes clusters, the Discovery Service also removes — from Teleport's registry — the Kubernetes clusters that have been deleted or whose tags no longer meet the filtering labels.

warning

Discovery Service exposes a configuration parameter - discovery_service.discovery_group - that allows you to group discovered resources into different sets. This parameter is used to prevent Discovery Agents watching different sets of cloud resources from colliding against each other and deleting resources created by another services.

When running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud resources or a different value if they are watching different cloud resources.

It is possible to run a mix of configurations in the same Teleport cluster meaning that some Discovery Services can be configured to watch the same cloud resources while others watch different resources. As an example, a 4-agent high availability configuration analyzing data from two different cloud accounts would run with the following configuration.

  • 2 Discovery Services configured with discovery_group: "prod" polling data from Production account.
  • 2 Discovery Services configured with discovery_group: "staging" polling data from Staging account.

The following snippet describes the different configuration options for the Discovery Service and their default values.

# This section configures the Discovery Service
discovery_service:
    enabled: "yes"
    discovery_group: "prod"
    aws:
       # AWS resource types. Valid options are:
       # eks - discovers and registers AWS EKS clusters
       # ec2 - discovers and registers AWS EC2 Machines
     - types: ["eks"]
       # AWS regions to search for resources from
       regions: ["us-east-1", "us-west-1"]
       # AWS resource tags to match when registering resources
       # Optional section: Defaults to "*":"*"
       tags:
         "env": "prod"
       # AWS role to assume when discovering resources in the AWS Account.
       # This value is an optional AWS role ARN to assume when polling EKS clusters
       assume_role_arn: arn:aws:iam::123456789012:role/iam-discovery-role
       # External ID is an optional value that should be set when accessing
       # your AWS account from a third-party service (delegated access).
       external_id: "example-external-id"
       # Optional role for which the Discovery Service should create the EKS access entry.
       # If not set, the Discovery Service will attempt to create the access
       # entry using its own identity.
       # If used, the role must match the role configured for a Teleport Kubernetes Service.
       setup_access_for_arn: arn:aws:iam::123456789012:role/kube-service-role
    # Matchers for discovering Azure-hosted resources.
    azure:
      # Azure resource types. Valid options are:
      # 'aks' - discovers and registers Azure AKS Kubernetes Clusters.
    - types: ["aks"]
      # Azure regions to search for resources from. Valid options are:
      # '*' - discovers resources in all regions (default).
      # Any valid Azure region name. List all valid regions using the Azure "az" cli: `az account list-locations -o table`
      regions: ["*"]
      # Azure subscription IDs to search resources from. Valid options are:
      # '*' - discovers resources in all subscriptions (default).
      # Any subscription_id: `az account subscription list -o table`
      subscriptions: ["*"]
      # Azure resource groups to search resources from. Valid options are:
      # '*' - discovers resources in all resource groups within configured subscription(s) (default).
      # Any resource_groups: `az group list -o table`
      resource_groups: ["*"]
      # Azure resource tag filters used to match resources.
      # Optional section: Defaults to "*":"*"
      tags:
        "env": "prod"
    # Matchers for discovering GCP-hosted resources.
    gcp:
      # GCP resource types. Valid options are:
      # 'gke' - discovers and registers GCP GKE Kubernetes Clusters.
    - types: ["gke"]
      # GCP location to search for resources from. Valid options are:
      # '*' - discovers resources in all locations (default).
      # Any valid GCP region or zone name.
      locations: ["*"]
      # GCP project ID
      project_ids: ["myproject"]
      # GCP resource tag filters used to match resources.
      # Optional section: Defaults to "*":"*"
      tags:
        "*" : "*"

Forwarding requests to the Kubernetes Cluster​

The Teleport Kubernetes Service is responsible for monitoring the dynamic resources created or updated by the Discovery Service and forwarding requests to the Kubernetes clusters they represent. To work correctly, it requires direct access to the target Kubernetes clusters and permissions to forward requests.

To turn on dynamic resource monitoring in the Kubernetes Service, you must configure the kubernetes_service.resources section as shown in the following snippet:

## This section configures the Kubernetes Service
kubernetes_service:
    enabled: "yes"
    # Matchers for dynamic Kubernetes cluster resources created with the "tctl create" command or by Kubernetes auto-discovery.
    resources:
    - labels:
        "*": "*" # can be configured to limit the clusters to watched by this service.
      aws:
       # AWS role to assume when accessing EKS clusters in the AWS Account.
       # This value is an optional AWS role ARN to assume when forwarding requests
       # to EKS clusters.
       assume_role_arn: arn:aws:iam::123456789012:role/iam-discovery-role
       # External ID is an optional value that should be set when accessing
	     # your AWS account from a third-party service (delegated access).
       external_id: "example-external-id"

When configuring the kubernetes_service.resources parameter, the Teleport Kubernetes Service is set to monitor EKS clusters that are discovered by the Teleport Discovery Service. The monitoring process involves a label matching mechanism to identify and manage the EKS clusters.

  1. Discovery and Label Matching: The Discovery Service identifies available EKS clusters within the AWS environment. The Teleport Kubernetes Service checks the labels of these clusters against the labels specified in the kubernetes_service.resources[].labels configuration, which is the selector array.
  2. Role Selection: The first selector in the array that matches the labels of an EKS cluster determines the role that the Kubernetes Service will assume. This role is essential for the Teleport Kubernetes Service to retrieve necessary cluster details from the AWS API. If no match is found, the Kubernetes Service defaults to its own identity.
  3. Interaction with AWS and Kubernetes APIs: Once a match is found and a role is assumed, the Teleport Kubernetes Service uses this role to access the AWS API. It retrieves information about the EKS cluster, such as configuration and status. Subsequently, the Teleport Kubernetes Service forwards requests to the Kubernetes API, enabling interaction with the cluster.

Both services — Discovery and Kubernetes — can be configured in the same Teleport process or separate processes.