Automatically Register Resources with Teleport

You can use Teleport's API to automatically register resources in your infrastructure with your Teleport cluster.

Teleport already supports the automatic discovery of Kubernetes clusters in AWS, Azure, and Google Cloud, as well as servers on Amazon EC2. To support other resources and cloud providers, you can use the API to write your own workflow.

In this guide, we will demonstrate some libraries you can use to automatically register resources with Teleport. We will use an example you can run locally on your workstation.

Automatic registration consists of the following steps:

• Look up resources in your infrastructure from a service discovery solution, e.g., the Kubernetes API server, Consul, or your cloud provider's APIs.
• Use the Teleport gRPC API to look up resources registered with Teleport.
• For any resources in your infrastructure that are not registered with Teleport, use the Teleport API to spin up a new Teleport service or register the resource with an existing Teleport service.
• For any resources that are registered with Teleport but not in your infrastructure, use the Teleport API to deregister the resource and, if necessary, remove the Teleport service proxying the resource.

Prerequisites

• A running Teleport cluster version 16.4.8 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

• The tctl admin tool and tsh client tool.

  Visit Installation for instructions on downloading tctl and tsh.

• Docker installed on your workstation. Get Started With Docker.
• Go version 1.22 or above installed on your workstation. See the Go download page. You will not need to be familiar with Go to complete this guide, though Go knowledge is required if you want to build a production-ready Teleport client application.

Step 1/5. Set up your Go project

Download the source code for our minimal API client:

git clone https://github.com/gravitational/teleport -b branch/v16
cd examples/service-discovery-api-client

For the rest of this guide, we will show you how to set up this API client and explore the way it uses Teleport's API to synchronize Teleport Application Service instances with an external service discovery solution.

Step 2/5. Define RBAC resources

Clients that communicate with Teleport's gRPC API assume the identity of a Teleport user and role in order to authenticate to the Auth Service, which authorizes the identity to perform certain API actions.

Your own Teleport user also requires permissions to impersonate the API client's user in order to generate credentials for the client.

Create a user and role for the client application

Our client application will authenticate to Teleport as a user with permissions to create join tokens, list applications, and delete records of Application Service instances registered with the Auth Service. The user will also have permissions to access applications with any label (app_labels), which is necessary for listing applications.

Define a user and role with appropriate permissions for the client application by adding the following content to a file called register-apps.yaml:

kind: role
version: v5
metadata:
  name: register-apps
spec:
  allow:
    app_labels:
      '*': '*'
    rules:
      - resources: ['token']
        verbs: ['create']
      - resources: ['app']
        verbs: ['list']
      - resources: ['app_server']
        verbs: ['delete']
---
kind: user
metadata:
  name: register-apps
spec:
  roles: ['register-apps']
version: v2

Create the user and role:

tctl create -f register-apps.yaml
role 'register-apps' has been createduser "register-apps" has been created

Enable impersonation of the client application

As with all Teleport users, the Teleport Auth Service authenticates the register-apps user by issuing short-lived TLS credentials. In this case, we will request the credentials manually by impersonating the register-apps role and user.

If you are running a self-hosted Teleport Enterprise deployment and are using tctl from the Auth Service host, you will already have impersonation privileges.

To grant your user impersonation privileges for register-apps, create a file called register-apps-impersonator.yaml defining a role:

kind: role
version: v5
metadata:
  name: register-apps-impersonator
spec:
  allow:
    impersonate:
      roles:
      - register-apps
      users:
      - register-apps

Create the register-apps-impersonator role:

tctl create -f register-apps-impersonator.yaml
role 'register-apps-impersonator' has been created

Assign the register-apps-impersonator role to your Teleport user by running the appropriate commands for your authentication provider:

You will now be able to generate signed certificates for the register-apps role and user.

Step 3/5. Export an identity for the client application

Like all Teleport users, register-apps needs signed credentials in order to connect to your Teleport cluster. You will use the tctl auth sign command to request these credentials for your plugin.

The following tctl auth sign command impersonates the register-apps user, generates signed credentials, and writes an identity file to the local directory:

tctl auth sign --user=register-apps --out=auth.pem

The identity file, auth.pem, includes both TLS and SSH credentials. Your client application uses the SSH credentials to connect to the Proxy Service, which establishes a reverse tunnel connection to the Auth Service. The plugin uses this reverse tunnel, along with your TLS credentials, to connect to the Auth Service's gRPC endpoint.

Step 4/5. Write the client application

In this step, we will walk you through the example client application.

Our demo application watches for containers running RabbitMQ, a popular open source message broker, in order to register (and, if necessary, deregister) its management API with Teleport. It does so by:

• Fetching RabbitMQ management API endpoints registered with Teleport.
• Fetching RabbitMQ containers.
• If a RabbitMQ container does not correspond to a management API endpoint, register the container's management API endpoint with Teleport by creating a join token and running a new Teleport Application Service container.
• If a RabbitMQ API endpoint registered with Teleport does not correspond to a RabbitMQ container, delete the corresponding Application Service container and deregister the RabbitMQ API endpoint.

Imports

The program, which you can find in examples/service-discovery-api-client/main.go, imports the following packages:

The client imports the following third-party code:

Global declarations

The program defines constants in a visible location so, later on, it's easier to make them configurable:

const (
	// Assign proxyAddr to the host and port of your Teleport Proxy Service instance
	proxyAddr      string = ""
	teleportImage  string = "public.ecr.aws/gravitational/teleport-distroless:16.4.8"
	initTimeout           = time.Duration(30) * time.Second
	updateInterval        = time.Duration(5) * time.Second
	tokenTTL              = time.Duration(5) * time.Minute
	networkName    string = "bridge"
	managementPort string = "15672"
	tokenLenBytes         = 16
	rabbitMQImage string  = "rabbitmq:3-management"
)

We will use these constants later in the program. They define some values we may want to change later, including:

Below the const declaration is the following type declaration:

type tokenDemoApp struct {
	dockerClient   *docker.Client
	teleportClient *teleport.Client
}

This is the only type we will declare in the program. It includes pointers to clients for the Docker daemon API and Teleport Auth Service API. This program will initialize the clients, then call methods of tokenDemoApp.

Get the management endpoint URLs of registered applications

Our program fetches the applications that are registered with Teleport. Later, it will compare these applications with the currently running application containers.

Teleport represents registered resources in two ways:

• Dynamic resources: These are configuration documents that you have applied against your cluster, e.g., app, kube_cluster, and db resources. Teleport automatically finds an instance of the appropriate service to proxy these.
• Service instances: These are Teleport services proxying specific resources in your infrastructure, which you specify in the service's configuration file.

In our application, we're creating one Application Service instance per app, so we'll list Application Service instances and examine the resources they are proxying. Other client applications may need to look up dynamically registered resources instead.

Here is the method that the application uses to fetch the URLs of registered applications:

func (t *tokenDemoApp) listRegisteredAppURLs(ctx context.Context) (map[string]types.AppServer, error) {
	m := make(map[string]types.AppServer)

	for {
		req := proto.ListResourcesRequest{
			ResourceType: "app_server",
			Limit:        10,
		}
		resp, err := t.teleportClient.ListResources(
			ctx,
			req,
		)

		if err != nil {
			return nil, trace.Wrap(err)
		}

		for _, r := range resp.Resources {
			if p, ok := r.(types.AppServer); ok {
				m[p.GetApp().GetURI()] = p
			}
		}

		// No more pages to request
		if resp.NextKey == "" {
			break
		}

		req.StartKey = resp.NextKey
	}

	return m, nil
}

tokenDemoApp.teleportClient is the *Client type from Teleport's API library. The ListResources method of the *Client type queries the Teleport API for a list of resources with the parameters named in a proto.ListResourcesRequest.

Since results are paginated, listRegisteredAppURLs executes this query in a for loop. As long as the query's response has a nonempty NextKey field, indicating the key to use to look up the start of the next page, the function executes the query again using the next key.

Eventually, we will expand this program to reconcile registered applications with application containers by comparing two maps (i.e., hash tables):

• A map of URL strings to types.AppServers, representing the applications registered with Teleport.
• A map where keys are URLs of management API endpoints for our RabbitMQ containers.

The listRegisteredAppURLs function generates the first map by iterating through the result of ListResources and, for each resources that is a types.AppServer, inserts the URL of its proxied application as a key within the map, assigning the corresponding types.AppServer as its value.

In the next section, we will show you how to generate the second map.

func (c *Client) GetApps(ctx context.Context) ([]types.Application, error)

func (c *Client) GetKubernetesServers(ctx context.Context) ([]types.KubeServer, error)

Get the management endpoint URLs of application containers

The next function fetches the URLs of our RabbitMQ containers:

func (t *tokenDemoApp) listAppContainerURLs(ctx context.Context, image string) (map[string]struct{}, error) {
	c, err := t.dockerClient.ContainerList(ctx, dtypes.ContainerListOptions{
		Filters: filters.NewArgs(filters.KeyValuePair{
			Key:   "ancestor",
			Value: image,
		}),
	})
	if err != nil {
		return nil, trace.Wrap(err)
	}

	l := make(map[string]struct{})

	for _, r := range c {
		b, ok := r.NetworkSettings.Networks[networkName]
		// Not connected to the chosen network, so skip it
		if !ok {
			continue
		}

		u, err := url.Parse("http://"+net.JoinHostPort(
			b.IPAddress,
			managementPort,
		))

		if err != nil {
			return nil, trace.Wrap(err)
		}

		l[u.String()] = struct{}{}
	}

	return l, nil
}

This function uses the tokenDemoApp.dockerClient field, the Docker API client library's Client type, to send the Docker daemon a request to list containers. The dtypes.ContainerListOptions struct instructs the Docker daemon to list only the containers that have the image we specify in the image parameter.

For each container returned by the Docker daemon, we look up the container's IP address within a predetermined network (the default bridge network). We know in advance that all RabbitMQ containers will have the management port opened, so we use the container's IP address and the management port to compose a URL and insert it into the map that we will return from this function.

Note that the map this function returns assigns URL strings to empty structs. In Go, the empty struct consumes no memory. Go programs often use the empty struct for the values of hash tables, since a program can search for a key in constant time without using the value.

Create a token for a new Application Service instance

After reconciling the map of registered applications with the map of application containers, we will need to:

• Launch Application Service instances to proxy unregistered application containers.
• Delete Application Service instances that no longer correspond to a running application.

To launch a new Application Service instance, we will create a join token. Teleport's services can join a cluster by presenting a token to the Auth Service.

The code below generates a join token for a new Application Service instance:

func cryptoRandomHex(len int) (string, error) {
	randomBytes := make([]byte, len)
	if _, err := rand.Read(randomBytes); err != nil {
		return "", trace.Wrap(err)
	}
	return hex.EncodeToString(randomBytes), nil
}

func (t *tokenDemoApp) createAppToken(ctx context.Context) (string, error) {
	n, err := cryptoRandomHex(tokenLenBytes)
	if err != nil {
		return "", trace.Wrap(err)
	}

	tok, err := types.NewProvisionTokenFromSpec(
		n,
		time.Now().Add(tokenTTL),
		types.ProvisionTokenSpecV2{
			Roles: types.SystemRoles{types.RoleApp},
		})

	if err := t.teleportClient.CreateToken(ctx, tok); err != nil {
		return "", trace.Wrap(err)
	}
	return n, nil
}

The example above demonstrates how to use *client.Client.CreateToken to join a resource to a Teleport cluster using a token. If you have an instance of a Teleport service running already (e.g., the Application Service), it is simpler to join the resource to the cluster dynamically using a method similar to *client.Client.CreateApp.

func (c *Client) CreateApp(ctx context.Context, app types.Application) error

spec:
  allow:
    rules:
      - resources: ['app']
        verbs: ['create']

cryptoRandomHex is a copy of a function that Teleport defines internally. It uses the crypto/rand package to generate random bytes, then converts them to a string in hexadecimal format, which we will use for the join token. You can use any secure cryptographic technique to generate a join token.

*tokenDemoApp.createAppToken calls cryptoRandomHex and uses the result by calling types.NewProvisionTokenFromSpec. This returns a specification for a join token that we will use when sending a request to the Teleport API, which we do via t.teleportClient.CreateToken.

In this case, we assign our token the TTL we configured earlier, plus the types.RoleApp role. This indicates to the Auth Service that the token is for an Application Service instance.

CreateToken returns an error if token creation fails. If it succeeds, we can return the token to the caller of *tokenDemoApp.createAppToken so it can run a new Application Service instance with the token.

func (c *Client) GetTokens(ctx context.Context) ([]types.ProvisionToken, error)

spec:
  allow:
    rules:
      - resources: ['token']
        verbs: ['list', 'read']

Start an Application Service container

Our program needs a way to launch Application Service instances. To do this, we will use the token we created earlier, plus the URL of a RabbitMQ management API endpoint, to launch a container:

func (t *tokenDemoApp) startApplicationServiceContainer(
	ctx context.Context,
	token string,
	u url.URL,
) error {

	name := strings.ReplaceAll(u.Hostname(), ".", "-")
	resp, err := t.dockerClient.ContainerCreate(
		ctx,
		&container.Config{
			Image: teleportImage,
			Entrypoint: strslice.StrSlice{
				"/usr/bin/dumb-init",
				"teleport",
				"start",
				"--roles=app",
				"--auth-server=" + proxyAddr,
				"--token=" + token,
				"--app-name=rabbitmq-" + name,
				"--app-uri=" + u.String(),
			},
		},
		nil,
		nil,
		nil,
		"",
	)
	if err != nil {
		return trace.Wrap(err)
	}

	err = t.dockerClient.ContainerStart(
		ctx,
		resp.ID,
		dtypes.ContainerStartOptions{},
	)
	if err != nil {
		return trace.Wrap(err)
	}

	return nil
}

The Teleport Application Service redirects traffic to a subdomain of your Teleport Web UI address to a registered application. We need a name for the application that is URL safe but does not clash with other registered applications. In this case, we use the IP addresses of our RabbitMQ containers, with dots replaced by hyphens.

Next, we create the container. The executable that serves as the container's entrypoint is the same one that the teleport image uses by default, but with additional flags that launch the container as an Application Service instance and set it to proxy our RabbitMQ container.

Finally, we use the ID of the container we created to run the container.

Remove an Application Service instance

Alongside creating Application Service containers, our client application will reconcile registered applications with running containers by removing unnecessary Application Service instances:

func (t *tokenDemoApp) pruneAppServiceInstance(ctx context.Context, p types.AppServer) error {
	host := p.GetHostname()

	if err := t.teleportClient.DeleteApplicationServer(
		ctx,
		p.GetNamespace(),
		p.GetHostID(),
		p.GetName(),
	); err != nil {
		return trace.Wrap(err)
	}

	fmt.Println("Deleted unnecessary Application Service record:", p.GetName())

	// Don't check errors when removing the container, since it may already
	// have been removed.
	t.dockerClient.ContainerStop(ctx, host, container.StopOptions{})
	t.dockerClient.ContainerRemove(ctx, host, dtypes.ContainerRemoveOptions{})

	fmt.Println("Deleted unnecessary Application Service container:", host)
	return nil
}

This function takes a types.AppServer, deregisters it from Teleport, and removes its associated Application Service container.

While Teleport deregisters stale Application Service records automatically, this can take some time after stopping an Application Service instance.

desktop.SetExpiry(time.Now().Add(10 * time.Minute))

To deregister the Application Service instance manually, we call the *Client.DeleteApplicationServer method, using the p parameter of the pruneAppServiceInstance function to get the namespace, host ID, and name of the Application Service instance to delete.

Next, this function stops and removes the Application Service container associated with the types.AppServer we want to prune. The hostname of an Application Service record is the same as the ID of the Application Service container, so we can pass the hostname to t.dockerClient.ContainerStop and t.dockerClient.ContainerRemove.

Reconcile application containers with registered applications

We have declared a number of functions to list, add, and remove Application Service instances and application containers. Next, we will use these functions to implement our reconciliation logic:

func (t *tokenDemoApp) reconcileApps() error {
	ctx := context.Background()
	apps, err := t.listRegisteredAppURLs(ctx)
	if err != nil {
		return trace.Wrap(err)
	}

	urls, err := t.listAppContainerURLs(ctx, rabbitMQImage)
	if err != nil {
		return trace.Wrap(err)
	}

	for u, _ := range urls {
		if _, ok := apps[u]; ok {
			continue
		}
		tok, err := t.createAppToken(ctx)
		if err != nil {
			return trace.Wrap(err)
		}
		fmt.Println("Created a new application token for URL: " + u)

		r, err := url.Parse(u)
		if err != nil {
			return trace.Wrap(err)
		}

		err = t.startApplicationServiceContainer(ctx, tok, *r)
		if err != nil {
			return trace.Wrap(err)
		}
		fmt.Println("Started an Application Service container to proxy URL: " + u)
	}

	for a, p := range apps {
		_, ok := urls[a]
		if ok {
			continue
		}

		if err := t.pruneAppServiceInstance(ctx, p); err != nil {
			return trace.Wrap(err)
		}
	}
	return nil
}

*tokenDemoApp.reconcileApps works by calling listAppContainerURLs and listRegisteredAppURLs to generate maps of registered applications and running application containers. It then iterates through the URLs within each map's keys, checking for the presence of the URL in one map within the other map.

If a URL within the map of application containers is not present within the map of registered applications, it means that one application is not yet registered, so we generate a token and start a Teleport Application Service instance.

If a URL within the map of registered applications is not present within the map of application containers, it means that there is an unnecessary Application Service instance, and we call pruneAppServiceInstance to remove it.

Initialize clients

The *tokenDemoApp.reconcileApps method performs a single reconciliation. The next step is to initialize our API clients so we can run the reconciliation in a loop within the entrypoint of our program:

func newTokenDemoApp() *tokenDemoApp {
	ctx := context.Background()
	ctx, cancel := context.WithTimeout(ctx, initTimeout)
	defer cancel()
	creds := teleport.LoadIdentityFile("auth.pem")

	t, err := teleport.New(ctx, teleport.Config{
		Addrs:       []string{proxyAddr},
		Credentials: []teleport.Credentials{creds},
	})
	if err != nil {
		panic(err)
	}
	fmt.Println("Connected to Teleport")

	d, err := docker.NewClientWithOpts(
		docker.WithAPIVersionNegotiation(),
	)
	if err != nil {
		panic(err)
	}
	fmt.Println("Connected to the Docker daemon")

	return &tokenDemoApp{
		teleportClient: t,
		dockerClient:   d,
	}

}

client, which we alias as teleport here, is Teleport's library for setting up an API client. Our plugin initializes a Teleport client by calling client.LoadIdentityFile to obtain a client.Credentials. It then uses the client.Credentials to call client.New, which connects to the Teleport Proxy Service specified in the Addrs field using the provided identity file.

The newTokenDemoApp function also initializes a Docker client. It uses version negotiation (docker.WithAPIVersionNegotiation) to avoid errors due to mismatches between your client's API version and the Docker daemon's.

The entrypoint

We tie our application together with the entrypoint function, main:

func main() {
	fmt.Println("Starting the application")
	app := newTokenDemoApp()

	k := time.NewTicker(updateInterval)
	defer k.Stop()
	for {
		<-k.C
		if err := app.reconcileApps(); err != nil {
			panic(err)
		}
	}
}

The main function calls newTokenDemoApp to initialize our API clients. It then calls time.NewTicker, which returns a Go channel that the entrypoint will use to run the reconciliation routine. A channel is a Go primitive that manages communication between concurrent routines.

After we create a ticker, the entrypoint receives from the ticker's channel (k.C) whenever the updateInterval elapses. It blocks until it receives from the channel, calling app.reconcileApps every interval.

Step 5/5. Test your client application

Run the client application to see how Teleport can register and deregister resources in your infrastructure to synchronize with your service discovery solution.

Make sure you have pulled the Teleport container image so the application can create containers based on it:

docker image pull public.ecr.aws/gravitational/teleport-distroless:16.4.8

In your project directory, run the following command:

go run main.go
Starting the applicationConnected to TeleportConnected to the Docker daemon

In a new terminal, run three new RabbitMQ containers:

for i in {1..3}; do docker run -d rabbitmq:3-management; done;

The terminal where you ran the application should show output similar to the following:

Created a new application token for URL: http://172.17.0.4:15672
Started an Application Service container to proxy URL: http://172.17.0.4:15672
Created a new application token for URL: http://172.17.0.3:15672
Started an Application Service container to proxy URL: http://172.17.0.3:15672
Created a new application token for URL: http://172.17.0.2:15672
Started an Application Service container to proxy URL: http://172.17.0.2:15672

Verify that the RabbitMQ instances are registered with Teleport:

tsh apps ls
Application         Description Type Public Address           Labels------------------- ----------- ---- ------------------------ -------------------rabbitmq-172-17-0-2             HTTP rabbitmq-172-17-0-2.3... teleport.dev/originrabbitmq-172-17-0-3             HTTP rabbitmq-172-17-0-3.3... teleport.dev/originrabbitmq-172-17-0-4             HTTP rabbitmq-172-17-0-4.3... teleport.dev/origin

Next, stop one of your RabbitMQ containers:

docker stop $(docker ps --filter "ancestor=rabbitmq:3-management" -q --last 1)

The terminal where you ran the application should show output similar to the following:

Deleted unnecessary Application Service record: rabbitmq-172-17-0-4
Deleted unnecessary Application Service container: 63facaa3033a

You should now see two registered applications when you run tsh apps ls.

Next steps

We have implemented a Teleport API client that keeps registered applications up to date with our running Docker containers. You can use Teleport's API to automatically sync your registered Teleport resources with your own service discovery solution.

Consult examples

Teleport includes its own automatic resource discovery solution, the Teleport Discovery Service, and you can consult its source to see how Teleport implements its discovery logic.

The Teleport code repository contains examples of production-ready Teleport API clients. While we currently do not maintain plugins that auto-discover resources, you can use these examples to see how to implement configuration parsing, retries, and other tasks.

Provision the client application with short-lived credentials

In this example, we used the tctl auth sign command to fetch credentials for the program you wrote. For production usage, we recommend provisioning short-lived credentials via Machine ID, which reduces the risk of these credentials becoming stolen. View our Machine ID documentation to learn more.