Version Control Best Practices With Teleport RBAC Roles

Imagine you've just deployed a working Teleport cluster and you're making changes to the Role Based Access Controls (RBAC) roles, fine-tuning all of your resource permissions, and making sure every role is following the principle of least privilege. You go for a week-long vacation, do some fishing and completely relax. Getting back you find that the DevOps intern you just hired made a bunch of changes to the roles screwing everything up. If you were managing all of your RBAC yaml locally, this could be a very annoying problem. If you were checking all of your roles into some kind of version control, however, it’s the easiest fix in the world (and probably wouldn’t have happened in the first place). Let’s dive in!

Almost everyone uses some kind of version control tool for their code. It’s like a magical time-travel machine, keeping track of every change you make, so, if you ever mess up something, you can easily go back to a previous version and start fresh. No more sweating over lost code! It’s a no-brainer. You would never push a change directly to a production server from your local machine, so why would you do the same with your Teleport RBAC?

Checking in your Teleport RBAC roles into some kind of version control lets you collaborate with other engineers smoothly. It keeps everyone in the loop, showing who changed what, when and why. This is crucial from a day-to-day standpoint as well as an audit and compliance one. While you might get away with managing your RBAC roles yourself locally if you have a tiny team, version control becomes increasingly important at scale for three main reasons: maintaining accountability thus providing a clear chain of custody, having a history of changes, and reducing overall complexity.

Accountability

PR reviews

Picture this: You've made some changes to an RBAC role to add a new RDS database, but you accidentally granted “admin” privileges to that pesky intern. Luckily however, instead of just pushing these changes to your Teleport cluster, first you have to submit a PR into your team’s RBAC repo. Your colleague catches the mistake and flags it in a comment. Problem solved before it even happened. PR reviews ensure code quality and maintain accountability. They allow other team members to review your changes, provide feedback and catch potential issues or bugs. And just like when talking about code changes, getting pull-request reviews on role changes promotes collaboration and knowledge sharing within the team. Some organizations even require reviews from members of their security team, on RBAC DevOPs PR’s. This keeps everyone in the loop while heightening accountability and security. Plus, if something does go wrong later, you can always trace back the changes to the responsible person and have a transparent discussion, making sure it never happens again

Branch controls

Branch controls are another key feature of version controlling your RBAC roles and are great for enforcing good PR practices; specifically of course, preventing pushing to the main branch. By preventing direct pushes to the main branch, you enforce a controlled workflow where any role changes need to be reviewed before they are merged and deployed to your cluster. Just like in your software development life cycle, engineers should have to create “feature” branches, make their changes, and then go through the PR process. It adds an extra layer of scrutiny and ensures that role changes are thoroughly reviewed before merging. This helps maintain the integrity of the RBAC structure and prevents accidental or unauthorized modifications.

Complexity

Another important reason to use a git-ops flow for managing access control is complexity. Rather than having to manually update all your various resources across your organization every time you want to make an access change, if you have a centralized git-ops flow that manages all of your resources, onboarding an employee can be as simple as a one-line change. In fact, if your Okta groups are automatically synced to centralized Terraform directory groups, you may not have to make any code changes at all.

At Teleport we actually utilize a git-ops approach leveraging Terraform to manage our own access controls internally. In Terraform we have a directory group for every app and every role, providing a comprehensive encoding of our Okta instance. This allows us to leverage attribute based access control (ABAC) to provide access to resources based on an individual’s team, company role, or even title. We then restrict group-admin permissions to only allow managing groups not already managed in Terraform. This elimination of console access, greatly reduces our attack surface and forces any changes in access control to be made through our git-ops flow. Also this paper trail of PR reviews provides a detailed, rich audit log of all access changes across our ecosystem. This flow is more secure, simpler to manage, and makes achieving audit and compliance standards much easier. For more information on our Okta Terraform approach check out this blog post from our Director of IT.

Speaking of audit and compliance, a great way to lint and evaluate roles and groups is Teleport’s predicate language. With predicate you can write tests that make sure all of your roles and resources fit custom constraints. This works great with a git-ops flow as well, allowing you to run automated tests on every proposed access change before the changes are merged and go into effect, improving security and reducing complexity.

History of changes

Having a detailed record of role changes is like having a time machine at your disposal.

Rollback to old versions

Sometimes, as we discussed earlier, changes sneak through the review process that can break things. Here version control once again comes to the rescue. If you find yourself in a pickle, you can effortlessly roll back to a previous version of the RBAC role and undo the problematic changes. It's like hitting the undo button on your mistakes. This ability to roll back provides a safety net and peace of mind, especially when scaling infrastructure and engineering headcount.

Bug hunting and troubleshooting:

Just like with code changes if you ever make a change and accidentally, such as lock out your entire EMEA team from Google Cloud (which I totally never did by the way), but you're not sure what exactly caused the issue, with version control, you can pinpoint the exact commit or change that introduced the bug. It helps in isolating and understanding the root cause, making troubleshooting a breeze. You can analyze the history, collaborate with teammates and find a fix more efficiently.

Code auditing and compliance

In certain industries, particularly the finance and medical sectors, auditing and compliance for infrastructure access are crucial. Version control tools provide a comprehensive history of RBAC role changes, allowing auditors to track the evolution of your role-based access controls. Coupled with Teleport’s built-in advanced audit logging, this makes it easy to ensure that regulatory requirements and security standards are met.

Implementation of version control for Teleport roles

At Teleport we recommend some standard operating workflows tying in all of the above principles.

YAML Approach

This approach is a slightly more basic way of handling Teleport roles in git-ops.

1. Create a repository that will contain all of your role yaml files. Organize this how you see fit.
2. Lock the main branch to require all commits originate from reviewed PR’s.
   • It’s a good idea to require approval from at least one member of your security team specifically.
3. Set up an automated CI/CD flow to apply role changes upon merge to main.
   • Create a dedicated RBAC-administrator role in Teleport and restrict the ability to modify and create roles to this specific admin.
   • Leveraging MachineID to provide an identity and Teleport role to the automated worker applying these changes is highly recommended. For more information and a detailed walkthrough check out our webinar on using GitHub actions with Teleport here.

Terraform Approach

This approach is more advanced and works better at scale, leveraging Teleport’s Terraform provider.

1. Follow the steps here to configure Teleport's Terraform provider in your Teleport cluster. This will allow you to make resource changes in your cluster using a Terraform service user.
2. Create a repository that will contain all of your terraform files. Organize this how you see fit.
3. Lock the main branch to require all commits originate from reviewed PR’s.
   • It’s a good idea to require approval from at least one member of your security team specifically.
4. In Terraform, create a directory group for every resource, application, and role in your organization
5. Map these directory groups to your Identity Provider groups.
6. Set up an automated CI/CD flow to apply terraform changes upon merge to main. This will leverage the Teleport Terraform provider you set up in step 1.
   • Leveraging MachineID to provide an identity and Teleport role to the automated worker applying these changes is highly recommended. For more information and a detailed walkthrough check out our webinar on using GitHub actions with Teleport here.
7. Take away console access to prevent manual changes from occurring.
   • Restrict group-admin permissions to only allow managing groups not already managed by terraform.
8. Alert on changes outside terraform
   • Connect your IDP to your security information events monitoring tool (SIEM) and set up alerts to fire anytime a group change is made by anyone other than the Terraform service user.
9. Rinse and repeat.
   • Continue to terraform all of your resources that admins usually configure via console. Ideally you’ll get to a state where none of your resources are manually managed anymore.

If you follow these steps, scaling your Teleport cluster as your team grows, and managing your existing RBAC roles will be easier and more secure than ever.

Conclusion

In a nutshell, version control gives you the power to maintain accountability, track role changes and roll back when things go awry. It enhances collaboration, keeps your infrastructure secure and makes troubleshooting a breeze.

If you aren't using Teleport yet, you can get started today with a 14-day free trial.