5 Key Steps to Securing a Default Kubernetes Cluster

5 Key Steps to Securing a Default Kubernetes Cluster - overview

Many day-to-day users of Kubernetes call it a win just to have their applications successfully up-and-running without any errors. Any further thoughts of best practices or security often get pushed to the backburner. With a little extra grooming of a default Kubernetes cluster, its security posture can be significantly strengthened. Learning to store secrets properly, limiting open networks and constructing containers that aren’t over-privileged becomes a must when dealing with production environments at scale.

This talk will focus around the default insecurities present in a Kubernetes cluster and 5 practical implementations that can be put in place to secure it. We’ll look at etcd and how it stores the cluster’s configuration data, including insecure secrets. We’ll discuss unrestricted pod-to-pod access and network policies, as well as enforcement of mutual TLS to encrypt internal traffic. Finally, we’ll take a look at pod-level security and best practices on that level, as well as securing access and RBAC/ABAC into the cluster itself. Users of Kubernetes will walk away with practical tools they can use immediately to tighten up the security of clusters in their own environments.

Key topics on 5 Key Steps to Securing a Default Kubernetes Cluster

• Kubernetes isn't safe by default.
• Securing etcd is critical to cluster security. This talk covers three ways to do that.
• Teleport provides secure access to Kubernetes clusters by way of short-lived, kubeconfig files and certificates via single sign-on.
• The audit log shows which user did what, providing info that is very helpful for teams.
• The 5 steps to securing a default Kubernetes cluster involve: etcd security or secrets management; network policies; pod-to-pod communication; secure access; and pod-level security.

Expanding your knowledge on 5 Key Steps to Securing a Default Kubernetes Cluster

• Teleport Passwordless
• Teleport Machine ID
• Teleport Server Access
• Teleport Application Access
• Teleport Kubernetes Access
• Teleport Database Access
• Teleport Desktop Access

Learn more about 5 Key Steps to Securing a Default Kubernetes Cluster

• Teleport Labs
• Contribute on GitHub
• Join our Slack community
• Participate in our discussions
• Why Teleport
• Get started with Teleport

Transcript - 5 Key Steps to Securing a Default Kubernetes Cluster

(The transcript of the session)

Travis: 00:00:00.679 Hello, everyone, and thank you for joining me for today's webinar. And thank you CNCF for hosting it. My name is Travis Rodgers. I am a developer relations engineer over at Teleport, where we provide identity native infrastructure access for engineers and machines. Now, we're going to see a demo of how Teleport works with Kubernetes later, but that's not the main reason we're here today. The main reason we're here today is to discuss Kubernetes clusters and how to secure them. Now, if at any point during the presentation you have questions or at any point afterwards, my email is right here on the slide at [email protected]. I'd love to hear from you. Feel free to email me. But if you want to access the broader Teleport community, then check out our community Slack. You can get to that at goteleport.com/slack. We are an open-source project. So the link to our GitHub is right here as well. And if you want to try it out, we have a labs page where we actually spin up the servers and teach you how to use it. That's at goteleport.com/labs. And finally, you can find us on YouTube where we do tutorials, webinars, videos like this. So, with all of that out of the way, let's get started with the webinar. Okay, so here's a diagram of how I think people learn Kubernetes. So they're at the starting point, maybe their job is asking them to learn these skills, to manage their clusters for them. Maybe they're learning it on the side. It's like a hobby or an interest. But usually, you start out here, and you think, "Hey, I got this checklist. I got to check off to learn Kubernetes." But what really happens is you get pushed into this scribble fast. There's just so many things to learn with Kubernetes, so many moving parts. It's a pretty complex machine.

Kubernetes isn't safe by default

Travis: 00:01:48.523 So you get in there, and you realize, "Hey, wait, I don't even know containerization. I got to learn that first." And then you get in there, and there's so many little parts to it. It's just pretty complicated to learn Kubernetes, exhaustively. So normally, you get in here, you scribble around a little bit, you come out, and you understand Kubernetes. Great. At that point, you're able to get your cluster up and running. You're able to deploy a cluster. You're able to deploy an application without errors, because we all know the first time we try to do that, we get lots of errors, crash loop back offs, things like that. So you can get the cluster up and running. You can deploy your application without errors. It's running out there in Kubernetes. And you can navigate the cluster. You can use kubectl. You know about services and stateful sets and pods and all of that good stuff. But there's one main key that's left out here. And that is the fact that Kubernetes isn't safe by default. You've deployed your cluster, you have applications running in it, you can manage the cluster, but you didn't do anything to the cluster's configuration. And that configuration that was deployed untouched isn't safe by default. It has to be configured to better increase its security posture to make your cluster more secure. Now, let me show you something that reinforces that. So this is the 2022 State of Kubernetes security report by Red Hat. So Red Hat polled 300 DevOps engineers and security experts. And here was the result, basically. Respondents worry the most about exposures due to misconfigurations in their container in Kubernetes environments. So almost half of all of these people polled worried about exposures due to misconfigurations in their container and Kubernetes environments. This was nearly three times the level of concern over attacks, which was 16%, and with vulnerabilities as the second leading cause of worry at 28%.

Securing etcd / secrets

Encrypt etcd data at rest

Travis: 00:03:44.847 And so in this video, I decided I would try to give you some tips on tightening up the security posture of your default Kubernetes cluster. Now, if you're using managed Kubernetes like AKS or EKS, you get a lot of these security benefits. But if not, you just deploy your cluster. You're going to have to do things to make it more secure. And that's what this video is about. So we're going to look at five ways that you can secure your cluster. So let's secure Kubernetes. Okay. So the first security measure we're going to talk about in regards to your Kubernetes cluster is etcd Security and Secret Management. So you probably already know etcd is a highly available key-value store for cluster data, including your secrets. So securing etcd is critical to a cluster security. If someone obtains read or write access to etcd, it's basically giving them root access to the cluster. So you're going to have to do your due diligence to secure this vital piece. Now, how do you do that? Well, we're going to give three ways today. First, you need to encrypt your etcd data at rest. Your default Kubernetes cluster does not do this. Your etcd data in your secrets are in plain text. So if an attacker were to get your etcd, they can just read your data as plain text. So the first thing you can do is encrypt that etcd data at rest. And I'm going to show you hands-on how to do that today. So let's go to the Kubernetes documentation. There's a page called Encrypting Secret Data at Rest. And basically, there's four steps. First, you need to create an encryption configuration object. So here's the encryption configuration object. You specify your resources. So here we have Secrets and Config Maps. And then you specify your providers or your encryption type. So today, we're going to be using AES-CBC, as seen down here in the tutorial.

Travis: 00:05:37.238 So that's the first step. Create this encryption configuration object. After doing that, you need to add three things to your API server manifest file. So let's walk through this. Let's create this encryption config file first. So I'm going to take this information and copy it. Now, I'm using Minikube today as my Kubernetes cluster. You can use whatever variant you want. But what we want to do is we want to ssh into Minikube. So I'm going to do Minikube ssh, because we want to take this encryption configuration object created as a file on the control plane node. So I'm going to ssh into my Minikube server, and then I'm going to cd into etc/kubernetes, and I'm just going to create a folder. So sudo mkdir. I'm going to create a folder called ENC. This is going to be my encryption folder. And I'm going to cd end of that. Now, I'm going to do a sudo vim, and I'm going to create a new file called EncryptionConfiguration.yaml. And inside of that file, I'm going to paste the contents we just copied from the Kubernetes documentation. So I'm going to go up here first. I'm going to erase this other resource. And I'm actually going to erase configmaps too because that's not really pertinent to what we're doing today. So we have resources secrets, and we have this provider, AESCBC. We have a key named key1 and a secret. We don't have a secret yet. We need to generate a secret. So to do that, on the documentation, there's actually a command we can run. So let's go back there, run this command. This will generate us just a random string of numbers and letters. So I'm going to open a new window. And just paste that in. And copy the string of characters. And that's going to be my key. This key is what's going to encrypt my secrets. I'm going to come here and just paste that. And I'm going to save it in that location.

Travis: 00:07:36.588 So right here in this ENC folder, I have an encryption configuration file. That much is done. So the next step is you want to add this encryption provider config flag on your Kube API server manifest file. So you want to add this as a flag and have it point to that encryption configuration object you just created. And then we need to add that folder as a volume and then mount it. So let's do that. So I'm going to go back a folder, go to Manifests, and within that, there is a Manifest file called Kube API server. So let's do sudo vim, Kube API server, open that up. So the first thing we want to do is we want to add that encryption provider config flag. So let's do this in alphabetical order. I'm just going to put it right below this. Encryption. And what was that, encryption-provider-config? So encryption-provider-config= and in the path to our file. So etc/kubernetes/enc/EncryptionConfiguration.yaml. So it points to our file. That's the first thing. Second, you want to go down — and I'm going to do a page down, down to the end and add a new volume. So let's do hostPath. And our path is going to be etc/kubernetes.enc. That's where our encryption configuration file is — in that folder. And we'll do type: DirectoryOrCreate. And then as far as the name, we'll just put enc.

Travis: 00:09:32.171 Now, we need to scroll up to the volume mounts section so we can mount this. So let's go right below this and do mountPath. And it's going to be the same location, etc/kubernetes/enc. name is going to be enc, and readOnly is true. And that's it. According to the documentation, those are the three things we need to do. And so let's go ahead and save that. And when we do that, it's going to recreate that API server pod. So I think when we exit, we're not going to be able to do anything yet. l get pods. Yeah, we can't do anything yet. It might take a minute to get this back up and running. But when this comes back up, any secrets that you create from this point on will be encrypted and will go and check that to make sure that it's working. Let's try it, again. k get pods. Okay. Our API server is back up and running. So what's next in the documentation? How do we verify this? So verifying that data is encrypted, we first want to create a new secret. So we're going to create a new secret, and then we're going to read that secret out of etcd using etcdctl and see that it's encrypted. So let's create a new secret. So I'm going to run this command, and it's going to create a secret called secret1. Secret created. And then, like I said, we're going to read that secret out of etcd. Now, they give a command here that you're supposed to run on the etcd server. I'm going to add a kube exec to it so that I exec into that etcd server and spit out that secret, and then I'm going to do the hexdump. So it's the same as this. I'm just adding a couple commands to it. And then I'm separating the hexdump part of the command. So it's going to go like this. So I'm going to paste in this command, and it's going to spit it out as a file called secret.

Problems faced in encrypting etcd data at rest

Travis: 00:11:30.601 And then I can do a hexdump. So cat secret | hexdump -c. Hit Return, and you'll see that we have the AES-CBC encryption here. So when it says key 1, it's going to be encrypted. So as long as you have the AES-CBC here showing, you know that your data, your secrets, are encrypted. And this is going to encrypt all of the secrets going forward from this point. Now, if you want to go back and encrypt all the previous secrets that you had before this, just follow the documentation down where it says decrypting all data, you're going to move this identity value, and you're going to run this command. And that encrypts all of your old secrets. So that's encrypting your etcd data at rest. That's the first thing you can do. But there are a couple of problems with this, the way we did it. So the first problem is going to be that the encryption key on the server is in plain text. So if somebody gets on your control plane node, they can just check out that encryption configuration file, and there's the key, and that key unlocks everything. It's in plain text. The second problem is that you face manual key rotation, which can be pretty cumbersome. And third, this encryption type is actually not recommended. The one that they show on their website, the one that we just followed. So if you go here and go back up to this provider chart, you'll see that there's a number of providers. So we use this AES-CBC provider. The strength for that is weak. It's not recommended due to CBC's vulnerability to padding oracle attacks. So what else do we use? Well, we could use Secret Box, which is strong, but it's a newer standard and may not be considered acceptable in environments that require high levels of review. There's an AES-GCM that is not recommended for use, except when an automated key rotation scheme is implemented, and then there's identity, which is no encryption.

Travis: 00:13:26.057 So what do they recommend? They recommend the KMS provider here, the recommended choice for using a third-party tool for key management. And the key to this KMS provider or key management service is that you can manage your encryption key outside of your cluster. So think AWS KMS, Google Cloud KMS, or HashiCorp vault. There's some great documentation for these providers out there. And that is the recommended way to go. That is the strongest and has the most benefits. Now, let's move on to number two. So the number two way of securing your etcd or secrets is to restrict access to your etcd. And that involves isolating your etcd server. So the only thing that should talk to etcd is the API server. What you can do is stick a firewall between the two, just to ensure that only API server traffic can get to etcd and nothing else. And then finally, the third step is to ensure that the API server is using TLS. All right. So the second way you can better secure a default Kubernetes cluster is by using network policies. By default, Kubernetes allows all pod-to-pod traffic inbound and outbound. So see here in this diagram, we have a namespace, all pods can talk to other pods freely. But if an attacker were to gain access to one of those pods, they would be able to freely move to other pods. So this is a problem. We don't want this traffic to be wide open, and it is by default in Kubernetes. So what's the solution to this? Well, the solution is network policies. Network policies restrict unnecessary namespace to pod-to-pod access on the network or transport OSI layers.

Travis: 00:15:16.853 So a better way to handle this with network policies is to first set an all-out deny policy. So instead of allowing all pod-to-pod traffic, allow no pod-to-pod traffic in a namespace. And what's neat about network policies is they are additive. So once you deny all this traffic, you can then add on traffic rules as needed. So set an all-out deny policy and then add allows as needed. Here's an example. Let's say you have a front-end application in React, you have a backend application or pod in .NET, and you have a MySQL database in another pod. By default, these three pods can all talk to each other in a namespace. And we're assuming here that they're all in the same namespace. But ideally, the front end shouldn't be able to talk to the database, and the database shouldn't be able to talk to the front end. But the front end should only be able to talk to the backend, and then the back end to only talk to the database. So how would you apply network policies in this scenario? Well, first, we would set, again, this all-out deny policy. So set this network policy to this namespace, and none of these pods can talk to each other. And from there, we can add allows as needed. So if we add another network policy to allow traffic from the back end of the database, we get this. So this network policy is applied to pods with this label. It's Ingress, and it matches the labels with backend at port 3306. So this network policy allows backend to talk to the database. It allows Ingress into the database. We would also want to create one from the front end to the back end. And by doing so, we restrict this wide-open access between pods to allow only the access that's needed. So this is network policies.

Pod-to-pod communication

Travis: 00:17:09.850 And if you need this kind of protection, not at the network or transport layers but at the application layer, then you can look into a service mesh solution like Istio, which allows networking policies at the application layer. All right, number three on our list and similar to the previous is pod-to-pod communication. By default, Kubernetes allows pod-to-pod traffic unencrypted. So previously we saw that Kubernetes allows free-flowing networking. Well, here we see that that traffic is unencrypted by default. And we definitely want to encrypt our data. So right here in the diagram without TLS, we go pod to pod in plain text for any attacker to pick up. With TLS, we go pod to pod with encrypted traffic. So what's the solution to this? Well, the easiest solution is to introduce either Istio or Linkerd to your cluster because both enforce mutual TLS between meshed pods. And that's enabled on both by default. So by introducing either one of these solutions, you get mutual TLS within meshed pods. In the case of Istio, Istio deploys an envoy proxy to each application container that will enforce TLS encryption within the service mesh. So here's the Istio mesh. Within that, within each application container, there's an envoy proxy that enforces TLS encryption. And if you go with Istio, make sure you set the MTLS mode to strict and not permissive. Strict enforces TLS only, permissive allows TLS and plain text. Number four, let's look at Secure Access. So any user that presents a valid certificate is considered authenticated.

Secure access

Travis: 00:18:54.741 With that in mind, we want to disable anonymous auth. And we do that in the kube API server manifest, like the one we visited earlier. We do that by adding this anonymous auth equals false flag. That will disable anonymous auth. Now, if anonymous auth must be used, like in your circumstances or with your company, you need anonymous auth for some reason, RBAC should greatly limit with these users can do. So use RBAC to greatly limit anonymous users as a whole. Number two, you want to disable the insecure port. So the API server actually has a secure and an insecure port. You want to go in and disable that insecure port. Why? Because this insecure port bypasses authorization and authentication checks. And to do that, again, you go to the kube API server manifest, and you set a flag of insecure port equals zero. Number three, you want a well-maintained RBAC. This is not something you said and forget. This is something that is well defined and well maintained regularly. So that's a well-maintained RBAC. And then finally, you want to introduce some kind of corporate solution management. So admin should consider managing normal user accounts with corporate solutions like Active Directory, Okta, etc, via OIDC. Now, an open-source application like Teleport provides secure access to Kubernetes clusters by way of short-lived, kubeconfig files and certificates via single sign-on. And I want to demo that for you today. I want to show you how secure and easy it is to access all of your Kubernetes clusters via the open-source version of Teleport. So I'm going to open up my Teleport cluster UI.

Teleport Kubernetes Access demo

Travis: 00:20:44.449 And I'm actually going to log in passwordless. So I can choose passwordless, and then I'm going to use a passkey that's stored on my phone. If you're not familiar with passkeys, they're a new and secure replacement for passwords. Teleport supports it, and I think more and more applications will as the year goes on. So I'm going to scan this with my phone and sign in with my passkey. Okay. So this is the UI of my Teleport cluster. I'm signed in as Teleport admin, that's a user. If I go to Kubernetes, I see that I have no Kubernetes clusters to access via Teleport. So we need to add one. What's neat about this is that if you go to team, you can create users, and you can assign them roles here in Teleport. So you can enroll all of your Kubernetes clusters here that you can access, and you can use RBAC to control who has access to what, and we'll see that in just a minute. So to add a Kubernetes cluster, just go to Kubernetes, add Kubernetes, next, and there's this wizard that walks you through it. So I'm going to copy this. This will add the Teleport agent chart to your charts repository. Let me go to my terminal and put this in. Again, I'm running a minikube cluster. All right. So my helm repo is added. And step two, generate a command to automatically configure and install the Teleport agent namespace. So Teleport service namespace, I'm just going to put Teleport. That's going to be my namespace. The name of my cluster is going to be minikube cluster. And click on generate command. What this does is it generates a command for you to enroll your Kubernetes cluster with Teleport. So the first thing it does, it creates this prod cluster values file with some information. And then there's a helm install based on the information of that file.

Travis: 00:22:37.619 So let's do this separately. Let's capture this first part and create that. All right. So now have the prod cluster values file. And then we do this helm install. Now, I'm going to add a label to my cluster. So I'm going to open up VSCode, go to new file, text files, find, paste this in. And then I just want to add, to this command, a set labels, env equals dev. So I want this cluster to have a label with env equals dev. It is an environment, dev cluster, not a prod cluster. That's all I'm doing here. So copy this command, go to my terminal and paste that in. Hit Return. And our Teleport agent has been deployed. Let me clear the terminal here. And while the container is spinning up, this wizard takes you through another couple of steps of setting up access and testing the connection. I'm not going to do this because I don't have any roles configured yet. So I'm just going to exit out of this once this is done. So I'm not really concerned about this timer that detects the Teleport service. All right. If I check my pods now, okay, get pods, A, you'll see that my Teleport agent is running. So if I go back out to Teleport and click on Kubernetes, I should have my cluster listed. That's how you enroll a Kubernetes cluster to Teleport. So from here, I can connect to the cluster. And you do so by clicking Connect and following these instructions. So number one, log in to Teleport. So Teleport comes with a couple of CLIs. There's the tctl cli — tctl, which is the administrative CLI. And then there's this tsh cli, which we can use to log in and log out and things like that.

Logging into Teleport

Travis: 00:24:26.212 So the first thing we need to do is log into Teleport via the command line, because that's where we're going to be using kubectl. So I'm going to paste that command in here. I'm actually going to erase this last part. And here I have the proxy flag for my proxy, which is teleport-tr.asteroid.earth. I have my auth and my user of Teleport-admin. So this will log me in. Enter my password and then tap any security key. I'm using my YubiKey. So tap that for second factor. And there we go. I'm now logged in to Teleport, via my Teleport user, Teleport-admin for 12 hours. And this is configurable, but valid for 12 hours. I have a 12-hour certificate. So I've logged in, what do I do next? Well, next is an optional command you can run to set a different kubectl configuration. We're going to use the global. So we're going to skip to step two, which is select the Kubernetes cluster. Now, let me show you this. If I do a kubectl config view, you'll see that my current context is minikube. We don't want that. We don't want to access minikube directly because that's on my local computer. We want to access this Kubernetes cluster through the secure functionality of Teleport. So that's what this command is going to do. This is going to issue us a short-lived certificate. So tsh kubelogin minikube-cluster. So let's run this command. Let me clear this. Run this command. Logged into Kubernetes cluster, minikube cluster. Try “kubectl version” to test the connection. Let's try that out. kubectl version. All right. Everything's working. Now if we do kubectl config view, you'll see that my current context is now teleport-tr.asteroid.earth-minikube-cluster. So it looks like we're using Teleport.

Connecting to the Kubernetes cluster

Travis: 00:26:23.025 And the last command is to connect to the Kubernetes cluster. So we just need to run kubectl get pods. So clear this again, kubectl get pods. What do you think is going to happen? Well, this user cannot access Kubernetes. Unable to list resource pods. This user cannot request Kubernetes access, has no assigned groups or users. Why is that? Because our Teleport admin doesn't have any roles. It has no privileges to access this Kubernetes cluster. Even though this Kubernetes cluster is enrolled with Teleport, we still have RBAC in place to control who has access to what. So let's do this. Let's go back to team and roles. Let's create a role for Kubernetes admins. So create new role. I'm going to call it here, kubernetes-admins. And this is a default template. So I can erase everything below this because I don't want any other privileges with this role. And I'm actually going to erase this Kubernetes users because I don't need that at the moment. And under Kubernetes groups, I'm going to put system:masters. And this is never recommended. This gives you admin access to the cluster based on a system:masters default role that Kubernetes has, but just for demonstration and this Kubernetes admins group, let's do that. Click Save Changes. And then let's go to users and my Teleport admin user. Let's edit and add that role to that user. So Kubernetes admins and save. Now our Teleport admin user has a role for Kubernetes admin access. So let's go back to our terminal. Let's do tsh logout because we need to get a new certificate. That role is in the certificate. Let's do tsh login again for Teleport-admin and log in.

Creating roles and managing access

Travis: 00:28:17.394 All right. Great. And you'll see now that Kubernetes cluster says minikube-cluster, and our Kubernetes groups for this user is system:masters. So at any point, you can do tsh status to see your status. And now that we have that system:masters group, we should be able to do k get pods, and do it successfully because we now have the privileges to do so. There we go. We can do k get services and whatever else. So that is the role we're going to give to Teleport admin. But what if we have other users come on the scene? Let's imagine that we have this role, we have these users come in that you just want to give them read access. They should only be able to read the pods of a Kubernetes cluster. So let's create a new role. Let's call it developer read. And again, let's erase all this stuff. We don't need it. And for Kubernetes groups — I'm going to erase the users also. For Kubernetes groups, let's imagine there's a group called developer-read. This is a developer read-only group. So let's save this. And what we're going to do is we're going to create some roles in Kubernetes. So this is not Teleport. This is Kubernetes related. And I have some examples that we're going to use. So we're going to create a role in Kubernetes called dev-read. Resources will be pods and pods/log, and the verbs will be get, list, watch. So a read-only role. Then we're going to create a role binding called dev-read binding that binds this dev-read role to a group called developer-read. So let's go ahead and do that.

Travis: 00:30:00.414 And actually I have both of these roles already on my system. So let me just apply that. So kubectl create -f dev-read role, I think it's called dev-read role. Let's create that role. All right. And then we create the role-binding. And again, this is not Teleport. This is just Kubernetes. This is creating a group in Kubernetes called dev-read that only has read access to pods. So now, in Teleport, this role that we created, edit, gives access to a Kubernetes group called developer-read, which is the group that we just created in Kubernetes. So what I can do now is I can create a new user. Let's go to users, create new user. Let's call him Bob, and let's select a role. Let's select the developer-read role. So Bob is a new developer on our team, and all he needs in Kubernetes is to read pod information. So let's save that. And here's an invite link that I would send to Bob. So I'm going to send this link to Bob. He's going to click on it and set himself up. So get started, Bob, let's create a password for Bob. And click Next. And we're going to use a hardware key. Try another way, in a USB security key, which is my YubiKey. And registration successful. Great. So here's Bob. Bob has no access to servers, applications. He does have access to this Minikube cluster because we gave him access.

Travis: 00:31:44.231 So let's go and log in as Bob. Let's do a tsh logout and do a tsh login as not Teleport admin but as Bob, and let's see what Bob can do. So Bob's logging in. Bob should only be able to read from Kubernetes. So k get pods. Let's see how it goes. No resources found in default namespace. Great. What about k get services? Bob can't do that because he is in the read-only group. Forbidden. User Bob cannot list resource services in the namespace default. So this goes to show you that you can have lots of developers, whoever's on your team, and you can give them different access to this Kubernetes cluster securely through short-lived, kubeconfig files. So Teleport admin has admin access to the cluster. Bob only has access to that developer read group. Now, one more thing I want to show you, let me log out as Bob and log back in as Teleport-admin. Other sign-in options, passwordless, so try another way. Again, I'm going to do my passkey. And now if we go back to team and roles, if you look at this role, developer-read, go to edit, there's a section called Kubernetes labels that has these two stars. This means that, basically, you can access any cluster, given that you have the right group or the right other access. But this labels gives you access to clusters with any label.

Audit logs

Travis: 00:33:36.833 Well, remember we added here, this env — if we go back to Kubernetes, we added this env: dev label. So let's say that this developer-read, we only wanted them to access prod clusters. We could come in here and do ENV prod, and then I can maybe change the name to developer-read prod, but what this is going to do is this is going to take Bob and not allow him access to that cluster because that's a dev cluster. He only now has access to prod clusters. So there's a lot of ways you can limit access for your users. Now, the final thing I want to look at in this demo is Audit Logs. So if you go to activity and go to Audit Log, we can see everything that everybody did. And more importantly, all the kubectl commands ran. Now, remember, Bob has access to the developer-read group. So everybody using that developer-read group — the logs are not really going to help us a lot. We're going to see that somebody used developer-read, developer-read — different people.

Travis: 00:34:40.573 Well, this audit log is going to show us which user did what — with lots of info. So right here at Kubernetes request, user Bob received a 200 from get pods. So this tells us that Bob ran the kubectl get pods command to this Kubernetes Minikube cluster. Imagine if you had 15 clusters and 50 users, you would be able to see the audit logs for every individual person here. If you need details on it, there it is. More information here in the Details button. We see here that Bob got a 403 when he tried to get services. So that audit log is very, very, very helpful for teams. So here's a diagram of what we just saw. So the users are going to do this tsh login to log into the Teleport cluster. From there, from their machine, they can run kubectl get pods or whatever commands to the Kubernetes cluster. At no point do they access the cluster directly. They only talk to this Teleport proxy. The Kubernetes cluster runs this Teleport agent and has an encrypted tunnel back to the Teleport proxy. And you can even put a firewall there for more protection. So that's the section on secure access and the demo of Teleport.

Pod security

Travis: 00:35:53.801 Let's move on to number five. Fifth and final on the list is pod security. So here's a bit of a backstory. A few years back, I got put on a team, and none of us really knew a lot about Kubernetes or containerization. And we were asked to containerize a lot of apps, create helm charts for those apps, and deploy them into Kubernetes. So going back to that first diagram, we started that journey of learning Kubernetes and containerization. And we thought we knew a lot. So we got these applications containerized, we figured out helm charts, we created helm charts, and we deployed them to Kubernetes. But we were application developers. Once that application was running, high-fived, checked it off the list, we're good to go. We didn't think much about pod security because we didn't know a lot about it. But pod security is very important. If you're pulling images, docker images, or helm repos from third-party websites, they possibly could introduce problems like root access or some misconfigurations that weaken your cluster security. And the reason I gave that example is because there are a lot of talented application developers that containerize applications, they throw them into Kubernetes, and they're good to go. They don't know a lot about maintaining Kubernetes and Kubernetes security. They're not really supposed to. They're application developers. But you need somebody on your team to overlook that security. So when it comes to YAML configurations that are introduced to the environment or containers that come into the environment, they need to be audited. Now you can do this manually by having someone or a team do that, or you can look into vulnerability scanners as listed here. You got Kubestriker, kube-score, KubeLinter. Vulnerability scanners that will look for potential holes in your YAML and your containers and try to find that problem before they come into the ecosystem.

Travis: 00:37:42.124 You can also utilize security contexts. As you can see here, there's a security context on this pod with runAsNonRoot as true. And what this does, if you have this setting, the Kubelet will refuse to start any image that defaults to the root user. So hopefully, when you containerize your application, you set a user ID and a group ID, but sometimes, if you're not careful, those containers will come over with root access. And by setting the security context setting, it will refuse to start the image that defaults to the root user. That's another protection. And there are more security contexts, things you can look into. You can see it here on the API page. So again, make sure your YAML in your containers are audited when they come into the ecosystem, either manually or with vulnerability scanners, and then use tools like the security context to make sure none of your pods are running as root. And with number five being done, that leads us to our conclusion. So again, five things to check on your default cluster is number one, your etcd security or secrets management; number two, network policies, use network policies; number three, pod-to-pod communication, make sure that's being encrypted; number four, secure access; and number five, pod-level security. And with that, thanks a lot for joining me in this webinar today. I hope you learned a lot.