Open Security Summit 2021: Using Teleport to Secure SSH and Kubernetes Access - Overview
Teleport is a Certificate Authority and an Access Platform for your infrastructure. With Teleport you can:
- Set up Single Sign-On and have one place to access your SSH servers, Kubernetes, Databases, and Web Apps.
- Use your favorite programming language to define access policies to your infrastructure.
- Share and record interactive sessions across all environments.
Key topics on Open Security Summit 2021: Using Teleport to Secure SSH and Kubernetes Access
- Teleport is an Access Platform with features that can be applied to both SSH and Kubernetes access.
- Teleport features include RBAC, access requests, and session and identity locking.
- Demo 1 of this talk covers enforcing RBAC SSH using Teleport.
- Demo 2 of this talk covers enforcing just-in-time privilege to Kubernetes access.
- Demo 3 of this talk covers how you can enforce Teleport's session and identity locking feature to lock out malicious users.
Expanding your knowledge on Using Teleport to Secure SSH and Kubernetes Access
Learn More About Using Teleport to Secure SSH and Kubernetes Access
Introduction - Using Teleport to Secure SSH and Kubernetes Access
(The transcript of the session)
Dinis: 00:00:03.434 Hi, welcome to this Open Security Summit Session in November 2021, where we're going to be looking at how to use an open source tool and the technology that these guys have developed to basically do something that anybody who has dealt with Kubernetes knows can be quite painful — which is how to connect to it, how to access, how to get data, how to make it so that you have a nice and secure way and scalable way to manage your identities and to connect to your resources. So over to you.
Introduction
Sakshyam: 00:00:35.907 Okay, hello everyone. In today's session, I'm going to talking about using Teleport to secure SSH and Kubernetes access. So I prefer this talk to give you a brief introduction about Teleport, the access features it supports that can be applied to both SSH and Kubernetes access. And then I'm going to show you a short demo on how some of these access control features can be used on keys that would help to manage secure access to infrastructure. So who am I? My name is Sakshyam Shah. I'm the Developer Relation Engineers at Teleport. Before Teleport, I have eight years of experience exclusively in cybersecurity. Worked on both offensive and defensive side doing pen test, Red Teaming security audits, and also building bespoke cybersecurity practice for enterprises. Besides cybersecurity, I love talking about new technologies and startups. And although I'm not that active in social accounts, you can ping me in either GitHub, Twitter, or LinkedIn. Here are my handles to all of these mediums.
Teleport overview
Sakshyam: 00:02:01.364 So first, what is Teleport? For those of you who don't know what Teleport is — Teleport is an open source project. We call it Unified Access Platform. It is a single solution to allow engineers and security professionals to unify secure access to SSH servers, Kubernetes clusters, web application, databases, and desktop environments across all assets in your infrastructure. So what brings Teleport to Access Management? So the base core part of Teleport is it's based on certificate based authentication and access control. So it entirely eliminates security risks that are related to using password-based access control. So no credential leakage, no password leakage, no user insecure passwords. And also, certificate means it also eliminates risks of using private keys to manage and secure access to infrastructure. So all the private keys are way better than passwords. But still, private keys hold risks of managing access to use it in situations such as: what happens when authorized user leaves your company? So what happens when authorized keys are leaked in data breach, and you want to stop anybody to misuse that keys across your servers? So it's somewhat easy in a situation where you have two to three servers. But in a situation where you have in 10s or 20s or even 100s of servers, then it's really hard to rotate private keys around your servers. So scope certificates help you eliminate all of those risks and allows you to manage secure access based on user identity. That can be also attached to identity providers, such as Okta or Gator or any providers that support OIDC, XML, and such.
RBAC
Sakshyam: 00:04:13.609 Next, Teleport supports role-based access control. So having this feature, Teleport allows you to manage access, depending on what role the user requires and possesses in your infrastructure and in your company. For example, a developer wants and needs to access resources that the developers [inaudible] request to access and other — and for example, a junior developer might not need a root access to a production server. So role-based access control helps you define such boundaries within your infrastructure such that only the users that are required to access servers or services are actually authorized to access those services. Next, Teleport supports complete session recording and replay. So whenever any security incident happens or during security audits, it helps to replay what the authorized users actually did in the authorization. So one of the challenges in cybersecurity is insider threat, or even any employees misusing their credential to act maliciously inside the infrastructure. So session recording and replay helps you resolve a situation where any users or employees go rogue or in any security audit events.
Access request
Sakshyam: 00:05:57.018 Next, Teleport also supports Access Request. So access request helps you implement these privileges and Just-in-Time access. So in many situations and production infrastructure, users are not required to have a continuous 24-hour access to those servers. And Access Request helps you implement control that would allow any users if required to access their infrastructure, they would need to request — give a context to what are the reasons they need to access server or services. And administrators can view that context and either approve or deny access to the infrastructure. So Teleport integrates with any ChatOps platform such as Slack, PagerDuty, or you can also use Teleport API to develop your own custom plugin to implement Access Request.
Sakshyam: 00:06:59.791 Next, apart from the certificate-based authentication role-based access control, session recording, and replays, Teleport also supports dual authorization that is related to access request that we presented earlier. Teleport also allows you to define role templates that can be used to assign [inaudible] policies dynamically to users who would access your infrastructure. Teleport allows you to impersonate another user such that you can use these features to create short-lived certificate that can be applied to CI/CD, so that you don't need to provision actual credential for users in cases that will be deployed in public infrastructure such as GitHub actions, GitLab CI. And by default, Teleport also supports second factor authentication. So you can enforce another layer of access control on top of existing certificate-based authentication. Teleport also supports privileged processing Multi-Factor Authentication for mission-critical servers. So users enforcing these features will require users to perform second-factor authentication every time they initiate a new session.
Session and identity locking
Sakshyam: 00:08:35.521 And lastly, Teleport supports Session and Identity Locking. So this features is one of the useful feature in cases such that what happens if you identify a user is acting maliciously in already authenticated session or you find that a user credential has been misused to access your infrastructure and you want to block access to that server. So rather than you blocking access to entire fleet of the servers, you can just implement this unlocking and block access to particular server or servers that the malicious user is using, so that you can avoid downtime to infrastructure, and yet you can successfully block the malicious access. So these features collectively help you implement secure access to your infrastructure via SSH or Kubernetes.
How Teleport works
Sakshyam: 00:09:36.913 So how does Teleport work behind the scenes? So at core, Teleport is a bastion server. So users that will be connecting to your infrastructure from either public network or even from inside corporate network, so they will use Teleport Web UI or Teleport SSH client such as tsh or Kubectl. And they will authenticate with Teleport authentication server which can be integrated with your identity providers. And upon successful authentication, Teleport will issue scope certificate, which can be used to access SSH servers, Windows servers, Kubernetes clusters, databases, or internal applications such as Grafana, Jenkins, GitLab. And depending on the access control features that are implemented, users may require to request for specific access request that will be notified to Administrator via ChatOps platforms such as Slack, Mattermost, etc. And depending on the context, administrator can approve or deny access. And similarly, users will be allowed or disallowed to access infrastructure. So this is collectively how users will access infrastructure when Teleport is provisioned within their infrastructure. So I'll be presenting demos. Three use cases at how users — where you'll be enforced and will be required to go to Teleport, or will be controlled via Teleport features. So first demo I'm going to give would cover Teleport RBAC. So this demo will cover enforcing RBAC SSH using Teleport. So in this scenario, what we have here is a Linux server. And we have a developer who accesses this Linux server frequently. This Linux server is configured with both root and non-root users. And the developer accesses this server frequently as both root and non-root users. But this developer doesn't require to have access to root account. So what we'll do is require this developer to access Linux server via Teleport where we can control such that root access will be blocked, and every non-root access will be allowed. So let me just switch back to my desktop. So here I have admin interface. [silence]
Demo 1
Sakshyam: 00:12:51.225 So this is a web UI Teleport and at this workspace. So I have a — so I will be demoing — in this workspace, I'll be demoing as a developer-user. And in this workspace, I will be using admin interfaces to configure Teleport. So just to give you a quick demo on — this is developer account, and the developer user can access Linux server both as root and non-root users, so just to give demo on accessing server as a root user, so the developer can easily access server as a root user. And developer can also access the Linux server as a non-root user. So what I will do is create a developer account in Teleport and configure the account such that user can access only as a non-root user. So first let me create a developer account. tctl users add developer --roles-access.
Sakshyam: 00:14:37.957 This access uses a predefined role in Teleport that would allow users to access resources. If this role is not assigned to a user, even they have a Teleport account, they will not be able to access the servers. And then we are also required to provide a login name which can use to access the server. So here we are providing non root as username, as a login name, that this user will use to access the Linux server. So if we would like to allow this user also access as a root user, so in that case, we would be also allowing root login name. So here I've created users using the tctl tool. It's a command line tool available with Teleport that helps you issue command that commands to administer Teleport, so I'm going to copy this link and send the developer a message that would require him to continue the setup process. “Hi developer, here's a link to signup for Teleport.”
Sakshyam: 00:16:23.987 So as an administrator, I've created a developer account and send a message to developer that he will be required to use signup to Teleport and use Teleport to access Linux server. So in the meantime, I will also configure Teleport node that will add Linux server as the SSH node to Teleport so that Teleport can proxy SSH request to Linux servers. So next is use tctl to open — type, node. So here's the invite token that will be used to connect with Teleport cluster. So I've already initiate SSH access to Linux server as an administrator. So in this window, I'll start Teleport. Teleport is already installed in the next server as well. So next, what I'll do is connect this server to Teleport cluster. So here is the predefined command I had already entered. I just need to replace the token.
Sakshyam: 00:18:05.887 So the command is teleport start with the role node that would allow this Linux server to connect to Teleport cluster. The web URL of the demo Teleport cluster I have posted is primary.teleporters.dev, and it will require token to join the Teleport cluster. So after issuing this command — so that this server is already connected to Teleport cluster. We can also verify this, so this shows and it's visible. Okay, so this shows that this Linux server is already connected to a Teleport cluster. Next, what we'll do is start the OpenSSH server entirely so that the developer cannot connect to this server via OpenSSH anymore. So systemctl, stop sshd in servers. So you can verify status sshd.service. So as a developer, I just tried to connect to the Linux server again, and you can see the connection is refused. So to access this server — now what I need is, as a developer, sign up to Teleport using this link shared by administrator and complete user registration process. So first I will be scanning this QR code to enroll two-factor authentication in my authenticator app. Next I'll just give it a password and login. So you can see my account as a developer-user has been created with Teleport. Now, next to access this server, I'll have to issue tsh login that will allow me to access via Teleport SSH client which is tsh. tsh login. I have to give it a proxy address or proxy server which is primary.dev and login as a user, developer. So it will ask me for my password and I need to supply my TOTP code. Now I'm authenticated. And so this is my Teleport account profile, and I have been granted a role of access. And I can access Linux server or any servers only as a non-root user. So I'll show you how I can access the server from Teleport UI itself.
[silence]
Sakshyam: 00:22:40.759 So you can see I can access Linux — as a developer-user, I can access Linux server as a non-root user. I can issue commands, even change directories, or perform apt-updates. You can see I can issue any command but I have to go to Teleport and not access directly via OpenSSH. So as an administrator, you can view my active session as well and see any audit logs. For example, a session has been started as a developer account. A developer account has local login, successful login which I did as a tsh login command. So as a user, I will exit from this search session. And so here comes the recorded session. Linux server access by developer account. I can just play and see every command issued by this user.
[silence]
Sakshyam: 00:24:56.673 So as you can see, my access to Linux server as a developer account has been completely blocked, if I try to access via open SSH, but I can still access a server by Teleport, but only as a non-root user. So this demo is — so now as demonstration for this part one is completed, it shows that how you can use Teleport access control features and role-based access control features to allow users to only particular accounts and block access to — even though the user can access the server, but is only allowed to access the server by a particular role. So the next demo I'm going to give is a —
Dinis: 00:25:54.702 So I have a couple of questions. You mind if we pass them on that demo?
Sakshyam: 00:25:59.482 Sure, sure.
Dinis: 00:26:01.609 So that environment, right, you had. Was Teleport running in like a central location, or was it running on the Kubernetes cluster or it's not Kubernetes [crosstalk]?
Sakshyam: 00:26:13.280 So Teleport is actually running in a separate Linux server. This is not just Linux server, but a separate Linux server.
Dinis: 00:26:22.686 Okay, this is not Kubernetes deployment. This is just normal SSH deployment.
Sakshyam: 00:26:27.391 Yeah, yep.
Dinis: 00:26:28.677 Okay, got it. So in this case, you had a Linux box in this C2, for example, or Cloud Compute that runs Teleport. And that's the one that when the server boots up, you register, you give it some credentials. So you register that server into the Teleport server, right?
Sakshyam: 00:26:49.242 Yeah, yep. Yeah.
Dinis: 00:26:50.975 Okay. Got it. And then, we use that to manage access. And I'm assuming a lot of these could also happen automatically, right, because if you have machines connecting and disconnecting, part of a CI pipeline, a bunch of other stuff, right? These would also capture that, isn't it?
Sakshyam: 00:27:08.567 Yeah, yeah. So anything, once the server is connected to Teleport cluster, every access going through that server will be recorded in Teleport.
Dinis: 00:27:21.737 So technically, if that user was actually a CI pipeline, you would also capture that, right? You capture, let's say, again, it was a CI pipeline to build a particular server — you will capture every command that was executed, right?
Sakshyam: 00:27:37.432 Yeah, yeah. Great. So the CI pipeline will be assigned a separate scope certificate from Teleport. And using that scope certificate, the CI pipeline will have a credential that would allow to access the server. So the credential will allow scope, roles. And once the CI pipeline credential is authenticated to Teleport, cluster, so everything that is executed by CI pipeline will be recorded to Teleport.
Dinis: 00:28:09.303 That's a really good feature. Right? That's very interesting because especially in this day and age, we tried to create really strong change of [inaudible] and a lot of mappings. And we have a bunch of servers that are built as part of the CI pipeline, right? And I like the fact not only you can manage going forward — you can connect that. So you could deploy servers that have Teleport enabled, even if you don't have a Teleport server, right? You can only add it if needed, right. Or you'll enable —?
Sakshyam: 00:28:44.430 Correct. Correct. Correct. Yeah, totally. Yeah. Yeah. Yeah.
Dinis: 00:28:48.480 Because some of these things like a lot of times, we don't want it access to the server. But if you need to, you need connect. That is good to have that process. Yeah, I think machine to machine access is very important because in some environments, the users don't have access to it. But you do connect to the environments to do a bunch of other things, right. And it's really cool to have the login even for debugging, right, even to debug and to understand what happened to the server. It seems that you found a great way to capture the history of what was done to the server. And I'm assuming that yeah, if we were running Helm Charts, you're capturing all the output, right?
Sakshyam: 00:29:23.149 Yeah, yeah. And if you run it within that Shell session, it will be recorded to Teleport.
Dinis: 00:29:30.360 Yeah. So that's cool. If you run an upgrade with Helm Chart or something like that, then you also see it right?
Sakshyam: 00:29:35.317 Yeah.
Dinis: 00:29:35.317 All right, cool. Great stuff. That was my question. Pretty good.
Demo 2
Sakshyam: 00:29:38.795 Okay, so this the first demo. The second demo is the use case I will be presenting or enforcing just-in-time privilege to Kubernetes access. So in this scenario, I have a Kubernetes cluster that's already been created in provisioned, and then I have an existing Teleport server. And so this Kubernetes cluster is occasionally accessed by a demo contractor, so the contractor [inaudible] requires to access Kubernetes cluster, perform some administrative tasks. So how we have configured Teleport and this contractor user in this scenario is that the contractor-user account have been already created in Teleport server but is provisioned with a role that requires contractor to initiate access request to Administrator stating what is the reason for access. And depending on the context, administrator can allow or disallow access to Kubernetes cluster.
Sakshyam: 00:30:48.764 So if you look at the Teleport UI, so we have the contractor account already created in Teleport with roles “access” and “contractor”. So to also further expand on the roles, how the roles will look like is — so for example, this is a contractor role. So contractor role — it's a YAML file spec that defines the contractor role profile. So if you see here — so this contractor has a spec that allows the contractor to request access to role, that is “kube-member” role; “kube-member” role has the spec that allows any users with this role to access Kubernetes cluster as a “system:masters” group. So this contractor is provisioned in such a way that he has access to a role, that allows to access request to another kube, another Kubernetes role, that is named as a “kube-member” that will allow this contractor to access Kubernetes cluster as a “system:masters” role. So the user is indirectly mapped with a role that would allow this user to access Kubernetes cluster, but to achieve that role, user needs to initiate access request. So next.
Sakshyam: 00:32:38.218 So this workspace here is a contractor workspace. So as the contractor — let me just try to log into Teleport cluster see — teleporters.dev, user, contractor. Need to supply my password and supply my TOTP code. So this is my contractor role profile that is printed here. So Kubernetes access for me, for this role (a contractor role), is enabled. This contractor can access Kubernetes cluster that is already enrolled within Teleport. So to access this Kubernetes cluster as the contractor, tsh kube ls. So you can see which cluster is enrolled in Teleport and tsh login. So just logged into this Teleport cluster via tsh. kubectl get nodes.
Sakshyam: 00:34:36.525 So you can see that executing command kubectl get nodes is giving Error. Error from server, this user cannot request Kubernetes access, has no assigned groups per user. So you can currently see this user's account is provisioned to access Kubernetes cluster, but at this time is not authorized. So to access the server, what we need to do is request your role. kube-member as a contractor account, so that administrator would be able to grant access. Request roles, Kubernetes cluster, user contractor. So here my request is initiated. So for this scenario I've integrated Teleport cluster with Slack. So it's a demonstration of ChatOps pattern that Teleport support. So whenever user initiate access request, so you can see the Teleport, login for Slack gives me a message that a user named contractor is asking for Kubernetes role new member.
Sakshyam: 00:36:18.031 So to view this access request in detail, so just click the link and it opens up Teleport UI. So this is Access Request UI where I can, the administrator, can view details of what the user is asking for a role and depending on the context. And as administrator, I can either approve or deny this access. So in this demo case, I want to authorize this contractor user to access Kubernetes cluster. So I'm just going to approve this time. It's optional message, submit a review. So I've authorized this access request from contractor-user. So as a contractor-user, now you can see my Access Request has been fulfilled.
Sakshyam: 00:37:21.091 And so I've been assigned with the Kubernetes group “system:masters”. So now if I execute this command, I should be able to get nodes that are available in Kubernetes cluster. So this is it. So I was able to get nodes, which otherwise would not have been possible without this Access Request. You can also see here that the Access Request for this role is valid for 11 hours, 56 minutes. So I can access this Kubernetes cluster until my Access Request is expired or explicitly been removed. So just let me execute more commands, kubectl get namespace.
Sakshyam: 00:38:21.958 So we have bunch of default namespaces. So I'll just try and demo accessing a Kubernetes pod within this namespace. So we have a bunch of pods running. So initiate a shell session here to this pod and it exec, /bin/sh. So you can see now as a contractor user, I have full access to Kubernetes cluster given this user is authorized with “system:masters” group. And I've just initiated a shell session to pod inside the Kubernetes cluster. So again, execute commands. test.txt. Permission denied. As you can see, this user has full access to viewing etc password. So I'll just exit this session. And you can see the session has been recorded. So this also concludes the part two demo of this talk that a user was able to access the Kubernetes cluster that otherwise would not have been possible without asking for a Just-in-Time privilege to the administrator.
Dinis: 00:40:42.794 Just a question on that one. And in this case, the Teleport basically serve — or components were actually running inside Kubernetes. Right?
Sakshyam: 00:40:54.251 [crosstalk] —
Dinis: 00:40:54.251 [crosstalk].
Sakshyam: 00:40:56.501 So expanding on that part, so Teleport supports installing Teleport inside a Kubernetes cluster, also allows to connect to Kubernetes cluster outside from — apart from being installed in Kubernetes cluster. So in this case, Teleport is not installed within Kubernetes cluster. So Teleport is installed and managed independently on its cluster. But in this scenario, it is configured in such a way that a service account for Teleport is created inside Kubernetes cluster. So that's how Teleport is able to manage access via Teleport proxy to Kubernetes cluster.
Dinis: 00:41:39.019 Got it. But this will also work completely offline unlike those clusters don't need any access to the internet. Right?
Sakshyam: 00:41:47.147 It just that Teleport clusters should be able to communicate with Kubernetes cluster.
Dinis: 00:41:52.476 Yeah, yeah, yeah. So as long as they can talk — so they can be on a VPC. Right?
Sakshyam: 00:41:56.434 Yeah.
Dinis: 00:41:58.162 That gives them access to that information. Yeah, now, so then again, one of the errors I'm thinking that this is very, very useful actually is not necessarily the human operator, although I think that's important for some developing environments in some, some situations. But the one I really like is — we already run multiple clusters, right, and to sometimes manage the relationships amongst them, and to really lock them down can be a bit hard. And I really like the fact that with this, you can distribute short-lived tokens. You can have a whole workflow, or even one token, even in the admin cluster — concept of the admin cluster — even when he goes and get some data from another coach so you can very effectively limit its part, right, because that's sometimes the hardest thing, right? Another question: in terms of — I guess this is running natively in Kubernetes. Right? So you can run this on any of the cloud providers, right? There's no limitation.
Sakshyam: 00:42:56.267 Yes. Yep. Yep. It's not just specific to any particular Kubernetes distribution. So this works any standard Kubernetes distribution.
Dinis: 00:43:07.569 Very cool. Good stuff. All right. Thanks.
Demo 3
Sakshyam: 00:43:10.276 Yep. So this concludes the demonstration part two. So next is the last demonstration for this talk — about session locking. How you can enforce session locking feature to lock out malicious users. So this is in continuation to demo two. So in demo two, I acted as a contractor-user. I executed some of the commands such as I did a [inaudible] etc password. So that's not what the contractor is supposed to do. And it may seem malicious if that's not the role that contractor is assigned with. So after authorizing contractor, what administrator should do is replay a session, a shell session that was executed by contractor. So let's say this is not what the contractor is supposed to execute inside a Kubernetes cluster. So this user seems malicious can be a German contractor trying to misuse the credential or maybe contractor’s credential has been already compromised.
Sakshyam: 00:44:43.314 So in this case, what Teleport supports is locking access so that this contractor would not be able to connect to Kubernetes cluster even though the contractor-user has a valid certificate and valid credentials to access Kubernetes clusters. So as an administrator, what I'll do is execute a command, tctl lock --user=contractor. What time, let's say, for example, 10 minutes. So executing this command, response has created a lock with name. It's lock ID. So this means that locking of use account has been enforced. So as a contractor, I still have a valid certificate, but as a contractor, if I try to access the same part that I had been able to access previously, it gives Access Denied. So this lock will be enforced still, since the lock was enforced for 10 minutes. So this contractor won't be able to access Kubernetes cluster, execute any other command as well. So Access Denied, so this user won't be able to connect to any Teleport cluster or execute any command to that cluster, or any resources that have been protected with Teleport. So this way, Teleport supports session locking that allows users to — for any given time for a given period. And for this demonstration, I've used command that is blocking the whole user account contractor as a whole user account, but Teleport supports locking access to individual servers or clusters as well.
Sakshyam: 00:47:00.596 So this concludes demonstration part three as well. So these are the three demonstrations for this talk. If you have any question, Dinis?
Dinis: 00:47:17.338 No, this is really good. Well, I definitely want to give it a test drive. Right? You seem to have solved a number of good problems. That exists, right, and when you're trying to manage identities at this level? So it's really cool. And all the stuff you showed, it's all the open source elements of it. Right?
Sakshyam: 00:47:38.653 Right.
Dinis: 00:47:40.169 Yeah. Yeah. So I think we'll be good. He's also actually managed to see if there's any questions or anybody on the chat and we'll just double-check. We're good. Cool. So is that the end of your slides?
Recommended next steps
Sakshyam: 00:47:56.635 So yeah, so next recommended steps for everyone is go and try Teleport at server level at goteleport.com/download. You can read Access Control Docs. More on Access Control Docs that is available in this URL. So how to work with access control for SSH Kubernetes and database and web apps using Teleport, and since Teleport is open source project, you can check out Teleport on GitHub at github.com/gravitational/Teleport. So this is it, concludes my talk for this event.
Dinis: 00:48:37.609 Cool.
Sakshyam: 00:48:37.609 So yeah, so any question I love to answer.
Dinis: 00:48:42.724 No, I think we don't have any questions on the chat. So thank you it was —
Sakshyam: 00:48:50.184 Okay.
Join The Community
