Securing the Supply Chain

Secure World: Securing the Supply Chain by Following the Audit Trail - overview

In this webinar, Donnie Hasseltine will address the challenges of securing the software supply chain and how an SBOM (software bill of materials) is an integral part of knowing what software is being included in a release. We'll review the problem and how an audit log can be integral to knowing what software has been updated and making sure build systems are secure. In the second half, Ben Arent will show how to secure CI/CD systems using short-lived certificates and how to provide secure and audited access to build boxes using the OSS Access Platform.

Key topics on Secure World: Securing the Supply Chain by Following the Audit Trail

• The software supply chain starts with the developers, and involves a number of other steps to get that software out. Every one of those steps is a potential problem.
• An SBOM (software bill of materials) is an integral part of knowing what software is being included in a release.
• Packagecloud is a unified way to manage all of your software artifacts in any language.
• The 4 essential elements of infrastructure access are connectivity, authentication, authorization and audit.
• An audit log can be integral to knowing what software has been updated and making sure build systems are secure.
• Through Teleport, you can gain complete visibility and pipe the audit log available into your SIEM or other tools to know what's happening.
• Using Teleport, CI/CD systems can be secured using short-lived certificates and secure and audited access can be provided to build boxes.

Expanding your knowledge on Secure World: Securing the Supply Chain by Following the Audit Trail

• How Teleport Works
• Contribute on GitHub
• Join our Slack community
• Participate in our discussions
• Get started with Teleport

Introduction - Enabling Compliance for Database Access

(The transcript of the session)

Tom 00:00:01.259 Hello and welcome to today's remote session. This is Tom Becktold. I'll be your host today as we get into securing the software supply chain by following the audit trail. And I want to say thank you to the folks over at Teleport for making today's session possible. So before we get started, I honestly, I had to do some research. I was reading through the abstract here, and there was a lot of stuff I didn't really quite understand. I'm not a developer. I don't do a lot with coding anymore. I used to kind of dabble in it, but I'm very terrible at it. But today, we are going to be talking about the challenges of securing the software supply chain, and how an SBOM — not to be confused, I had to look this one up, not to be confused with an F bomb, which probably happens around these things sometimes. But this is a software bill of materials. I actually learned that. So I was impressed that I got something new today. So it's an integral part of knowing what software is being included in a release. And that really is important. We're going to review the problem, how an audit log can be helping with this, and making sure that your build systems are, in fact, secure.

Tom 00:01:06.806 In the second half, one of the gentlemen is going to show us how to secure CI and CD systems, as well as show us a short demo. So got a lot of ground to cover today. We want your questions, so please submit those as we go through in that Q and A box. Definitely want to make this as interactive as possible. I do have a poll question for you here shortly. And I think the gentleman might ask you some yes or no questions along the way. So if you wouldn't mind, if you see a yes or no question pop on your screen, please give us an answer if you feel that you want to. So anyway, let's move on and meet the guys, shall we? So first off, we have Mr. Donnie Hasseltine. He is the CEO of Packagecloud. And he's going to kick things off here just in a minute. We also have Ben Arent. He is the developer relations over at Teleport. Really happy to have both of these gentlemen on. I talked to them at length yesterday, and they really dazzled me with a lot of stuff that I had no clue about. So I think you guys are going to like today's session. So that being said, I got to go through our housekeeping slide really fast. Today's slides, some different documents, some URLs, they're available in the resource list. The two gentlemen here, Donnie and Ben, they gave me a really good list of a lot of links. That is in there. It's actually named Links from Donnie and Ben in the resource tab. You want those. Seriously, there's some great stuff in there. There's also a couple of case studies or PDFs in there as well that have some more information about [inaudible]. But definitely grab those links if you can.

Poll questions on software supple chain issues

Tom 00:02:35.119 So direct your attention back to the slides, if you wouldn't mind. I've got a poll question for the audience. So what is your biggest software supply chain concern? You came here today for a reason. Right? So what concern kind of launched that for you? Was it CI/CDs being hacked? Was it malicious third-party package or code issues that you might be having? Maybe it's an insider threat issue. Who knows? Or other. Let us know what other is. If you do choose other, if you wouldn't mind, throw that in the Q and A. Just put what it is. We'd love to hear what that is. So just throw that out there and we'll kind of check back with you guys on that part of it later. But let's give you guys a minute or two. It looks like some folks haven't voted yet. We're looking at about 20% so far. I'll just keep talking until we get enough votes. I'm kidding. I'm totally kidding. We'll show you the results in just a second. We'll give you guys just a moment more. But again, what's your biggest supply chain concern? CI/CD getting hacked? Malicious third-party packages or coding? Inside threat actors? Or other? So type that out now, there in that Q&A box. We'd love to hear what the other might be. So here we go. Let's take a look and see what we've got for results. So far, the biggest thing is that malicious third-party package at 66%. That's quite alarming, I would say, the CI/CD getting hacked. Donnie or Ben, what are you guys thinking about these results? Is this shocking to you? Or is it kind of on par with what you've been seeing?

Donnie 00:04:03.242 I think it's on par. I'll let Ben comment on there. I think a lot of times, especially when we take a look at just on the bottom part, it's always expect a little bit of another. Right? Because there's always the unknown unknowns out there. And they could have some specific concerns there. On the inside threat actors, I think that that happens a lot. We're going to talk about a couple of examples today. But you generally want to trust your team. Right? So I think there's a natural tendency to say, "I don't think it's someone on my side." So you start looking at what are the things outside that are affecting you. And really, it's not just third-party package and tools. It's really the dependencies that we're going to talk a little more about today, that tie to the SBOM. And CI/CD? I mean, I think that, maybe a year or so ago, people might have seen this a little less. But we've seen the CI/CD getting hacked in the past year. So I think that's probably raised some awareness up. Ben, what do you think about those results?

Ben 00:04:53.426 Yeah. I think that this is great. I'm glad it's not all other. We're going to cover these topics in this talk. I would love to know what the others are, Tom, if you could share some of those. And hopefully, we'll cover those in the talk, and probably address some examples.

Tom 00:05:07.337 So far, one person said executive order.

Ben 00:05:11.153 Oh. Donny has a section on that.

Donnie 00:05:12.885 I'm wondering if Michelle — yeah. I'm wondering if Michelle is talking about the executive order that just came from the President on SBOM and requirements of SBOM for doing business with the government. Or are you talking about some type of executive order? I'm curious if it's more specific. Michelle, are you talking about President Biden's recent executive order or another one?

Tom 00:05:37.977 We'll come back around for that one. I bet that's probably what it is.

Donnie 00:05:40.240 Yep. Got it. Yeah. SBOM. So specifically about SBOM, we're going to get that in detail. And then I see [inaudible] talks about you want to trust your team, what about the insider teams, other players in your chain? I think that's a really great point. That kind of talks to the third-party package and tools. What is your third-party vendor security system or process? And do you really know what's out there? A lot of times you just have a vendor answer questions. You have no way to validate if they're answering those questions truthfully or not. So I think that's a great point.

Tom 00:06:09.953 Definitely. Definitely. Really appreciate the audience. So far, you guys are already engaging with us. This is great. This is already kind of taking shape, and having an exciting kind of a discussion going on. So let's move on and get things kicked off. Donnie, it's you for a while.

Introducing the speakers

Donnie 00:06:27.232 Yeah. So I'm going to just go ahead and start off with the introduction. We'll go ahead and jump just straight into this. So first of all, Donnie Hasseltine. I come from a military background, then shifted into cybersecurity with a [inaudible] firm called Xenon Partners of which Packagecloud is a part. And now I'm currently running as the CEO and general manager of Packagecloud. And let me throw over to Ben real quick for just a quick intro.

Ben 00:06:51.216 Thanks, Tony. Yeah. Hi, everyone. I'm Ben Arent. I'm on the developer relations team at Teleport. I have a background in developer-focused SaaS tools. And one interesting link here. I have a link here for securing open source infrastructure. And this is the sort of initiative that we have, this sort of providing a free service to the open source community, to sort of secure a lot of their aspects of their infrastructure. And we'll sort of touch on some of these topics. But I might sort of dive straight into it. We're going to talk about the supply chain, the software supply chain. But I figured out a step back a bit. And there's a famous economist called Michael Friedman. And he talks about everything that you need to make a pencil. And no one person can make a pencil. And actually, I had to find a pencil because all I had was a stylus. But for everything from the wood — so you have to plant the trees to get the wood. And then you need the iron for the saw. You need the graphite, which is in a mine. And then you need to assemble it. And so you think a pencil is a relatively simple object. But the free market has to make the pencil. And that's sort of the same thing applies to the supply chain. We have raw materials suppliers, manufacturers. You can't really get full visibility into what everything is happening.

The software supply chain

Ben 00:08:13.116 And to some degree, this is also true for the software supply chain. We have a similar sort of process. You go from getting your raw materials, which could be sources and dependencies; your factory, sort of your build systems and your engineers; you have to ship it, which is sort of do you trust the network; your storefront, which could be application repositories; and then last up, you sort of deploy these systems. And depending where you fit, there's supply chains within supply chains. And I think Donnie will sort of touch on this sort of aspect. So similar to the pencil, there might be a very small library that you have. You think, "Oh, it's just two dependencies." But there might be a lot more behind the scenes. And one thing that is pretty different in 2022 is the discussions that have changed a lot. Traditionally, the last five or six years, we had lots of discussions around like, "We don't trust these products." Huawei or Russian antivirus software, we can't just use these. But of late, we've seen we can't necessarily trust American-made software because, in the example of SolarWinds, there was a nation-state attack in the CI/CD system, which sort of compromised a sort of very trustworthy piece of software. And I think, Donnie, you're going to go in a bit of depth for this one.

Donnie 00:09:40.129 Yeah. Thanks. Thanks, Ben. Yeah. So I want to break that out a little further. So if you look at Ben's initial one, we just talked about a couple of the key pieces to kind of [inaudible] that software supply chain. I want to break it a little forth and a little further out. And realistically, it could be a deployment integrity on the far right side of this. This is adapted from a Google Security blog through a project package that I was working with them on. But you start with a developer. And I think a lot of times you just think, specifically, at that realm, but you will have a number of other steps to get that software out. And the real issue with that is every one of those steps is a problem, or a potential problem. And we've already seen in the comments here a great comment. When Ben talked about the traditional logistics supply chain, that is very well documented. [inaudible] made this really great point. But the software supply chain is less documented. And that is absolutely true. And I think that we also find, in software, is the ability to pull a lot of disparate pieces out. And it's very hard to understand what's in those pieces out there. And if you look across this entire supply chain, every one of these components has been attacked. I mean, if we're to take a look at the first one, submitting bad code, case last year, about a year or so ago, when the University of Minnesota did some Linux hypocrite commits. They attempted to potentially introduce code vulnerabilities through the Linux kernel by using patches. And they were able to get those in before they were caught and stopped. Compromised the source control. In this case, from the PHP self-hosted Git server, attacker compromised that and was able to put two malicious commits directly in. Webmin, when you talk about modifying code, they were able to build — the attackers were able to modify the infrastructure. So it started using source files that didn't match the source control.

Donnie 00:11:24.596 This one that everyone knows about, right, the SolarWinds hack, Sunspot, is when the actual build platform for updates was compromised. Once you get that software out in the wild, it's obviously calling back out on — calling back out to dependencies. And in this case, attacker added an innocuous dependency that was able to connect to the software. And then once the software was out in the wild, he was able to modify that dependency to basically push malicious code into the software. The Codecov breach which occurred last year, where there were leaked credentials, were able to upload a malicious artifact to a Google Cloud storage bucket. And then when you look at package managers like ourselves, like Packagecloud, there was a way to run some mirrors for several package repositories, which were able to basically direct someone to a malicious site that gave compromised software out. This is a really fascinating one that really got us interested in, too, an Alex Birsan's Dependency Confusion attack. And what he did here was really interesting. I think there's another term they'll use called typosquatting. And where what he did was he noticed that many of these major companies and major software, their dependencies called out to private repositories and private dependencies. He was able to make public ones that mirrored the name. And in many cases, your software will default to the public version. And when that happened, he was able to open a door to inject malicious code. Thankfully, there was a white hat researcher reported it, and a lot of these got patched. But it also shows how fragile that ecosystem is.

Ben 00:12:58.532 Thanks, Donnie.

Donnie 00:13:00.258 Yeah.

Ben 00:13:00.936 Oh, are you going to take this one?

Donnie 00:13:02.906 No. No. I was just kicking off to say these three things. But go ahead, Ben.

Ben 00:13:07.698 No, you can go ahead.

Donnie 00:13:10.748 Really, as we said in this last slide, there's a lot of ways you can compromise software supply chain. We're going to focus in on three today, CI/CD and meatware, and then also SBOM. So I know we had some comments about the executive order that is now requiring companies to actually generate these SBOMs. And then Ben's going to talk about the CI/CD and meatware and some other ways that that can be a problem. So I'll turn it over to Ben here.

Software supply chain problems

Ben 00:13:32.668 Yeah. Thanks. And Craig, I like your comment. They often say in the restaurant industry — you said bad code, malicious, or just poorly written. They say you can either have it good, fast, or cheap. But you can only pick two. And I think that kind of applies to software. This is regardless of all three of them. But ultimately, making good secure software is pretty tough. And I think that's kind of why some of these examples that we'll go through can have different solutions to them. And there's not necessarily one solution to all problems, but these are three that we sort of brought up that are sort of interesting. And meatware is sort of like a semi-hacker term to refer to people [inaudible] in their machines. And ultimately, people are writing code, people are accessing systems. Often people are their weakest link. And in the cyber domain, there are insider threats. And I think one that was pretty interesting, Ubiquiti, which is a router company, they were hacked. But actually, they were hacked. But Insider actually said that he came as a whistleblower. It was like, "Oh, our company has been hacked," and someone was demanding a ransom. And he leaked it as a whistleblower. But actually what happened was he had hacked, he basically downloaded everything, gave them a ransom, and said, "Oh, I'm trying to extort so many bitcoin for giving this data." And this is a very interesting insider threat. He had access to a wide amount of systems, which he shouldn't of. And this could be mitigated by better monitoring of the network, see if there's data exfiltration. Endpoint protection would help. And just, generally, applying principles of least trust is sort of a good place to start.

Ben 00:15:25.589 And then next up, we also have all of these systems have people in them. And there's also outsider threats. So this happened in January that an open source JavaScript developer said he made a political statement to break thousands of apps. So he deliberately corrupted his NPM package because he was sort of annoyed and frustrated that open source maintainers don't get paid. Probably not the best way to get paid is to corrupt your package. So probably better ways to do this. But this could be mitigated in the Node ecosystem. It's a little tougher to deal with vendoring and pinning dependencies because many developers will often run an NPM update locally, which can cause all other sort of havoc in itself. And actually in this case, GitHub have the centralized maintenance of NPM, and they actually pinned it to an older package. So it reduced those packages from breaking. And I think this is sort of interesting. Then Donnie will go into this, that you may not necessarily think you're using Colors or Faker, but this could be a dependency, something else. And then you could completely break your deployment pipeline.

Ben 00:16:42.779 And then moving on to CI/CD, there's lots of attacks. Donnie gave a few examples. And actually, this is one that we publicly disclosed at Teleport. And this is sort of an interesting one because we have a blog post here of how you could access our cloud infrastructure via a pull request. And this is because many CI/CD servers have a lot of privilege, and you can do lateral movement across services. Why is it important to secure them? Often your CI/CD is sort of the source of truth for your build, and you want to be able to create sort of atomic builds which you can trust. And so you need to have complete trust in your CI/CD service. And so there's a few things that we did to reduce this possible attack. By the nature of getting other people's code into your code, you're potentially opening up a possibility for someone to inject something bad. The code reviews for external PRs is good. We actually put in a new bot system. So two people have to approve any external PRs coming in, which is a very unique situation for open source projects, but we also work with some of our customers and vendors. And then also, just locking down your CI/CD services so there can't be lateral movement between build containers and images. And then also just generally vendoring your dependencies can be a useful way to deal with some of those possible SBOM attacks. And then I think, Donnie, you're going to talk about the SolarWinds CI/CD attack?

Donnie 00:18:20.056 Yeah. Yeah. Happy to. So just to give a quick, broad overview of this, and this occurred last year, and I think one of the things that to notice about this is, first, when you talk about actual major breaches and hacks, it's important to understand that there's, especially when they deal in this case where they're from an APT or an advanced persistent threat, they've been in there for some time. The persistent part of that APT is because they've actually gotten in, and have been in your network for quite a while. So I think in this case, I think September 2019 is when, in retrospect, we did the post-mortem, threat actors initially were able to gain access to the SolarWinds' internal network. And then they started testing ways to inject code into their Orion product. The code got injected in February 2020. The update that SolarWinds pushed, that tens of thousands of people downloaded, was pushed out in March. And FireEye first caught wind of it by seeing some malicious traffic in their network in December, and then announced it. And that's when this all blew up, in December 2020. So it was there for a while.

Donnie 00:19:27.847 And it's important to understand, when you think about defensive points - I'm going to go a little bit into the cybersecurity piece here — is understanding the cyber kill chain, which understands the process with which someone needs to use to actually go through a malicious code inject. It starts out reconnaissance and doing research. Then once they figure out a way to do it, you start weaponizing that back door. You find a way to deliver it into the software. Once it's delivered and working, then you try to exploit it. And you go through this installation, command and control back to the malicious actor, and then eventually you need to get hands on keyboard where they have full control of your network. And in this case, that attacker at the top part there, they were able to compromise that, and compromise a very specific DLL file that was used for updates for the Orion core business platform. And that effectively opened a backdoor. So if I were a customer of SolarWinds, I downloaded their software. Once I downloaded that update, it created a backdoor. That software then started doing research inside, and reconnaissance inside, the target network. And it, eventually, kind of called back out, and kind of gave a report back to the hackers. From there, they were able to go back and forth, and eventually pull in additional credentials, make lateral move in that network, and then allow the attackers to get specific access to that individual network. And then they begin to prioritize which organizations they had access to, and started trying to do direct access to their networks, and see what they could steal or exploit. So this is a very, very complex attack. But it's important to understand how many steps in the process occurred, and how long it took for this actually to be exploited.

Software supply chain solutions

Donnie 00:21:05.707 So with that, I'll just jump into the last piece. Ben talked a little about CI/CD. We'll talk about the human element with me where the SBOM is something that's really come to the forefront right now. Mainly because, as Michelle noted, President Biden's executive order. This is something that had been pushed for a while. And CISA, Jen Easterly and the CISA team, did a great job at informing the White House and getting this out there, which is specifically that if you're going to do business with the federal government, you need to basically have an SBOM, and have it published, and have it accessible. So it just shows you have an understanding of what's actually in your software. It's important to know that the SBOM is a list. Right? It doesn't necessarily give you full comfort, but it gives you a sense of what's out there. So moving forward, why is this important? And I think Ben gave a great example of the pencil early on. But if you just think about it, think about something like the food we eat. You can look on the back of any food package and see every ingredient. If you have a specific dietary concern or preference, you can look very quickly and say, "Am I eating something I don't want to be eating?" Right? "What are my calories? What are these different vitamins here?" But software doesn't really have that. The SBOM is really that piece to say, "These are all the things that are involved in this." Or if you were to take Ben's pencil example, "Here's a list of all the suppliers and all the different components that actually contribute to this small, low piece of tech that we use every day."

Donnie 00:22:27.650 So the real question you need to be asking is, do we know where our code comes from? And I think a lot of times when we talk about really reputable large companies, we think, "Oh, well, it's secure because they have control over it, and they're the ones who write the code." But as Ben kind of hinted at earlier on, is that code is probably made up of a lot more things out there. And that's where it gets a little scary. Because 90% of all software is composed of open source dependencies. Which means when you download something from a reputable company, and a secure company that takes it seriously, only about 10% code is really the customized code that they wrote and they fully, fully control with their developers. The rest of it is really out in the wild. And this is a great cartoon from Mexico City that I love. Because there's a lot of stuff on the internet, that the internet hangs on, that we just don't quite realize. And a really great example that happened in, I think, 2016, where a developer had written a whole bunch of these little quick small code libraries, and published them to the NPM registry and others. And he got in a dispute with NPM because he had a project, and a company that had that trademark name wanted to basically take control of it, and wanted to kind of drop the name of that project. And it was really probably done very bluntly and not very nicely. And he got mad. So he said, "Fine." NPM basically came back and said, "Yeah, we're going to forcibly change your project name." He got mad. So he just unpublished all the software he had written over many, many years. It just so happened there was one 11-line piece of code called Left-Pad, which just lines up things on a website, that most of the internet used. And as soon as he unpublished that from the NPM registry, just companies all over the world just broke. Websites crashed. The internet broke for a few hours until someone just quickly figured what it was and rewrote the 11 lines of code and republished it. So this is just a small point. This guy gets in an argument with a bigger company and you think, "How does that affect me?" Well, it affects you because of these open source dependencies.

Donnie 00:24:30.293 And I'll just try to give you another visual example of what this means. When we look at Packagecloud, we have a lot of customers. And we have a lot of visibility on their repositories. And we try to work with them to help them see where they might have issues. The real concern, and Ben hinted this earlier, is that some of these dependencies, we're stepping out of that initial software, the open source dependencies. But some of those dependencies now have indirect dependencies, which are a little bit scarier. So just take a look at this. This is a small piece of software in the NPM registry. It's used by a lot of people, several of our customers out there. If you look at in the registry, you see, okay, it was published four years ago. It's been updated once or so. And it has two dependencies. Very small, simple piece of code. And that's great. You know what? That clue means that I need to look at those two dependencies and just make sure they're secure. The problem comes is when you peel back that onion, you find out there's 102 dependencies to those two dependencies. So what you thought was just a small little piece of software that you can kind of take a look at and see, is that a problem, you find really looks like this. And this is used by Google's deps.dev, which is in the links where you can type in open source dependencies and visualize them. But it just gives you an idea of how big the problem is. So in large pieces of software that have hundreds of dependencies, they have thousands of indirect dependencies. And if you are really trying to secure that, you've got to understand that when you're building your SBOM, you want to know what each of those individual things are, and make that list. And it can be a very comprehensive and long, and in some cases brutal, list. But why that's important is so you know where your software comes from, and you can communicate that with your customers. And then when something happens, like Log4j, for example, when that issue just popped last year, many didn't even know that they actually used that individual Log4j dependency. So having an SBOM allows you to very quickly see what we are using, this is where we're using it, and then move to the next steps of, basically, trying to seal it up and protect yourselves.

What should be in an SBOM?

Donnie 00:26:28.985 And I'm just going to briefly cover in what should be in an SBOM. There's a couple of different formats, common formats for SBOM. There's only three of them that are out there. There's SPDX, which is the software package data exchange. We had a software identification tag. And there's CycloneDX. And I think we have some links in that. But those are ways, they're open source, you can go in and help use those to build your SBOMs. But these are really the key pieces that are in the SBOM. And if you look, this is a link to the NTIA report which it says, "This is what you have to have in it." So when you talk about the executive order, you need a list of all the components of your software. And each one of those components you need this information on. So you can actually be sure of what's in your software, look harder at securing it, and then move from there.

Donnie 00:27:17.578 So I'll just pause there and maybe just do a quick pulse check. Because I think we've kind of covered the meat of what we're going to talk about, and we're going to slide in to talk a little bit about some of the things, some of our experiences at Packagecloud in dealing with this problem. And then Ben's going to talk a little bit about experiences at Teleport dealing with this problem. So I think you guys should see it. I just want to make sure everyone's following along. If there's any other questions that you want us to hit on right now, where we have a quick, natural stopping point, or pause points.

[silence]

Tom 00:27:55.294 I think they might have gotten a little shy on us, or they're pondering some questions.

Donnie 00:27:59.567 All right. There we go. [laughter][crosstalk].

Tom 00:28:02.207 I was just checking, giving them a minute to see. I did want to mention, for the audience members real quickly, we did upload the slides. You may have to refresh to get to them. It was a little bit after we started. So I apologize for that. We didn't have the PDF uploaded in time for the start. But yeah. Send in any questions you have now. Anything coming in? Let's see here. Kind of quiet. That's okay. Let's continue on.

Ben 00:28:27.036 Donnie, I have a question for you.

Tom 00:28:29.034 Oh, good.

Donnie 00:28:29.884 Go ahead, Ben.

Ben 00:28:30.789 So if there's this new executive order, what sort of time frame do people need to address it? And which companies have to go about this SBOM duty?

Donnie 00:28:44.453 Yeah. I was just looking at it, and I'm blanking out on the time frame, Ben. But the key thing is it's specifically saying anyone that does business with the federal government. So it's all the software companies and the federal government. And it's, also, the main timeline was telling all the government agencies to, basically, ensure that they had a list of the software they used and obtain SBOMs in that time frame. So it's mainly directed at the government because with the US system, it's a little hard for us to, without passing an actual law. So if you're a company that doesn't do business with the government, then you probably have a little more room. But it's still a really great practice. And that's what we're trying to highlight as a way to know what your code is made out of, so you can better protect yourself.

Ben 00:29:26.752 Yep. And so I guess people who are doing FedRAMP certification, it will likely —

Donnie 00:29:30.987 Absolutely.

Ben 00:29:31.000 —just be part of that to keep their compliance in check. As I talked to someone recently, you never obtain compliance. You only lose it. So this is probably the case in which you —

Donnie 00:29:42.577 That's great. You probably need to look at it as a continuous process. Right? Because I think, realistically, this is one of the challenges with some things like SOC 2. Right? There's Type 1 where it's a moment in time, and there's the Type 2, which looks over six months. But even over six months, you have gaps in that. No one is constantly watching all the time. Yeah. There's a couple of great questions in there.

Ben 00:30:07.481 Donnie, just repeat the questions for people.

Donnie 00:30:10.863 Yeah. Sure. Absolutely. So the first one is: SBOM can be used in cloud and on-prem devices, additionally installed in all endpoints. Yeah. Any piece of software can have an SBOM, whether it's on cloud or on-premise. And I think that's one of the beauties of open source projects. Right? The reason why I think a lot of software developers and a lot of security professionals like the open source projects is because the code is in the open. This is especially true when you talk about cryptographic standards. Right? We get very nervous when someone says, "I have this new encryption thing," but is not willing to show people how it works. Because if you don't show me how it works, there's no way for me to prove that it's actually secure. So there's an aspect of that that's actually really important. Kivi has a great question about, are there IP concerns? And I think there are. I think there are some companies that have SBOMs that will only distribute it under NDA to existing customers. So I think there is that. There is some concern about, do I publish my SBOM publicly? Or do I kind of keep it close hold and deal with it when I need to? Those are business decisions, obviously. But I think that as we pointed out, the real key point is understanding all those other components you're not directly building. So if there is a piece of code that your company internally develops on their own, it's their IP that they have built on their own, you're really talking about that small percentage of the code, right, that you built on your own, that really wouldn't be in the SBOM. You would put, "Custom code. We developed it." But the other piece is when you distribute that, it's not actually saying — it's not showing the source code. You're just saying like, "These are dependencies that I use that make up my source code." So I think that's one of the ways that you get past it, is you're not necessarily showing your actual source code and your IP. You're just saying, "These are the pieces of the pie." So to use Ben's example, you're saying, "I used graphite, And I used graphite from these mines." Or "I used rubber from these mines for the eraser, I used woods from these forests." But you're not going through how you actually assemble that pencil on the back side.

Donnie 00:32:08.698 Julian [inaudible], he says, "SBOM content, provision of clients, and IP protection must be all explicit," absolutely, "in contract language." This would be something that would be in your master services agreement. Certainly, if there's any concerns, if it's not fully open source, you'd want this probably under NDA. And so you have some ability to control over it. But it's important to have that in contract language. And then Mark said, "How much resources does it consume?" That's an interesting point. And maybe I want to make sure I'm understanding your comment or question, Mark. Is it the resources to build the SBOM? Because that is certainly an issue. It can be time-intensive, and that's why there's some products out there that will help you generate it. But are you talking about how many resources your software consumes once you've deployed it? I just want to make sure I understand that, but. Okay. I see John's asking about that. John, we'll get information to you on the slides. And okay, Mark, machine resources. So yes. I don't know that SBOM requires you to say, "This is the type of machine resources they used." And I think that's probably variable, based off of machine. But that's a really interesting point. I think we've all seen that, right, when we see our machines slowing down because of a piece of software. What's interesting about that is, if you see it slowing down, that's an indication of other problems. Right? We all know that. So the SBOM would give you an idea of being able to look at what other pieces of that software might be putting a drag on that.

What is Packagecloud?

Donnie 00:33:39.893 Let me just push on here, and we'll continue to answer questions as they come in. But I'm going to briefly just say what Packagecloud is, and how we're kind of working with customers to address some of these things. And then I'm going to hand off to Ben to do the same thing for Teleport, and talk about how he works with them. So Packagecloud is a unified way to manage all of your software artifacts in any language. One of the things you will see is we have some customers that have a very specific — like they just released Debian packages, or they released a Maven package. If you're building out infrastructure to distribute those packages, you have to build it kind of separately. One of the nice things about Packagecloud is you can put all those repositories in one place, which gives your developers, and your security team, and your quality assurance team one place to look at all of your outward-facing software that you're handing off to your customers. We do this in the cloud. We also do it on-prem for customers that want to either distribute it internally on their infrastructure, or even distribute it externally to customers using their own infrastructure. This usually sits at the end of that CI/CD workflow. So when we think about the software supply chain, it's on the far right side of that slide. And we really just want to make sure we're getting a very fast and simple service, where software developers can get their software out very quickly and efficiently.

Donnie 00:34:54.544 But in doing that, when we see some of the problems we've talked about today, we noticed that almost all of our customers are starting the left side and looking left to right. Because we're on the far righthand side, we have a unique ability to really look right to left on that software supply chain. And I think that the thing that we're also looking at, is it's not that we're just trying to protect our customers. We're really trying to give visibility and help protect our customer's customers. Because once they deploy that software through us, their customers are the ones downloading it. Right? And we have that piece of, if there is a breach or problem with that software down the road, that could hit on us. It could also hit on them. So other things we can do at our through point where that software flows through, we can provide better visibility. And that's caused us to rewrite a pretty aggressive 2022 roadmap on this to add a few things, specifically the SBOM like we talked about, We've also looked at doing a single point of control for dependencies. Now we see some of our larger customers like GitLab, GitHub, Netflix, they have very, very big security teams that are super aggressive. And we have a great conversations and working with them. And they're able to handle a lot of this on their own. We have a lot of smaller customers that don't have the resources to do all this. So because we're holding your software repository and those artifacts, can we take those artifacts and help you get an SBOM out of it? Can we also said that once you deployed it out to your customers, instead of calling out to the wild where a malicious inject, either mistake or malicious, now infects your customers through your software, can we point all of those dependencies through Packagecloud, where we can internally review, mirror, and scan for vulnerabilities, to make sure that your customers are only getting not just the approved software, but the approved dependencies that you put your stamp of approval on?

Donnie 00:36:39.540 And then to help those developers and our customers, giving the ability to go back and look very closely at the code ancestry. So not just your code, the IP that you built, but look at in that IP that you've built, what are verified commits? What are unverified commits? What are the reputation scores of the developers involved here? But then take it a step further, and look at all those open source packages. So you might see an individual has only been on GitHub for two weeks, and has done no deployments. So it suddenly drops like five or six pull requests and an open source dependency. We have flagged that as probably a low reputation and something we should look at a little closer there. So these are the pieces that we're adding to the platform in 2022 to try to do a better job of handling some of these security things, and give our customers a little more firepower to handle it. And Craig, thanks. You dropped the executive order link in there. We will see if we can add those links to the resources. So I appreciate you adding that in there. And then obviously, if you search for America's supply chain, transportation, industrial base, or Biden's Executive Order 14017 of February 2021, those will give you some of the data you need. So we'll see if we can add those links there. Let me hand off to Ben to talk a little about Teleport, and kind of the pieces they're doing on their side to secure the supply chain.

What is Teleport?

Ben 00:38:03.465 Thanks, Donnie. I'm really excited to view the 2022 Roadmap. I think, well, we'll dive into it. So let me just dive into what is Teleport. So Teleport is the easiest, most secure way to access all your infrastructure. And also interesting, we're an open source, open core company. So all of our development happens in the open in GitHub. And that sort of alludes to why we had this malicious pull request. So we sort of run into some of the issues that sort of Donnie mentioned. But what does that really mean? So nowadays, developers and teams have a myriad of resources for multiple clouds. We started off providing just SSH service credentials. But nowadays people are accessing Kubernetes clusters, databases like web front-ends. And then you have multiple teams across multiple clouds. And it becomes much more difficult to access all of this infrastructure in a standardized way across all of these different clouds.

The 4 essential elements of infrastructure access

Ben 00:39:01.348 And we see this sort of broken down into what we call the four main essential elements of access. The first is just connectivity. And this may seem sort of relatively simple if you just have one AWS account. Multiple people have multiple AWS organizations. Or we see people who have Edge devices, or you're deploying software into a hospital, or we even have people who have Teleport in cars. It's amazing where machines are, and who gets access. And the first step is authentication. Just proving that myself, Ben, has access to these machines. And then it's once I'm authenticated, am I actually authorized to access those machines? And if I can access those machines, how long can I access it for? And I think this kind of goes back to the Ubiquiti hack. He was authorized. He was authenticated to access the Ubiquiti network. But probably wasn't authorized to download all of the code. And that's just due to not having that much — not deploying principles of least trust. And then last up, a very key part is just auditing. You need to know what people are doing in these systems. And I think something that I'll touch on later is also, what are systems doing to other systems? Because it's not always a human. It can be also machine to machine interaction.

Teleport Access platform capabilities

Ben 00:40:29.859 And then to just break it down, Teleport's Access Platform, we deploy sort of zero trust. We're certificate-based. And I'll sort of go into what this means. And then providing just-in-time access to only provide the resources when you need them. And then also, just getting the complete visibility and creating the audit log, which you can then pipe into your SIEM or other tools to know what's happening. And then we cover all of these resources. So databases, servers, Kubernetes clusters, desktops, and applications. And so you can really secure and take these devices off the internet, provide basically very secure, easy access for developers, but make it very easy to get access to. So I've a slide on how Teleport works. But I'm going to skip over this one, and sort of talk about one of the problems with — I think there's a transition here. Okay. So with sort of traditional keys or passwords. So let's say Bob is hired, and he joins, and he gets his new laptop. He runs ssh-keygen. He now has a public-private key. Maybe an Ansible script gives his public key to all the servers so he has access. If Bob is terminated, you may have to run your script again, or remove that key from those servers. He might still have access to the resources once he leaves. And we see this across a lot of our customers. Once an employee is onboarded, it's much easier to onboard people onto the systems, than necessarily cleaning up and removing them from all of those systems. Back in the days of working in a corporate HQ, security would unplug you from the computer, escort you off the office. You no longer have access to the network. But nowadays, with everyone working from home, everyone has complete access to the network. So it's much harder to sort of stop their access.

Human access – Teleport demo

Ben 00:42:24.434 And this is sort of where SSH certificates come in. This is a feature that's been around in OpenSSH for a while. And instead of using sort of long-lived public-private keys, Teleport builds on top of OpenSSH's certificates. So we issue short-lived certificates for access. So Bob is employed at the beginning of his career. Maybe he takes a few weeks of onboarding. Five weeks in, he logs into Teleport. Then he only gets an eight-hour certificate for access. And then this is only available for his window. If Bob is fired, he doesn't have access anymore. First, he would have to authenticate and get access to it. So I'm going to do a quick demo for the human aspect here because I always love a live demo. And I have to flip this over. Great. So this is the web UI. We support a full terminal. I'm going to authenticate with GitHub. And here I have an inventory of all of my machines. So if I come in here, you see what I've done is I've just logged in as the root user. And this has executed a pam-script. It sort of told me that all activity has been recorded. This actually has PAM, which creates the user. You can go about just using this host as you would normally. There's other benefits. You can share and do other things. But I'm going to just exit this. And in our audit log, you can see that I have the session, and this session ended.

Ben 00:44:14.366 And I think another benefit here is we have session recordings, which is almost like a DVR for your terminal. And so along with raw outputs, it's very easy to just go in and sort of watch what happened, and see what an individual did. And this is the same for both Kubernetes, databases, and desktops. Database is a pretty interesting one because servers in sort of the modern era was seen as cattle, instead of pets. And so there's less sensitive data on them. But when you provide someone access to your PostgreSQL database, there's potentially lots of valuable information in there. And the same, we also have support for Windows desktops. So you can cover all of your access needs. So I'm going to stop this demo. I'm back.

Ben 00:45:17.576 And so let's talk about — we've seen the humans from meatware. Let's talk about the human aspect. And this is sort of a similar problem as I went before. So for machine access, you set up your Jenkins server or whatever CI/CD service you have. You create keys for it, public-private keys to all of your build infrastructure. You probably set it up a while ago. It's been awhile since it's rotated. No one wants to touch the build system. And then there's a Docker vulnerability. People get access to the machines, they get their SSH key. Next thing you know, the CPU is going crazy, and you find out the keys were leaked from GitHub and they are crypto mining on the machine. And so did we rotate them? What happens here, like this point in the ops lifecycle, it's not broke so you don't fix it. Then there's this big scramble to rotate your keys, and sort of trust the machine-to-machine access. I think there's time for a question, Tom, if you want to ask one? Feel free to ask me questions as we sort of go through this.

Teleport cert renewal bot: tbot

Ben 00:46:28.057 And so this is a new problem that we've solved at Teleport. And we're solving this by providing the same short-lived credentials for machine access. So instead of using public-private keys, you can use tbot, which is an open source project to create and issue new short-lived certificates. And what's pretty cool about this project is it's another open source project, in which tbot will automatically renew certificates for you on whatever case you need. So this is sort of an example of our script. I won't dive into it, but it will always renew certificates every 24 hours, or whatever period you like, to really make getting renewed certificates very easy. Just to review, Teleport has a new machine-to-machine bot that we have sort of in Alpha. It's not available yet. It will be coming out in Teleport 9. If you're interested in this, and this is a problem you run into, please reach out to me. I'm Ben at goteleport.com. And it will also have the audit log for this machine-to-machine access. And another capability which is super interesting is the ability to lock the renewal of certificates centrally. And I think that's sort of that [inaudible] chain which, going back to Donnie's point, it's the SolarWinds hack. Once you realize that you've been compromised, how do you lock and remediate these actions? All right, I'm going to take a pause. Go to Donny. He can go through some of these other projects that are interesting.

Tom 00:48:06.610 Actually, real quick, guys, can I just pop in really fast? I want to back up just for a second. I'd actually kind of want to do a pulse check with the audience here. You had a very gracious offer for this new item that you guys have. So if you guys are actually interested in this, I'm going to send a pulse check out. Please hit yes or no if you are interested in learning more from Ben on this cool new item. So please go ahead and make your choices now. We'll give you a few seconds here to make a choice. But we'd certainly love to send you guys more information, if it's appropriate. But let's do that. And I'll end that. And sorry to butt in. But I just wanted to give that an option for the folks in the audience. So there you go. Back to you, Ben.

Ben 00:48:51.554 Thanks, Tom.

Donnie 00:48:52.659 Yeah. Thanks, Tom. Yeah. I'll just hit the bottom three. And then I think Ben has some of the top ones there. But the three I put in there that are kind of interesting is the Linux Foundation's SPDX. That's just one of the ways you can basically create SBOMs. So if you want to look at, you can very quickly find the others if we — I think the others are in the links we shared. But that's a way to take a look at it if you don't have an SBOM, or you think about how to do it. Two big things that we've worked with Google owners is SLSA which is supply chain levels for software artifacts. We're starting to work on a, basically, a matrix of all the things you need to do at different levels so that when you download software, you know that that chain has been checked and verified. And they also have another open source project, deps.dev. And that's where you can kind of look up common open source dependencies, and then get that quick information back. If you, for example, see you're using an open source dependency, you want to quickly learn about it, type it in there. You can see a graph of all its sub-dependencies. And you'll get all the information on whether there's a current CVA associated with it. So, Ben, do you want to just cover the top ones you added?

Ben 00:50:01.157 Yeah. I'll keep going up from the bottom. So Panther SIEM. This is another product that we use. I think it's great to have all this audit log of activity. But it's also good to have a system to reduce the noise. We're just sort of friendly with the guys, no direct relationship. We just love the product. Another interesting project is reproducible builds. I think it's very important that the software that you build is sort of true. And that other people can create the same software. So it's a great project here. Package Hunter is a similar project for compromised packages. There's a great blog post from GitLab. And then last up, Sigstore is another new — I think it's a CNCF project now, for both signing and distributing projects in an open source manner. And they're relatively new. They're just getting started. But another good community project. So moving on, oh, I think I've gone a step too far.

Contact info to reach the speakers

Donnie 00:51:03.769 So I think we clicked it at the same time, Ben. There we go. And it looks like the [inaudible] is a little jogged here. But we just want to put out ways you can contact both us and our teams, either directly or at Support our LINK. Please follow us on LinkedIn Twitter. Our community slacks are there. You're welcome to join there and ask direct questions there. And then I think Ben already gave his email. I'm [email protected].

Q&A

Tom 00:51:29.247 Excellent. Thank you, gentlemen, I've got some questions here that I've got for you as I was taking some notes. Audience members, we've got time for your questions as well. So if you have a burning question that we haven't gotten to yet, throw that in the Q and A. But my first one here, let me pop over to my notes, what would you guys consider, if you can make the recommendation, what would be in your mind the most secure CI/CD platform? Or maybe throw a couple of them out there? Anybody?

Ben 00:52:03.369 I think it's a great question. I think —

Donnie 00:52:04.593 Yeah. Ben, my only caveat is I always get, from the cybersecurity standpoint, get nervous to say what's the most secure one out there. Because you always have to assume they've been breached. Right? So I don't mean — that's not an insult. I'm not saying I know some secret information. But the key thing is maybe asking questions like, what does their process look like? Can you validate what their security processes are, evaluate their SBOM? But then you've got more specific experience with them. So I'll let you jump in there. Sorry to interrupt.

Ben 00:52:33.430 No. I think that's great. So you probably have to evaluate whether you're looking for a hosted provider, or one that you run yourself. Because there's benefits of running it yourself, but then [inaudible] the maintenance overhead. So I think this kind of comes to Amazon's shared responsibility model, which is basically everything. So Amazon is responsible for securing the data center, but you're ultimately responsible for making sensible IAM credentials. And I would apply the same thing for looking for CI/CD services.

Tom 00:53:06.081 Okay. Excellent. And along those lines, since kind of making some kind of recommendations, what's a good tool for alerting and remediation for when an attack does happen?

Donnie 00:53:19.676 That's another great question, Tom. And it really depends on a lot of different things. Right? So let's just talk at the most basic level. Right? If, for example, you're using GitHub, which a lot of our customers are using, just going through their security checklist and enabling things, like putting down your security file, your README file, and explaining how to contact you if there's an issue or vulnerability. Turning on Dependabot alerts. So when a dependency is compromised, or there's a vulnerability discovered, you get a thing that pops up that you can quickly look at, validate, and do a pull request on. So that's at the really basic level. I think when you get higher up, it really depends on the infrastructure you're using. Right? And how you configure it. If you're using AWS, like Ben said, you can configure the security hub, kind of tie it to a service like PagerDuty, and set it up so that when something happens, you're getting immediate notifications. And a lot of times it's just a queueing that — setting your parameter so if there's an anomaly, your cued that something looks weird to get a human to look at it. Right? The problem with modern software is it's so complex that you're really just looking for something that just breaks, as we'd say in the military, breaks squelch on radio to say, "Okay, this looks weird enough for me to send a pager out. And then I want to get my engineers to look at it and verify whether it's malicious or not." Ben, do you want to add anything to that?

Ben 00:54:42.868 I think the one tip, if you're using AWS, which a lot of people are, just look into using AssumeRole for your team. By using AssumeRole when teammates access AWS, you get a much broader CloudTrail log. Without using it, you don't get as much telemetry. And then make sure that that CloudTrail log goes to some sort of alerting platform. It's a great way to figure out if someone was to accidentally change the fireball rules or something more malicious. But that's probably one tip. All right. Brenda had a question. How do your tools work with OT or scatter networks operating the grid? So Brenda, I know we have some customers in the energy space. My actual brother-in-law works in the UK grid, and actually, he told me that they had a cyber operation. And he had to tell him that there weren't many computers in the operating room. It was all mechanical. But I guess maybe the US grid is being updated. I'm not that familiar. But I'm sure our team would love to chat more. The grid is digital. And there's always sort of interesting edge devices. So just reach out to goteleport.com, and we're happy to chat more. And then for SBOM, Donnie, do you want to take that?

Donnie 00:56:07.253 [crosstalk]. Yeah. I'll just say specifically on that, Brenda, yeah. I don't know that we have any SCATI customers. But really with Packagecloud, we're talking about deploying software. So in most cases, for those type of customers, we'd be in a situation where it would probably be an on-premise hosting that they can control within a network and firewall they have control over, to distribute internal software within that network. I think the thing with SCATI you always have to be careful of is, any aperture where you're calling out really becomes a window where there could be malicious ability to try to reach in and control that. So I don't think we have too many customers using it for that specific reason. And Brenda just added, "Some are digital and some are serial connect." And you're absolutely right. So I think for the ones that are digital and serial connected, I think Teleport offers some pretty good options. And I think that it really depends if they're distributing or utilizing software. And that's where Packagecloud would come in.

Tom 00:57:09.956 Excellent. Thank you, gentlemen. At this point, I want to give each of you just maybe 30 seconds. Let's start with Donnie. If nothing else, what would you like the audience to kind of take back with them, or maybe just start doing tomorrow?

Donnie 00:57:25.316 Yeah. Thanks, Tom. I think the biggest thing is just know where your software is. And just ask the question. If you're on a development team, it's worth just going to your team and just asking some of these questions. Like, "Do we have an SBOM? How do we manage when a vulnerability pops like this? What are the pieces we're doing?" So in many cases, you might not have the answer. Your team may not have the answer. But just the process of asking the question opens the dialogue, and lets you highlight, "You know what? We need, maybe, to step outside and get someone to come in who's got some expertise to talk about this." So try to know where your code comes from, and ask smart questions to stimulate that discussion in your team, so you can protect your organization.

Tom 00:58:01.370 Makes sense to me. I like it. Good, sensible advice. How about you, Ben?

Ben 00:58:07.482 That's a great question, Tom. I just think just review the systems that you have working. Many access systems and CI/CD systems, they often set up once and people forget about them. So maybe just ping the engineer who's working on them, like, "When was the last time we actually updated X?" Or "Bob left the company last year. Did you remove his SSH keys?" And then let's sort of ask those questions. But okay, maybe time is to sort of clean things up and do your spring clean of your infrastructure.

More webcasts on the way

Tom 00:58:40.211 Good stuff. I like it. Be vigilant. Be aware. Find your stuff. I like it. This is good stuff. Excellent. Well, thank you, gentlemen. Really good stuff. Like I said, I learned a lot today on a topic that I really didn't know a lot about. So I really appreciate it, you guys. You made it clear for me. I really loved the examples. The pencil example really sold it for me. So like, "Oh, okay, now it makes sense. I get it." So thank you for that. I appreciate it. We have a lot more remote sessions coming up. We have, let's see, resolution intelligence for situational awareness. That's happening on Thursday, the 27th. Data privacy insights. We've got a special one on Friday. It's actually Data Privacy Day on Sunday. So very apt to have this one on Friday. So we're doing that one. Nuclear ransomware is coming up, 3.0. We thought it was bad, and then it got worse. Uh-oh. That one's in February, So that one should be — I think that's a Roger Grimes. It should be a fun one there. So the link is in the resource tab there. You can sign up for any or all of those. Again, I want to say thank you to Donnie and Ben for just a great presentation, and the folks over at Teleport for making today's remote session possible. If you haven't already, you can download your certificate of attendance. It's in that widget bar. Look for the little thing that looks like a certificate. But that should be good for one CPE credit. If you have any problems getting that, you can always email me, [email protected]. This concludes today's remote session. We'll see you on Thursday.

Ben 01:00:10.014 Great. Thanks, Tom.