When should a startup hire a CISO?

Key topics on A Security Chat with Donnie Hasseltine

• Skilled cyber practitioners are in high demand, and transitioning to the cybersecurity industry is not a difficult transition for those already practicing cybersecurity in the military.
• Xenon is a tech private equity team that buys and operates B2B SaaS companies, with a focus on distressed assets.
• Acquired startups tend to have a general lack of knowledge of basic security hygiene and where the risks lie in the company.
• There is no such thing as a normal day for a CSO.
• Companies that can’t yet afford a CSO should at least put one person in charge of security.
• TeamPassword is a password manager, and password management is a critical tool for not just business but also for individuals.
• It’s important to consider what leads to hacks, such as the recent SolarWings hack.
• Addressing computer security vulnerabilities requires addressing the human aspect of security since it may be the source of vulnerabilities.

Expanding your knowledge on A Security Chat with Donnie Hasseltine

• Teleport Database Access
• Teleport Kubernetes Access
• Teleport Application Access
• Teleport Getting Started
• Teleport Access Platform

Transcript

Ben: Welcome to Access Control, a podcast providing practical security advice for startups. Advice from people who've been there. In each episode, we'll interview a leader in their field and learn best practices and practical tips for securing your org. For today's episode, I'll be interviewing Donnie Hasseltine, CSO at Xenon partners and CEO at TeamPassword and TeamsID.

Getting into the Security Practice

Ben: All right. Thanks for joining us today, Donnie. To kick it off, can you just tell me how you got into the security practice?

Donnie: Yeah. Thanks, man. It's great to be here. So I started getting into cybersecurity as a military officer. And even though I was an infantry officer which is generally viewed as a little less technical role, as I led larger units in formations, I could certainly see that cyberwarfare was kind of a critical part of my mission and taking care of my team. And I think personally, I've always been an intellectually curious person. And when you start seeing everything kind of explore across the internet, I was kind of getting frustrated because I knew I could order anything I wanted on the internet and I could do anything I wanted, but I didn't really understand what was happening when I clicked that “buy” button or clicked things on there. So my final tour in the Marine Corps, I get stationed to Silicon Valley. I began interacting with startups, military innovation groups like the Defense Entrepreneurs Forum. I saw the the Defense Innovation Unit. I got involved in Stanford's hacking for defense class. And I think these experiences all kind of highlight the importance of cybersecurity. And from there I ended up applying to Brown University's executive master in cybersecurity, and that program was a really great program for me. It lined up on my interest, and its format allowed me to do it while I was still on active duty. And through there I met Xenon's founder Jonathan Siegel and when I decided to retire from the Marine Corps —

Ben: Is there any division between the different services — between Airforce, the Marines, the Navy as far as where cyber lives?

Donnie: Yeah. That's a great question. Organizationally in the DoD, you have the — and I'm going to try to just simplify it for those who aren't familiar with the bureaucracy there but basically headquarters in the joint commands like USCYBERCOM. USCYBERCOM is basically the central command for all cyber operations in the US military. And each service has a component of that. So within CYBERCOM, you have Air Force Cyber, you have Navy Cyber which is 10th Fleet, you have Army Cyber and you have MARFORCYBER. And so what you end up having is there are parts of that, the services keep for themselves to help control their networks and do the service-specific things. But they also give cyber practitioners to the joint community that really solve cyber offense and defensive problems across the entire military enterprise. They each do it a little differently as far as how they train individuals and how they bring people in. But the whole point is — once you kind of enter that community, especially nowadays you're going to interface much more with the other services because you're going to probably have more of a joint mission that kind of cuts across service lines.

Donnie’s Tips for Veterans Seeking to Join the Cybersecurity Sector

Ben: Yeah. Thank you for your service. And do you have any tips for other veterans who are looking to get into sort of cybersecurity once they leave the force?

Donnie: No, that's a great question. And I think cybersecurity is a great industry to transition to. So I'd highly encourage those who have an interest in it to kind of explore it. I would say that for those who are actually doing cybersecurity in the military they probably don't need my advice because I think that they'll be pretty well set up to transition into the industry. There's always a need for skilled cyber practitioners. It's in high demand. So guys like that and gals like that — they're already going to have contacts to be able to transition well. But I think from my perspective, I did more of a pivot, like I said, from ground combat arms to cybersecurity. So that's a pretty significant pivot. And for veterans who are trying to do that, it's a little harder but I would say it's important to realize they already have the basic tools and the right instincts. I would say when I first got into the industry through my grad school program I was struck that when I interfaced with cyber professionals they thought like my infantry marines. What I mean by that is their first thought when they see something, when you give something to them, their very first thought is, "I can break that. I can get around that." And I think that's really the essence of a hacker. It's someone who's always going to find a way in and a way around things. So I think that if you kind of double-down on that instinct that we try to teach people in the military, that's a great starting point. And I think that the next piece I'd say is — the military kind of tends to breed generalists. Kind of Jack-and-Jills of all trades where the industry tends to look at specialists. But what the military tends to get right is leadership and people really well. So again, focus on those instincts of getting around obstacles. Focus on people, and then from there, understand that the industry really looks at experience, certifications, and education in that order. While you're still in, start working to improve your education and certification before you leave the service, and that way you can find an entry point in and then kind of grow your plan and kind of come back up.

Donnie: If you're open, you keep learning, you immerse yourself. You're going to do very well in the cybersecurity community. And I think the last thing I would say is just understand is it's a super extensive industry. So just be aware of imposter syndrome. I think everybody in the industry tends to have it. Some of the CSOs and practitioners that I know have it. Just understand that as you get smarter, you get more aware of what you don't know. But understand if you immerse yourself, you're going to keep learning and you're going to get very, very capable, very quickly.

Overview of Xenon and Its Work

Ben: That's great advice. So you've been working as a CSO for Xenon Partners for two years. Can you just tell me a little about what Xenon does?

Donnie: Yeah. Sure. Xenon is a tech private equity team and it focuses and buys and operates B2B SaaS companies. So what's kind of unique about Xenon is — we tend to look for distressed assets. In some cases, that can be a company that's really on the verge of going out of business. It could also be a decent company that's a VC fallen angel. The growth is flat, and it just hasn't gotten the growth that the initial investors were looking at. Sometimes, it's a case where a founder's been running hard and kind of wants to take a break or wants to do something a little different. So not all of them are like — none of them are bad companies. I don't mean that by distressed, but there's something that's distressing them from kind of going to the next level. So we look for those companies that have a solid product, have recurring revenue but may have those gaps in operations and marketing where our expertise can come in and drive growth. One of the things we jokingly say in Xenon is — we kind of refer to ourselves as mad scientists running a pirate ship. Which is kind of a crazy imagery. But the point I like to make is like, "We sail round, we find treasure in unlikely places. We find things that are passed over by more traditional VCNP firms. And then from the crazy standpoint, we tend to have a very high risk tolerance for trying new things. But at the same time, we try to be scientists about it and try to be data-driven to make sure we're making the right calls.

Donnie: So we will do some crazy ideas and do some crazy experiments, but only if we can track the data and make some degree of intelligent judgement on where we're going from there.

Approach to Cybersecurity Upon Acquiring a Startup

Ben: Can you tell me about what's the first thing you do once you've acquired a startup?

Donnie: Yeah. I mean honestly, we try to do this ideally on the security side. We try to hit this pre-close actually. So what we really want to do in those initial conversations with the company we try to frame what the risks are and then really dig into them during the due diligence phase. Even if we don't have a solution, to paraphrase the former Secretary of Defense Donald Rumsfeld, he would say the known knowns. We want to eliminate as many unknown unknowns that exist. In other words, anything that we don't know we don't know. So if we can get a clear picture on what we know and what we don't know, that really helps us kind of frame how we need to approach this. The other piece — as you know, Xenon takes a very hands-on operational approach. So unlike some due diligence phases where you kind of just learn the company and you hand over the keys to start running, we try to do a running hand-off. So during the due diligence phase, we want to work with the founders. We want to slowly integrate our team and start running the business actually so that when the deal fully closes we're actually already running and sprinting. On the security side, we try to get the basics if they're not in place already. We start addressing the critical concerns of the way the company is run from a security standpoint, and then we kind of implement our base cybersecurity plan and start building off the framework. And in some cases, companies are a little more mature in the security range and in those cases, we try to take what they've done and really build off that and just make sure that we're taking off our best practices.

Ben: Okay. Through that sort of transitional phase of operating the company, were you still going through the acquisition stage as far as getting access to all of the systems?

Donnie: Yeah. So to be fair, any due diligence phase is super uncomfortable. It's kind of like a root canal. We try to make it easy, but that's one of the things you try to discuss and have initial framing conversation like, "This is how we approach it. We're going to want to start doing things." So once the diligence phase starts, obviously we sign NDAs and agreements which allow us to kind of exchange information on that, and we try to get access to those things. And most of the time, we try to be super transparent with the founders and the leaders of those companies. And once we explain that and once we've stepped past and signed the LOI, we usually have a pretty good track record or kind of getting access to what we need to and working together. And honestly, if there's stuff that they're uncomfortable with, we work around that. We say, "Okay. Let's box that off. Let's have a conversation about it again." So we know what's there, but if it's something that they don't want to quite fully hand over the keys to that area, we just kind of mark that off as if we can nail down what we think the risks are here and whether we need to be concerned about it, that will be something we kind of hit on post-close.

Security Issues Witnessed from Acquired Startups

Ben: And then once at the close of the acquisition, what's the biggest thing that you often need to fix from a security perspective?

Donnie: I think the biggest thing you see in acquired startups is — and this is an all-in, but I think there tends to be a general lack of knowledge of just basic security hygiene and where the risks lie in the company. And sometimes that comes up when we even talk to the founders and owners and say: "Did you guys realize this was going on?" And we've sometimes stumbled upon things that they didn't even know it was out there. So I think that — and this isn't a slight. In startups, you're highly focused on nailing on the product, you're highly focused on market fit and you're really trying to drive revenue that everyone is looking at you for. So you tend to sometimes to just put off the security stuff unless it's really emergent or tied to your product. So really understand and helping us understand how security fits in the lifecycle of the products and the lifecycle of their customers. It really is no longer something to be a sub-unit of the business, and it can be an existential risk that really touches on everything that a company does. So that's where the biggest thing is. Just what's their level of knowledge and understanding of how it fits in their life cycle and what can we do to make sure we're dialing in on those risks and make sure we're addressing them.

Ben: Yeah. And do you have a standard checklist you sort of go through and —?

Donnie: Yeah. Certainly. So one of the things we tend to do — I mean there's hundreds of frameworks and things out there for cybersecurity. We tend to use Cloud Security Alliance as a consensus assessments initiative questionnaire — the CAIQ. Sometimes people 'cake'. It's a great checklist. I mean there's plenty of checklists and frameworks out there. We like that it maps pretty well across all of the different standards whether SOC 2 or ISO 27001 or some of the privacy regulations like GDPR. So we kind of use that as a checklist very quickly to say, "What are you doing and not doing?" And that's kind of what we base a lot of our cybersecurity plans off of because what we eventually want to do is get them to — even if they're not ready to jump into a third-party audit like a SOC 2 to self-certify to Cloud Security Alliance and get that questionnaire up. So it really helps the conversation with customers who have the security questions with the company.

What a Normal Day Looks Like for a CSO

Ben: So now you've sort of settled in, you've made the transition, what does the normal day-to-day look like as a CSO of this organization?

Donnie: Well, jeez, then I'd say, first of all, if you ask any assistant then they'd probably say there was no such thing as a normal day for a CSO. But I think I can boil down my daily tasks, recurring tasks in a few areas. The first one is kind of just what I call mowing the lawn. It's just looking across the portfolio and the companies in the fund and keeping the hygiene up. Checking the status of training, engaging teammates, touching base with general managers and directors of operations and kind of just ensuring follow-through on our policies. So just maintaining that the stuff we put in place is still in effect and is still moving along and there's no problems with it. Stepping past that, there's a little bit of proactive work where you're constantly kind of keeping your ears open to assess risks across the portfolio. In our weekly meetings, a lot of times we'll do an experiment and we'll try something new and just listen to that and say: "Okay. What are my security concerns when a GM talks about that and then can I have a conversation with him to kind of flesh that out before we go through something that could open a door we're not expecting?"

Ben: Yeah. And do you have any example of something kind of specific?

Donnie: I don't know of any specifics I would want to go into right now, but I think sometimes when we talk about changes to the products or we're going to do a new update or a feature that we want to add it to, sometimes we'll have customers that we'll have conversations with — we really want this feature generally because it's a convenience issue. And that's the trick — is securing convenience is almost two polar opposites and you're trying to kind of run that [inaudible] on that sliding scale there. So a lot of times, you come up with a good feature that customers like and we think we can do, but we don't always think what are the second, third, or the consequences of that feature. So part of my goal is kind of keep my ears open and kind of ask those questions so we can kind of be prepared to kind of close off those gaps before they happen. The other thing I'd say I do is — there's certainly reactive work where you're checking industry alerts, checking the recent CVs that are published, handling patch management across the company. And that's just not only for our company devices but also personal devices. I don't want — even if an employee has a device that doesn't touch the company network — I want to make sure they're aware of patches and concerns for that device because it does no good if our employees have insecure devices that could jump over into our network. And also I just say that's all [inaudible], right? You don't want an employee who is super secure on the business side but then he just makes a small mistake on the personal side and gets his bank account wiped out. That affects morale, that affects teamwork. So we want to make sure we're taking care of them.

Ben: Do you use anything for device management at all?

Donnie: It depends on the company. Some companies are pretty mature on that. All of our companies use G-Suite. So for the ones that are on the low end of the security maturity side, we tend to use the MDM tools that exist in Google Workspace. And then when you get to the higher ones, there are some different products that we'll sometimes use for the more mature companies out there. And I think that the other thing I would say — I'm just thinking of — is we tend to do too, is when you talk about some of the smaller companies, a lot of times they don't have a full security team. And so another piece that I do across Xenon is really work the sales angle where when customers of our companies ask about security, I want to step in and kind of coach our sales team to understand how the security's architected and also a conversation with the customers to understand the use-case but also explain why that company within our portfolio is the right and secure solution for them. So that's kind of the external-facing kind of role where there's a sales side to security, which I think sometimes we forget about, or some CSOs just don't have to deal with that.

When a Startup Needs a CSO

Ben: Yeah. I know it goes just beyond the standard security questionnaire. So obviously having a CSO is kind of a luxury for most startups. At what point would you say a startup would need a CSO?

Donnie: Yeah. That's a great point. First of all, there's a cost associated with that, right? I think that in some ways every startup needs a CSO, but in the early days before you're going to hire someone, I think that getting the cyber hygiene right is key. And one way to do that, one of the first recommendations I say is — just put somebody in charge of security. Put that in the job title of somebody. Maybe it's your CTO, maybe it's a software engineer, but just pick one guy or gal and say like, "Look, you're going to be our security person. Maybe we're going to pay you a little more, maybe we're going to do something else, but we want you to be our security point of contact. We're going to empower you and go across that." And I think the other thing you can do really well especially in a startup role where a lot of times everyone is pitching in together is make security part of everyone's role. One great idea I've seen used in the industry is where one security team realized he couldn't cover all the parts of the company. So he went in and picked an individual with every single business unit in the company and kind of empowered them as kind of the security person and kind of communicated security information to them and kind of worked with their boss. So just kind of creating that culture where security is just thought of in your day-to-day affairs is a huge, huge step in the right direction. As you mature, as you start talking about a broader infrastructure, as you start talking about getting third-party audits like SOC 2, that's when you probably really need to be considering having a CSO on your team. How experienced or what their backgrounds are can depend a little bit on your product and your company. And I think that it's a little a bit of a subject of call, to be frank. If you get a flush of funding early on, then yeah. If you can afford it, jump and get a CSO in early and start building infrastructure early. But if it's something you can't do, put somebody of security and then start building that security culture where people are at least asking those questions and bringing them up in your day-to-day meetings.

Ben: Yeah. I like the point of assigning one person because you know they say the fastest way to starve a horse is to assign two people to feed it.

Donnie: Exactly [laughter]. Exactly. Someone's got to have that as a responsibility, and it's got to be in the job description. It's got to be like, "We all know Ben's our security guy. Until we get more people in, Ben's our security guy." And that really helps.

Ben: It's a very clear — and then like I said, if they have any questions, they can escalate it or sort of talk about it, but at least it's one person.

Donnie: Exactly. Exactly.

TeamPassword and the Problems It Solves

Ben: You've also recently taken on the role as CEO at TeamPassword which is one of the acquired companies. Can you just tell me about this product and what problems it solves?

Donnie: Yeah. So TeamPassword's a password manager and password management’s a pretty critical tool for not just business but I'd say, individuals. It's really a must-have as long as we're going to have passwords. The big benefits I'd say — this isn't just TeamPassword; this is any password manager — is they tend to do things humans are poor at. Generating and remembering long complex unique passwords for every site. You don't want to reuse passwords, and you want to make them long and complex. And we're just not wired to think like that. Password managers safeguard, encrypt and protect your data. They're available to teammates on multiple devices so you don't have to carry books around or notes around or use another insecure shared document. The other nice thing about password managers, I think people sometimes forget, is they help prevent you using passwords in the wrong places and that can also be a final attack against a phishing attack. Let's say a really good phish hits an employee. He or she clicks on that link. They go to a website that's a credential-harvesting website. A password manager's going to notice that that URL and that link may look close to human that it passes muster, but the password manager's going to say, "That's not the right password." It's not going to let you autofill. So things like that are really helpful to have queues to say, "Wait a minute. Maybe I need to dig a little deeper than that."

Donnie: TeamPassword specifically is a small password manager. But what we love about it is it's super intuitive, it's super simple, it's straightforward. So it may not have some of these different bells and whistles you get with some of the larger ones out there. It's great where if you already know password managers, it's super easy to adopt. And I'd say if you've never used a password manager before, it's a great start point because we had a company the other day where, I would say, it took us maybe a 20-minute call to get them online, get their entire team onboarded, organize our groups and then take the old spreadsheet they had with passwords and get that imported securely in a device and get them up and running. And so they were using it really on day one.

Ben: Yeah. So your main competitors are sort of post-it notes and spreadsheets.

Donnie: Yeah. Exactly. It's kind of scary. Especially if you're in the IT industry and cybersecurity, you tend to make the assumption that everyone's using a password manager but the more we talk to customers — I think there've been studies about this that show that less than a third of businesses and individuals are using password managers. So it's really interesting when you talk to them. And I say, "What are you using now to manage passwords, and it's generally not an ideal solution, right? And I've actually told some customers, "Look, even if you don't get with us, just please use a password manager. Please shift on it because it makes everybody safer when you're using secure authentication."

Thoughts on Passwordless Systems

Ben: And there's this kind of movement in the industry for people, on the more cutting edge, to move to passwordless systems. So what's your thoughts on where passwords are sort of going or disappearing?

Donnie: Yeah. So I mean ultimately passwordless is where the industry's headed, and it's where it should be headed. I mean I know that's odd coming from someone who runs a password manager say we need to go away from passwords, right? But from a cybersecurity perspective, passwords are almost always the weakest part of the authentication chain and the weakest link in that process. So I think everyone should be striving for that passwordless solution. There's some challenges with that, and a lot of them go with technical implementation and cost, right? Is if you talk about multifactor authentication — multifactors are generally the factors you talked about — are things you know, which are your passwords; things you have, which are devices or tokens; and things you are, which are like biometric data. A lot of times your second check from a password login is that multifactor like, "Okay, I'm going to send a one-time password to Google Authenticator, your phone, just to give you another check to make sure it's you." When you get into passwordless, you take away the password of that. So you're still using factors of biometrics, using factors of tokens and things like that, and you're using usually some degree of public-key cryptography to authenticate. But what's also interesting is it starts pulling in a lot of behavioral data and metadata, which on the company side usually can be done especially if you have a mature system with login and things like that and all, and you've got a lot of SIEM data, security, and incident data you can draw from.

Donnie: But there's also the privacy concerns that really come with that. And not to go deep into that, but to get passwordless to work well, you're pulling a lot of other information. So for example, a properly architected passwordless solution is going to really achieve a really great balance of security and convenience. To the user, it's going to be super convenient but to the backend, it's going to be super highly secure. So if you think about this, it's like the system is then looking and comparing data. So it says someone just walked through the front door using Donnie's key card. Someone just stepped into Donnie's office using his key card. Someone just logged into the company account with a company machine with a specific MAC address, so we know on the right IP address and Donnie's personal and his business phone are within three feet. We can verify through near-field or Bluetooth of that computer. And there's even crazier stuff out there, Ben. I saw something the other day that looked at you can put sensors in office chairs and people actually sit specific ways and you can identify a person with pretty good accuracy just based off the sensors in the seat. So you start collecting all that data and then you get to a point where one of those things is probably not enough to say, "This is a safe authentication," but when you have 50 or 100 of those little checkmarks, you can pretty confidently say let that person log in. So in a perfect world, Donnie and Ben walk into their offices, they sit down, they open up their computers, and then they just immediately are logged into all the systems out there.

Donnie: And that's really what you want to go to. And I think larger companies who have a lot of enterprise security software and have that login and have mature cybersecurity teams are able to implement that. At a smaller SME level, it's hard to do behind some basic SSO and SAML technologies, but really that's still going back to a single password entry point.

Ben: I guess you do give up the kind of privacy aspect of it to some degree which the severity of the information you have in your organization and everything else you probably opt into.

Donnie: Yeah. You really have to — I mean if you're an employee you're going to opt into it. I think the United States is a much more tolerant thing. I think in Europe with GDPR, it's a little more challenging to figure out how to manage those privacies. What you really have to do is take another step where you're making sure that data is anonymized. So even if you have all these data points you're not tying it to a specific, name, face, email but you can still authenticate that effective cyber persona. You're basically instructing a persona of the employee, but just making sure you're doing it in such a way that's not exposing their PII. But again, that's where I get into really doing that the right way. There are some challenging cost implementation cases right now to do passwordless right. But again, it's definitely the right way to go. I mean take out the weakest link and find other ways that you can be confident that your systems are only being logged into by the right people.

Thoughts on Recent SolarWinds & Exchange Hacks

Ben: It sort of brings us to a good segue from recent password kind of leaks. SolarWinds had SolarWinds123 as their password but then in many ways, this is a limit of the CICD service that they were using. They needed a password.

Donnie: Yeah. Yeah. That's a great point. When you boil it down to it, first of all, everything tends to come with default passwords. Even your home Wi-Fi system, the first thing you should do is change those to something that's secure. If you're using the right password management systems, then you're not going to have employees create something like SolarWinds1234. They can create something random. But you also have to make sure that the system allows for those complex passwords, right? And so there's a lot of — it's very easy to go down — I think SolarWinds it's a single intern who actually set that password as SolarWinds1234, but there's a lot of other things that surround that, that led up to that point where that was even possible.

Recommendations for Security Tools

Ben: Yeah. A few services which can scrape your private or public git repos for secrets that have been leaked. Do you have any recommendations for tools?

Donnie: Not off the top of my head. I can dig up some and hand them to you to maybe post later for your users, but I think there's a lot of open source tools out there that are pretty good. And I think in Xenon sometimes we've written our own scripts to run through and identify IP and other points in there. There's a couple of companies out there too I'm blanking on that actually do that where they will kind of log in and watch your GitHub repositories and just look for keys or passwords or things like that, that shouldn't be out there. But it's certainly worth auditing. Look, if it does need to be a public repository, make it a private repository and secure it. Make sure your GitHub team, only the right people are managing role-based access control and you're using multifactor authentication to those. But if it's got to be a public one, make sure you're actually reading the code to make sure that you're not exposing something on there that you shouldn't be.

Ben: Yeah. And I know because I saw actually PHP Project move to GitHub because their internal git — which is sort of an interesting thing because when you think of these other services, you have to maintain them and support them. So like I said, it's risks of using a third party but also risks of not using it.

Donnie: Yeah. No, that's a great point. That's the inevitable question that I think a lot of companies have when they realize they need something. Do you build on your own, or do you buy it? I mean even companies like Google will oftentimes realize it's cheaper and easier for them to buy it than to build it and just integrate it into their system. The challenge is if you build it, then you've got to maintain it. And I think that's one of Xenon's acquisitions is PackageCloud and that's a key aspect. Look, if you're deploying software packages, you need a way to securely get that out and you can build it. But then you've got to maintain it and make sure you're staying up on it and securing it. Then you need kind of a whole business unit to maintain it. So a lot of times, it's better to use a reputable third-party solution, but you've got to do your due diligence on that too to make sure they're doing the right things. They're a reputable company. You've got to have some degree of vendor management. Know what's out there. I mean I think not to go down the rabbit hole of third-party services, but I think one of the things when I look across our portfolio and started interviewing GMs early on, I found we were using hundreds of third-party things out there and then having to go in and let me investigate one of them and ask the question, "Is this really necessary?" Okay. We have three different companies. They're all using different products that do the same thing. Can we all agree on one and use just one. And try to start lowering your attack surface down and then also track it.

Donnie: So I know if I'm seeing — if I'm talking about that reactive aspect of my job, if I see there's a potential breach or something that pops out there, can I quickly identify which of my companies is using that product and quickly jump into them to lock down any potential vulnerabilities associated with it?

Ben: Marketing departments are very keen on putting random JavaScript in pages everywhere [laughter]. So probably less JavaScript from vendors is better.

Donnie: Right. And I mean I'll give you another example out there. There's a company we were looking at a while ago for Xenon and they sighted several of their customers. And it was funny because one of their customers, I was good friends with the CSO. And I reached out to the CSO and was like, "Hey, what do you think about this product?" And his response was, "We're not using that product." And I'm like, "I think you are." And he's like, "Hold on." When he went in, he's like, "A subset of one of our Salesforce teams was able to put that in their budget and buy it but because of the way they had architected it I just — the CSO didn't even know that they were using a third-party solution out there. And luckily they were doing it the right way. It was secure. There was no issues with it, but it can be very challenging out there when you're empowering some of your businesses to use third-party or open-source software, but there's not a good process to make sure it's vetted or checked so you even know it's even out there. That's what I say you don't want to have unknown unknowns in the cybersecurity industry.

Procurement Process

Ben: Yeah. And so do you have a procurement process? I know this is very enterprise.

Donnie: Yeah. Yeah. That's a good question.

Ben: Often when we sell our software, it goes through our procurement department but that's for a 3,000-person organization. But it kind of seems like you're recommending smaller startups to also kind of go through some sort of initial and background check prior to buying software as well.

Donnie: No, absolutely. I mean — have something. Good as [inaudible] but don't be ignorant. Do something. I mean — I think even if you're just creating a Google Sheet — I think that was how we started out in Xenon is like, "I'm just going to make a Google Sheet that has every product I can find that we're using and I'm going to cross-reference it across companies. I'm going to audit it with the GMs, and then I'm going to take on more from the CSO standpoint of going in and taking a look at those companies and kind of doing the reverse third-party vetting. And mostly these companies are reputable companies and you can go to their security page and pull their SOC 2 certificates or look at what they've done out there and kind of QC that. And I think when you start doing that, you're like, "Okay. I can check off these certifications. I can see how they're approaching this. I can read and get to a level of confidence. And in some cases, if there's more questions, then we reach out to them and say: "Hey, can you give us more information on this?" I've actually had some people — ironically one company reached out as a customer of one of our companies and they said, "Look, I need your compliance certificates." And I said, "Look, this is a very small company. It hasn't done a third-party compliance process yet. Why do you need that?" And the answer that came back was, "Well, we're going through SOC 2 right now so we've got to prove to our auditors that it's secure." I'm like, "Well, you don't need to have —" every one of your vendors doesn't need to be SOC 2 for you to be SOC 2. How do you manage your vendors? How do you manage third-party software?" And they were like, "We don't have a process for that." And it was like, "Well, let me walk you through some things." In this case, we use the CAIQ lite, which is again a quick 75-question list from Cloud Security Alliance.

Donnie: I'm like, "Let me shoot this to you. It's a questionnaire. Take a look at it. See what questions you have. Show this to your SOC 2 auditor. And then, the next step is you probably want to implement this." So it was kind of funny where we had a customer checking our security and we ended up giving them security advice to help move forward. PackageCloud Supply Sidechain Hacks**

Ben: That's a great story. You kind of mentioned the reason you acquired PackageCloud. And I know over the last year or so, supply chain attacks have been on the rise. Can describe sort of a supply chain attack.

Donnie: Yeah. PackageCloud is an interesting one because recently, we talked about a dependency attack done by a white-hat hacker where he found a way to take a look at the software package the software relies on, and started creating similarly named repositories. So when the software went and looked for the private repository, it would see the public one and kind of draw data from there. And it gave a vector to kind of insert malware. So I think that supply chain attacks are complex. They're kind of hard to orchestrate, but they're super dangerous because the one we just talked about is — I may be a super-secure company but if one of those third-party services I'm using gets hacked, that may be an entry point into my company without me ever realizing it. And that was really how SolarWinds played out, right? We talked about the password compromise which allowed the attacker, which pretty commonly now believed to be a nation-state or an advanced persistent threat, advanced because it's a criminal or nation-state enterprise that has a lot of expertise and toolsets persistent that they've been in your system for a while. What they end up doing is compromising the update process for SolarWinds Orin product. So they got an update server. They were able to get into it. They were able to slide malware into the secure legitimate updates that went out. That then deployed out to anyone using that product and that gave the hackers — I would say cybercriminals because they're good hackers out there too, right? It gave the cybercriminals an ability to get into different systems that really had nothing to do with SolarWinds.

Ben: And I guess also everything was signed, and it looked like it was a legit package. I mean it was a legit package.

Donnie: Yeah. It was a legit package. They got it in prior to signing. So once it went through that QA process, it was missed. It was hidden. They had set it so that the QA process was skipping those scripts. And then it was packaged up, secure signed and launched out. And that's very different than a supply chain attack that releases a fake package with a fake signature, tries to do certificate tampering or otherwise get a false certificate in the chain. This is one that was legitimately signed by SolarWinds before it went out.

Ben: So what are sort of some techniques that detect this prior to shipping out the software?

Donnie: Yeah. That's a really hard question to get into because it's super subjective based off the product. But I think the big thing is — really dig into your quality control process. Don't just put that on the side burner or don't subcontract, and don't just rely on simple strips and things to kind of go through to say, "Yup. Everything looks good." You need to take the time to look at the code and just make sure it's doing exactly when you think it's doing and there's not something that you're not extracting, right? And again every way you do that is a little differently but even something where you're kind of hashing sections of the code and comparing that, there's ways you can probably do that and use services to help do that. But I think that you really just got to ask yourself: what is my quality assurance process? Do I have one, first of all? What is it? And then how can I double down on that process to make sure that it's going about in the right fashion and really to flag things that look suspicious before you sign them and send out to your customers.