Offensive Security and the JavaScript Ecosystem with Adam Baldwin

Key topics on Offensive Security and the JavaScript Ecosystem

Auth0 is a platform that provides centralized login and identity for other companies.

The offensive security team at Auth0 is an internal team that is a trusted adversary that attempts to hack the company and then provides a report, which is something that a regular adversary on the internet won’t provide.

Challenges faces as VP of Security at npm were scale and availability — keeping the registry online so that you could get your packages.

Malicious packages on npm were definitely a challenge. The damaging attacks were when an account was actually taken over.

The problem with 2FA is that it wasn’t friendly for publishing.

One security tip for building new applications is having less attackable surface.

The importance of mentorship with Adams story of 

Expanding your knowledge on Offensive Security and the JavaScript Ecosystem

 Welcome to Access Control, a podcast providing practical security advice to start-ups, advice from people who've been there. Each episode we'll interview a leader in that field and learn best practices and practical tips for securing your work. For today's episode, I'll be talking to Adam Baldwin, a.k.a. Evil Packet, Offensive Security, Auth0. Adam was previously VP of Security npm and founder of ^Lift Security, an application and pen testing company focused on the JavaScript ecosystem. Adam is a two-time DEFCON Black Badge holder. Hey, Adam, thanks for joining us today.

 To get started can you tell me how you got into the security industry.

 Well, I'll try to keep it short because it's a relatively long story which started in my youth of course. I grew up in a small farming town in Minnesota with nothing to do and computers and ended up getting in trouble for getting into a local bulletin board system. That sort of sparked my curiosity, but also got me a mentor and really got my interest in security, reverse engineering programming, things like that. That sparked my interest. I didn't really join the industry until much later during my time at Symantec, where I got to meet a lot of the [inaudible], people that sort of guided and mentored me in a way in the offensive side of things. And that's really where I sort of — my career took a turn to specifically the security industry.

 And what were you doing at Symantec at the time?

 So at Symantec, I ended up there doing everything from firewall support to ended up doing consulting. So I ended my time at Symantec doing social engineering penetration testing, those kinds of things.

What It Means to Have an Offensive Security Team

 And then you're currently on the offensive security team at Auth0?

 Can you, sort of, describe to listeners what does it mean for companies to have an offensive team?

 The offensive team at Auth0 — we're an internal team that is a trusted adversary. So we're a friendly adversary, meaning that we're going to attempt to hack the company and then provide you with a report. A regular adversary on the internet is not going to provide you with that thing. So we're that friendly sort of, a goose in the garden causing chaos.

 Yeah. And do you sort of run game days? How do you plan sort of attacking the internal organization?

 It kind of depends. Really, we're there to provide the adversarial perspective. So we're trying to think evil. So as new features are going to come out or somebody might say in product, "Hey, this sounds like a really good idea." And we're going to provide that other side of the coin as, "If we build that, what could we do with it — that could have disastrous effect for our customers, or customer data, their customers, right, or ourselves. Does that open us up to risk?" And sometimes you have disagreement. So sometimes they might say, well, that doesn't seem like a risk or that's theoretical. And we might have to say, well, okay, now we have to prove it. And so, we have to actually attempt to pull off those attacks against the organization to know if they're possible. And sometimes we're going to get caught, and we're going to learn what our capabilities are for detection. And sometimes we're not. And we're going to learn where we can improve.

 Auth0 is a platform that provides centralized login and identity for other companies. Is that right?

 So if people get access to it, they sort of can have the keys to the kingdom.

 Right. So we're, in a sense, taking on a lot of risk for those customers, right? So we're taking on their threat model a bit.

 So we're currently a very small team. There's just a few of us. And so, we're definitely a minority in the organization, even within the security organization.

 How does reporting — do you work up to a CSO or is it within engineering?

 I'm actually in Product of all places, but which seems —

 I guess that makes sense with the early development — it's best to get early feedback on features.

 Right? Yeah. So the traditional pen test role is going to — we're going to want to test features as they come out, right, and things like that. My role isn't your typical red team, which is going to report up, say, to a CSO. So we're testing features, we're really focused on product. But the industry has gone from having, kind of, both that role as well as kind of the red team role which might be a little more of attack simulation or assume breach, other sort of campaigns that go beyond the product.

 Yeah. So you've been working in the node ecosystem for a while, I think, since 2013. What have you seen change over time in the world of node?

 A lot, and not a lot. Yeah, I started with node back in .4, whenever that was way, way back when. We've seen changes. We sort of have three components. I think we have what, Node Core, we have JavaScript the language, and we have the ecosystem. Those are the three sort of major things I think of when I think of a node. We've seen all those evolve, but also stay fairly consistent. The language has evolved. We got new things since then. New operators, new language features. Node — same way. We've got new APIs, new things like that, and the ecosystem has continued to evolve for safety and as well as scale. npm growing up, getting larger and larger and then getting acquired from GitHub to give that some stability. So I think we've seen a lot mature, I think, about it mostly from a security perspective. So I've seen a lot of maturity in terms of release processes. Maturity of the release process from Node Core, they seem to kind of work like clockwork. I feel that team really works hard on those.

Challenges Faced as VP of Security for npm

 You sort of touched on — you worked at npm, and npm was acquired by GitHub. Prior to the acquisition, you were the VP of Security. It's sort of an interesting company, I guess npm, providing hosting packages to other people, including their software. Can you talk about some of the challenges that you faced as a VP of security for npm?

 It was lots of challenges in terms of — I think the biggest was scale. It was availability. Our biggest security problem was keeping the registry online so that you could get your packages. And beyond that, my challenge was the constant conflict between keeping npm safe and keeping all the users of npm safe from themselves.

 And what do you mean by keeping them safe from themselves?

 When you run npm install, it's inherently a dangerous action, right? You're taking code off the internet and you're executing it, right? And people complain about packages. We do that in a lot of other ways, too. It's not just npm install, right? Any time you download something and execute it, right, you're performing a risky action.

 What about malicious packages on npm? Were there any initiatives to help people or stop them?

 Yeah, that was a challenge, right, so malicious packages were definitely at the forefront. So we were trying to do two things. We were trying to keep integrity of packages. So we wanted to keep people, obviously, out of our system so that we could keep packages flowing and keep those packages with high integrity. But of course, anybody can publish anything. So you just create an account and publish a package, evangelize that package and get somebody to install it. It could be, of course, malicious. That's no different than just sending your friend something and telling them to run it, and effectively — we did have efforts we were, at the time that I left, executing all of the packages in a sandbox. And then we were starting to just kind of do analysis on those trying to see when you install them, what files did they touch? Is it something that's different than it was before? And I still haven't seen tools or companies using this data, right? We're still going off of like: does it have a known vulnerability? Basically, technology that we invented, or not invented, but basically technology that we created like, what, eight-plus years ago with the node security project. We're just looking at the bomb, right. We're look at the S-bomb or the building materials. What dependency or known vulnerability? That's what we're looking at. We're not looking at all these other things currently, but we did have some efforts to do that. The thing is — is that wasn't very prevalent. It was a needle in a haystack. You get a piece of malware or a campaign that kind of dropped a few things here or there. Somebody's account get popped and you'd see all their packages get republished with a backdoor or a bug door in it. But it was a drop in the bucket, those publishers, when you're talking about somewhere between 7,000 to 20,000 publishers a day of new packages, we're seeing one, two, a few published here or there. It was a very, very minute piece of the registry.

 And the attacks that were more damaging were not just the random things that could get published, but when an account was actually taken over. So that was really the ones that we worried about. The [inaudible] where an actual package author's account is taken over and then republished. Some of these were republished for a popular package.

 And what did you do to sort of secure these accounts? Do you have any tips for securing them?

 The problem with 2FA is that it wasn't friendly for publishing. So you could either have 2FA and lock down your account but as soon as you wanted CI/CD or 2FA as part of that, it became a real hassle. And I know that they've added some features since then that I'm not super familiar with. But they're working on that problem because that's a huge issue. Right? Automating the publication.

 The publishing of it. Yeah. And then making sure that comes from the signed author instead of the malicious person.

 Right. And all these things we have sort of in Git native, right? We can have signed commits and we can see that the commit came from somebody. We can enforce that with a pull request sort of workflow and things like that. All things as soon as you step out, kind of, we lose that into the publication process. And what exists on GitHub isn't necessarily what got published into npm. We don't have real good tools to do that diffing and we're not looking at those signals. And that was the efforts that we were trying to do right during the acquisition, right, when I was done was make those signals available to developers to look at. So that you could use automation to say, well, merge that or publish it or whatever, unless these are true or install it, unless these are true.

 And I guess that makes it easier for maintainers if they have an external pull request, [inaudible] comes in, this can help protect them. Another project you started was the Node Security Project. Can you tell people a bit more about this project?

 Yeah. So it doesn't technically exist anymore, but it was an effort to bring security as a core value to node. So we saw in the early days of node, if we learned anything from all the other ecosystems that came before us, security was going to be a big deal. We were going to need to know all kinds of different things. We need to have those processes mature.

 People won't seem to see it. They're more serious about getting the applications out, and not necessarily focused on security, which I guess is the tradeoff between sort of user experience and getting things done.

 What are your thoughts on the ease of the language, the security integrity?

 The ease of language is what has brought new developers, right? And I don't know the numbers, but there's some ridiculous number of new JavaScript developers sort of coming to the ecosystem every day. Right? And that's great, that introduces people. It reduces the barrier to entry. It gets people building things. And the thing is, every time I build something, I try to get it working first and then I meet the rest of my requirements. Security was, as far as I know, was never a requirement for the language, right. It was created very hastily, and we've been trying to backpedal all these years to add integrity to the language but that's your trade-off, right? So you have this language that is very approachable, that gives me consistency between the browser and the server. I can just think in one language. But there's a lot of coercion going on. There's a lot of like patterns and there's ways of doing things that aren't well-paved. And so, you end up, I don't know, lost in terms of trying to build on security afterward. I don't actually think that there's — even if you were to prioritize security as the language, you'd still end up with people trying to make it work and then trying to build on security.

Security Tips for New Application Builders

 Building new applications, do you have any advice about how they can, sort of, go about thinking of that sort of top list for security aspects that they sort of address first?

 I hate that this list hasn't changed in ten years. This is what I don't like. And I was thinking about this before the podcast. But using less packages is the starting point, right? So having less attackable surface is the best thing. So if you cannot use third-party components, don't. The problem is that's easy. It's easy to grab something off shelf, somebody else maintaining it, somebody else take care of it and bolt it in. You just bought a pile of risk by npm installing that thing potentially. Starting point is: use less packages. Okay. Now everybody is going to say: "Well, I'm already using a bunch of packages." Well, the second thing is: keep them up to date. That sounds silly. And there's going to be somebody out there who is going to be like, "Yeah, but what about malware?" So use something like the [inaudible] or some automation to keep your dependency updated. Do that, and someone's going to say something about malware. Basically, the amount of malware being produced, if you're just automating those sort of updates, you're not going to have — I think it's a less risky action than [inaudible]. Well, there is a kind of an optimal dwell time for updates, this is a few years old, it used to be something like six weeks. So if you waited six weeks, the likelihood of something being in that package, getting seen by the community is pretty high.

 So there's kind of a sweet spot to kind of like — to lag but that's going to vary from organization to organization. And then finally, I think is: start taking advantage of signals that we haven't been looking at. So when you do update, when you do have these sort of gates is to look at the behaviors of the package and then compare those so dif them. This package should only do X, okay, why is the new version doing Y? The difficulty is, it's not automatable in a really easy way, it's time-intensive for a human and it's boring, it's not interesting. So I think that we'll continue to just sort of put our head in the sand as it comes to third-party packages, [inaudible] not going to save us, having a list of packages, we already have that. It's all the behaviors that are going to be derived from that package or between the two versions. And then, of course, that could be unique for your environment. We've seen targeted malware that only executed in a very specific environment, the Copay incident.

 Basically, it was a piece of malware that would only execute if it was run within a project that had a specific package.json description, which happened to fit their wallet, their cryptocurrency wallet.

 Oh, so it was basically like a cryptocurrency malware that was smart enough to [inaudible].

 Yep. That was attacking the build process to get into the wallet. So it would not — it would effectively not own any other build processes. Yeah, it would only —

 How much of these evil packages do you think are related to cryptocurrency and mining?

 Yeah, the things we saw was people goofing off. You could see your — oh, when you saw us at our MRF/, it removes all your files. That's not really that interesting. And then you would have environment stealers, security researchers, stuff that would just steal environment variables and kind of like pilfer some data. And then you had your targeted stuff or cryptocurrency miners. But the targeted stuff that we saw was cryptocurrency related.

 So if you were — we actually have a lot of startups who are cryptocurrency companies. And if they have a node project, would you have any practical advice for them? Should they use any external packages or should they just try and write everything from scratch?

 I think that the integrity of their build process for their wallets is important enough that they need to be carrying packages from one repository to another and doing the analysis. I think that the risk there again, it depends on your threat model, right? To them that seems like a significant risk, whereas my website doesn't really matter.

Top Tips for Companies That Are Part of the Software Supply Chain

 Yeah, we kind of touched a bit on the software supply chain. Do you have any tips of sort of how to secure that software supply chain?

 Yeah, I think the first the first step is for companies to realize that they have a supply chain problem. Most companies see it. It's an iceberg, right? So you see the little, the couple of packages you're using inside of your package.json, or part of your go [inaudible] or whatever. You see those couple of packages, but it goes deeper. They have dependencies, and so you have to realize that you have a problem and that that problem is going to cost you significantly to solve. There's some tools, there's registries. I don't need to go into the vendor space, but there's vendors that provide alternative registries that you could have your own private registry. And there's open-source versions of those, too. So you have your packages in the bucket. But if you have sort of a NAT, the evil thing, you brought this thing over and dropped it in here and it's just over here. You have to have processes to ensure the integrity in the behavior of those things as you lift them over in your environment. We've gone way too long as an industry wanting security and convenience for free. I don't think we're going to get there. I think that if you want efficiency in this process and have the thousands and thousands of dependencies that a typical project has, it's not going to be efficient. If you want to be secure, it's not going to be efficient. It's not going to be cheap. It it's going to be automatable to a point.

 So you also mentioned earlier — you're on the product team. So if someone starts a new product and instead of bringing up some of these concerns, how do you sort of balance between these sort of pros and cons of both security and sort of convenience for sort of getting the thing shipped and getting it out there?

 It's a business tolerance for risk. You have to play that game. And unfortunately, the longer it all goes without an incident, the more risk taking I've seen an audit get. And I'm mostly speaking from my consulting days.

 Is there any way to quantify the risk? Let's say you're a new person, security team, and you notice all this risk and you go to management, and you're like, "There's all of this risk." How do you sort of quantify that this risk is there and that's sort of a monetary kind of cost [inaudible].

 When the offensive team walks in and hands you your keys, that's when you know, right? So the offensive team or a pen test team or a red team or whatever you want to call it can do that, they can run a simulation. If we happen to compromise this package, where do we get? And running that scenario is going to quantify that, right? Is this possible? It gives you a chance of seeing what that would look like. And you can run that as a tabletop, right? Just sitting down with a couple of engineers, a couple of security people, a manager or a product person and being like, "Let's think about how we would take down the company." Right? How would we do that? And have fun with it. And then say, "Well, that's probably not practical." And you kind of put things into one bucket and then you put the things that are practical in another bucket. And the things that are kind of unknown you put a question mark and maybe you want to get those tested. But that's a that's a good cheap way. Have a half hour conversation about what would happen if somebody compromised one of these packages. And you're going to quickly see, "Oh, they're going to steal all these environment variables, once they had those environment variables now, what does that give them? Oh, they can get into the server. Okay, once they have access to that. What does that give them?" And you can kind of see how you're going to — that tree form.

 And then you can have initiatives to either address the strengthening of which part of the process is sort of lacking, right?

 Yeah. And that point, like what you actually do to remediate it varies drastically from org to org. But that's going to be one way of uncovering how that sort of snowball is going to run through the org.

Advice for Those on the Defense Side of Security

 And so, what advice would you give someone on the defensive side trying to make your job harder on the offensive side?

 Go back to those the things we talked about with the [inaudible] security supply chain. But things that that cause us problems on the human side of it. So if you look at the Verizon data breach report, social engineering is at the top, right? So technically, trolls networks are getting tighter and tighter from the outside, and so we're having to come in through creative ways, social ways. Challenging something, having a protocol or practice for verifying identities, things like that that's going to stop us. Passwords — don't reuse your passwords. I hate that I'm still having to sort of say this, but it's easy to reuse passwords. But don't do it, right? Especially between your personal stuff and your work environment, absolutely not. Password saves, password, whatever. 2FA, of course, is great but you have to weigh that against convenience. I use keys where I can, so if something has a key, I use it. I enroll back up keys and then keeping things updated. It makes things harder for me if you have an environment with up-to-date software. That said we still get in because of configuration issues or mistakes or an emergency procedure, let's say opened up something temporarily and it never got permanently fixed. Those to-do's pile up, those to-do's create risk. So, from the defensive side, keeping track of all those things that you're worried about on a risk register, something like that, just to keep track of and prioritize.

 Whose sort of responsibility, would you say, it is to sort of go and tidy up and fix all of these open issues?

 On the one hand, it's like it's everyone's responsibility. Security is a spread load across the organization. While you're in there fixing up a test, if you can you see something out of date or whatever you can, you can update those or whatever.

 Just making it part of the current practice to always be on the lookout?

 It's our responsibility. But then you have to have a verification process. And that's again where you're auditing your compliance, your offensive teams. All the, sort of, checks and balances come in to be like, "Are we doing the thing that we say we're going to do?" Right? Are we doing those things? And unfortunately, we're human, we're fallible. We may not have malicious intent, but we may just use a crappy password because it was convenient. My work laptop wasn't working, so I wrote it over here. I did it on mobile, so I just typed in a bad password and then that password persisted. Or maybe we just copied it or screenshot it and put it into Slack. We never deleted that screenshot, but it's still sitting there.

 Yeah, I'm always recording product demos and they always have both the QR codes and the tokens, and it's a very quick cleanup procedure afterwards. It's always easier to create infrastructure and tokens than it is to remove it.

 And another thing that that I'm going to bring up just briefly, it may not be on this topic for this discussion, but I've noticed a lot of tools focusing on looking for secrets and sensitive data in Git repositories. And like I mentioned earlier, what exists on Git and what exists in npm often differ. And so, I've noticed a lot of packages leaking secrets to npm. So you'll get to publish and there's a .env file in that thing from testing. And it may be nothing or it may be something. So keeping track of the artifacts that you publish to the public, such as any of those packages, [inaudible] npm go [inaudible] images, those kinds of things. Doing a quick run through of those are going to limit any sensitive data that I might have as an anonymous external attacker, that's going to make my job harder as well.

 Actually, I saw that GitHub now has better exec UX and tags on the API tokens. So if you see an API token, you know what the token says. If whether it's like a read or write or it's for the org or the repo, which I feel definitely the end user experience for creating tokens could be more obvious.

 Yeah, I think that that was a very good move on their part. At the same time adversaries have better patterns now to find or validate secrets. But so, do defenders, right? You can go through and profile all the secrets you're using in production and then build regexs even if you don't have a fancy tool and look for them in Slack and npm and GitHub, that's something that's possible. I guarantee you, if you go look, you're going to find something.

 Do you have any tips for many of these sort of secrets of sort of third parties? And often let's say using an email provider, you might create an email provider token, but if is a long-left token. Do you have any advice for sort of how to rotate and manage and sort of keep track of these sort of third-party tokens which sort of connect to cloud services?

 No. I'm usually the one that's causing the token to refresh, I'm not on the other end having to do that. The only thing I will say is make sure that that rotation is done, and that you don't try to take the easy route, which is unpublish the artifact. Once something's published in npm it's public and there's 100 mirrors that just took it. It's out there, make sure that that rotation —

Ben Arent chats with Adam Baldwin about what offensive security means and other security challenges.