Lessons From Billions of Breached Records - overview

Security flaws, hackers and data breaches are the new normal. It’s not just those of us in the industry facing these foes every single day; it’s everyone. Whether you’re online or offline, you simply cannot exist today without your personal information being digitized in systems which are often left vulnerable and exploited at the whim of attackers. But who are these people — the ones who seek to break through our defenses and exploit our data? And how are they continually so effective at doing so, despite our best efforts?

In this talk, you’ll hear from the creator of “Have I Been Pwned” about the lessons he’s learned after processing more than 11B records of breached data. You’ll get a glimpse behind the scenes of what caused some of these devastating incidents and how they continue to wreak havoc today, despite how much more aware the industry is becoming. It’s a frightening, eye-opening and entertaining look at infosec and data breaches.

Talk breakdown:

  • 0:00 Introduction
  • 1:50 Lessons From Billions of Breached Records
  • 36:14 Q&A
  • 56:31 Join us on March 15 with Michael Coates.

Learn more:

Key topics on Lessons From Billions of Breached Records

  • People will sign up to anything with their gov email address or their work email address.
  • Have I Been Pwned, a website launched by Troy Hunt on December 4, 2013, allows Internet users to check whether their personal data has been compromised by data breaches.
  • Have I Been Pwned has some 11.7 billion pwned accounts. The complete corpus of data is something much, much, much bigger than that.
  • Invited as a security expert, Troy Hunt testified before the US Congress in 2017 about the impact of data breaches.
  • With regard to hackers, the representation of who it is that we’re concerned about is often very different to the reality.
  • Cyber bureaucracy has driven us down a path of very bad practices around the way we manage our passwords.
  • A lot is changing within the organization, and our traditional views of perimeter security are just completely falling apart.

Expanding your knowledge on Lessons From Billions of Breached Records

Introduction - Lessons From Billions of Breached Records

(The transcript of the session)

Ev: 00:00:02.100 [music] All right. Good afternoon, everyone again. So we have plenty of people who have already joined. So welcome, everyone, to our Security Visionary Speaker series. This webinar is sponsored by Teleport, the easiest and most secure way to access computing infrastructure. And today we have Troy Hunt. Troy is a software guru and a security researcher. He’s been advising and educating intellectually curious people like yourself on information security and data breaches worldwide. By the way, Troy is famous for his data breach notification service called Have I Been Pwned. As you will be listening to him, and he’s going to be talking about his lessons from billions of breached records, to kick this off, Troy, if you think about all these lessons you’re going to be sharing with us, what was the most may be surprising or most disturbing thing that was kind of your personal takeaway. What have you learned about the world by looking into all of these breached records?

Trust and information privacy

Troy: 00:01:03.235 So part of it is that I think people will sign up to absolutely anything online. It has just shocked and amazed me the things that people put their email address into. And then the other part that’s really surprised me is people will sign up to anything with their gov email address or their work email address or their Gmail email address. That has shocked me. And I’m happy to answer more questions about the weirdest things I’ve seen about that later on as well.

Ev: 00:01:29.527 So it looks like we’re just trusting people. Humans, by default, we always give each other the benefit of — the principle of charity, we like to apply it?

Troy: 00:01:38.587 Yes. And I’m reticent to sort of refer to that in a negative way as well because isn’t it sad that you can’t just go to a website and sign up and not expect your data to be preserved? That’s a bit of a sad state of affairs. Thank you, everyone, for joining us here on the internet as we continue to do everything on the internet as opposed to in-person. And as Ev mentioned, I want to talk about data breaches today. And I was doing variations of this talk for many years and it used to be lessons I’ve learned from one billion breached records. And then it was lessons I learned from two billion breached records. And then I got to the point where my talks couldn’t keep up with the amount of data that was actually flowing into this service. So I had to get a little bit more generic. I figure this will give me a good runway in terms of how long I can keep talking about the scale of data breaches.

What Have I Been Pwned shows

Troy: 00:02:27.839 But to start putting it in context, Ev mentioned Have I Been Pwned. So this is as it looks this week — I think I added one more since I’ve prepared all these slides — 582 different websites or other services. Sometimes it’s not just websites. Sometimes it’s IoT things which are often just sitting in front of websites. I’ll talk more about those today. But 11.7 billion pwned accounts. So 11.7 instances of someone’s personal data being exposed in a breach that I have. And if you think about this, as though it’s sort of the proverbial tip of the iceberg. So what’s in Have I Been Pwned is just what has been breached, what has been leaked, what has been available for me to download or someone to send to me. And the complete corpus of data is something much, much, much bigger than that. None of us really know.

Troy: 00:03:22.305 Of that 11.7 billion, about 25 of them are me. [laughter] So I’ve been in 25 data breaches myself. I can’t stop myself from being in data breaches. I don’t like getting emails from me when I load data into Have I Been Pwned, but it happens and it will continue to happen. So I thought I’d talk a little bit about where this service has come from and some of the things that I’ve seen firsthand. And I like to sort of start to talk about the beginning, which is now more than eight years ago. December 4, 2013, I launched Have I Been Pwned. And when I started up, I thought, “Oh, this will be fun. It’ll be something my mates will use.” Not everyone was convinced it was going to be legitimate. They thought I was going to sort of mount an email harvesting campaign. Which I always thought was funny because look, I started with about 155 million records anyway. So if I want email addresses, I’ve got plenty of email addresses.

Testifying before the US Congress

Troy: 00:04:20.578 But what I found really fascinating about the service is just how much things have escalated. Not just in terms of the volume of data, but the opportunities and the exposure it’s given me. And as I’ve sort of gone through running this service, there’s been all of these points along the way where I’m just like, “How on Earth did I get here?” And I guess the penultimate example of that was when I ended up here in Congress, which was very surprising for me. [laughter] So when they reached out, they’re like, “Hey, it’s Congress. Would you like to come to America and testify?” And I’m like, “You know I’m Australian, don’t you? Are you really sure?” And they’re like, “Yeah. Yeah. It’ll be cool. Come.” So I went, “All right. That’s good.” But I had to get dressed up. I had to get dressed up for you today. I literally live on a beach. I had to put on a shirt. [laughter] So this is what — this is a T-shirt. I call it a shirt. So this is about as fancy as it gets. So I had to go and buy every piece of clothing in that photo.

Troy: 00:05:19.977 Definitely the suit, the shirt, the tie, even the socks. I didn’t really have any socks that were suitable for going to Congress. And I didn’t want to take it all too seriously because you can see the guys next to me look very, very serious. So I put out a tweet, and I said, “Hey, has anyone got some socks they want to send?” Because socks are a cool thing these days in terms of InfoSec companies and swag. And I got some cool socks that I ended up wearing to Congress. And just to show you what those socks look like, they say this — they came from Sophos — “For those about to code, we salute you.” And I thought, “This will be fun. This will be my little bit of lightheartedness amongst a very serious situation.” And I guess sort of the most impactful example I can give of just how much things escalated from that first little tweet and that first little Genesis of an idea of a data breach service not only did I end up in Congress wearing these socks, but it got so weird that you can now buy a stock photo of my socks from Shutterstock for $199. [laughter] So I keep looking at this going, “How did I end up here? This is strange.”

The perception vs. reality of hacker identity

Troy: 00:06:26.685 So all I wanted to do at Congress was talk about hackers. And really, this is what’s key to the data that flows into Have I Been Pwned and what’s key to the organizations that are working in this industry to try and keep the hackers at bay. And I do like to give a bit of context about who these folks are. Who are the hackers? Who are we protecting our things from? So I thought I better make sure that everyone understands exactly who the adversary is. I went and I found hackers. This is what hackers look like. You’re probably familiar with these folks. Probably familiar because you see this imagery all the time. We see it on news articles all the time. We see it from InfoSec companies all the time. And we see this same theme. It’s always the hoodies. The green screen. Hackers love green. We see that all over the place. Binary. [laughter] Lots of binary. And you sort of look at this imagery and go, “Why? Why do we need to see this?” Because this is very sort of predictable kind of pictures, right? So we don’t need to see this in order to understand there’s actually someone else behind it. And this is what I sort of want to make a point about. The representation of who it is that we’re concerned about is often very different to the reality.

The TalkTalk data breach

Troy: 00:07:48.731 Now, let me give you an example of precisely what I mean by that. So I’m going to take a data breach that was very noteworthy. It’s TalkTalk. Now, TalkTalk is a very large British telco that had a massive data breach in 2015. And this breach was so large that they said it did them £77 million worth of damage, which is enormous by any standard. It’s hard to think of many other data breaches that have an impact of that scale. Now, when this happened and there was news about it, there’s always a lot of commentary. There’s always experts providing commentary. And one expert left some advice around who they thought was responsible for the breach. They said, “Looking at all the data available to us, we believe it was Russian Islamic cyber jihadis.” This was the term used to describe those responsible for the breach. When we think about the images we just saw before, this elicits the same emotion. This sounds scary, just the way these words are strung together. It also sounds kind of ridiculous in this context. And I’ve done this enough times live to know there’s people looking at this laughing because it is laughable. It’s a crazy idea. And what’s particularly crazy about it is how inconsistent it is with the reality of who is often responsible for these breaches.

Troy: 00:09:15.164 This is the guy who mounted the TalkTalk data breach. Now, it’s not clear whether he’s wearing a hoodie or not. He’s leaving court here. His face is obfuscated because he’s a child. It’s a 17-year-old boy. And what I find most fascinating about this is not just the fact that clearly this reality doesn’t align with the imagery and the representation of who’s responsible for these cyber-attacks, but it’s also the fact that a child, legally a child, using free software downloaded off the web — he used SQLMAP. He mounted a SQL injection attack — can do £77 million worth of damage. And that is just enormous leverage. And for those of us defending systems, this is what we have to deal with. The fact that a very inexperienced, ill-equipped adversary can do huge amounts of damage with very, very little tooling. They do take things very seriously in the UK, though, when you mount a cyber-attack like this. They took his iPhone away, which is a really big thing. I imagine for a 17-year-old kid. Taking your iPhone away it’s like, “I’m basically in jail. [laughter] My life is gone.”

Passwords: yesterday and today

Troy: 00:10:26.139 Now, one of the things that worries us most with data breaches in particular is the impact on passwords. So I thought we’d spend a little bit of time talking about passwords and where they’ve come from and how the cyber bureaucracy, let’s call it cyber bureaucracy — I don’t think I’ve used that term before — has driven us down a path of very bad practices around the way we manage our passwords. This is an image of MIT in the ‘60s and it’s believed to be the first-ever instance of a password on a computer system. This is their Compatible Time-Sharing System. And it’s fascinating to think about what the environment looked like back then. So this guy is here in a room of computer. The computer is just the entire room. It’s got tapes and I don’t know if people have punch cards or things there. If I got into the room, I wouldn’t know how to use the computer. And this was one of the first, I guess, poignant observations about passwords back in the ‘60s. If someone else knew your password, what are they going to do with it? They’ve got to have physical presence because they’ve got to get into the room and then they’ve got to have sophisticated knowledge because they’ve got to know how to actually use it on this computer system.

Troy: 00:11:40.538 So passwords were very different in terms of the risk and the impact when we go back 60 years. But in their essence, they work the same then as what they do now. And it’s very, very simple. Think of it this way. A password works by you having two strings in your head. One is a username; one is a password. If those two strings match the two strings stored on the computer system, whether it’s the room of computer here or a modern-day web server, if they match, then you log in, and that’s that. Very, very, very simple model. This is why passwords have lived so long because they are such a simple model to implement. So back in the ‘60s, we had passwords, but you could get away with taking a lot of shortcuts. You could use your dog’s name because we wouldn’t have Facebook for another 40 years, so no one knew what it was. Unless they were your friend and then they probably went to break into your account. 20 years later, though, things start to change because 20 years later, we start getting computers in the home. And not just computers in the home, but computers that can remotely access other computers.

Troy: 00:12:47.787 One of those early examples is what’s in the image just here. This is a Prestel. And the Prestel was launched back in the mid-80s, rolled out to about 90,000 subscribers. You can see they’ve got a telephone on the desk there. It had one of those classic acoustic couplers where you could dial into a service somewhere and log on to your account. There’s a video clip here that I’ve just cut down, zoomed in, slow mode enhanced, where this guy logs on to his account. Now, as he logs on to his account, watch closely in the video. See if you can figure out what password he’s using to log on to this Prestel service. All right. Everyone get that? I like it. It's just so much mileage out this video. I think what’s amazing about that is this is the ‘80s, right? So now we’re talking about 40 years later, I still see that password everywhere where someone can actually get away with using it. And we’ll talk about the places you can’t in a moment.

Password predictability

Troy: 00:13:50.204 Now think about the ‘80s compared to the ‘60s. We’ve now got 90,000 people using a device in their home remotely accessing a system. So passwords like that now pose a risk, which they didn’t pose 20 years earlier back at MIT. But we’re really only just getting started. Let’s go another couple of decades forward. Let’s go into particularly the noughties, 2000-plus, where we start getting not just millions of websites, but billions of people using them. And we looked at examples like the Prestel guys password, and we said, “We need to do better. We need to make passwords stronger, otherwise other people will figure them out.” And the way we decided to do this as an industry was to introduce arbitrary password complexity criteria.

Troy: 00:14:38.262 So here’s how arbitrary complexity criteria works. I got a bit of footage here of Bob. Bob’s trying to register on a website. And you know what it’s like when you register on a website. You’ve got to come up with a password. So the websites saying, “Look, Bob, your password needs to have at least one uppercase character, at least one lowercase character. It’s got to have at least one number. It’s got to have at least one non-Alpha numeric character. It’s got to be at least eight characters long, and it can’t be any one of the last three passwords you’ve used. Plus, it also can’t contain the company name.” And Bob is losing his freaking mind because now he just can’t sign up on the website. And we’ve all been there, right? We have all been the Bob where we’re trying to just create an account on the website. And what’s even worse, in my opinion than that, is when I try and generate one from my password manager and the website comes back and says, “That’s not allowed. It’s got an apostrophe.” And I just go, “Well —“ or, “That’s not allowed. It’s more than 20 characters.” Thank you, PayPal. So we’ve got all of these really odd, arbitrary rules that drive humans nuts. And they also drive us to very predictable, weak patterns.

Troy: 00:15:45.646 Now, when I’ve done this talk in front of a live audience and I can see people and I can see their reactions, I’ll say to them, “Imagine you go to a website and you want to use the same terrible lowercase character password that you use everywhere. And the website says you’ve got to have at least one uppercase character.” What do you do? And everyone knows what they do, and they know it’s bad, so they don’t want to answer. And everyone’s looking around just waiting for the first person to say it. And eventually, one person’s like, “You capitalize the first letter.” And everyone laughs a little bit nervously because they’re like, “Oh, now, the hackers know. They’ve worked it out. Someone’s going to get my password.” But it’s funny because everybody does it and yet we somehow think that this actually improves security.

Troy: 00:16:28.781 What happens is we end up with passwords like this. And this is a good password by empirical measures. Uppercase, lowercase, number, non-alphanumeric. Also, 15 characters long, which is pretty good as far as a password goes. But your human brain is looking at this going, “I see words. I see words. I see character substitution. This, to me, looks to be very predictable.” So the problem is that the way we’ve been thinking about password strength did become very academic, and in many cases, very bureaucratic. This was just a compliance manager somewhere, “You got to have uppercase, lowercase, and so on and so forth.” And we’d end up with things like this. And then to make it worse. We said, “Well, a hacker might get your password.” What if you enter into a phishing site? So because a hacker might get your password, you must change it every 90 days. And I’ll ask the audience again, “Okay. So what do you do after 90 days?” And everyone’s looking around nervously again — and you all know this already — eventually, someone says, “You just change the number on the end.” And again, everyone laughs nervously. And they’re like, “Oh, they’ve figured it out now.”

Troy: 00:17:38.112 This is actually great because it’s a very scalable model. You can just keep adding one, adding one, keep going, keep going. What you can do is you can take the number on the end of your password right now, divide it by four, and that’s how long you been at the company. Do the math. It works. I got up to 20 something in my last company. We laugh about this, but it’s a problem. It’s a real problem. And a combination of arbitrary complexity criteria and mandated rotation lead us to choosing weak passwords. It’s either choosing weak passwords or then having trouble remembering them because most people still try to remember their passwords rather than to use a password manager. And we see all of this evidence of that manifesting itself.

Troy: 00:18:22.110 A good example here. Remember, a few years ago, there was that misfire of the Hawaiian emergency response system, where everyone got the sudden push notifications to their phone, which said, “Look, there’s missiles coming. Take shelter.” It was very, very stressful for everyone. Afterwards, they go and interview this guy standing here in his SOC where he manages all of these notifications. And there’s something alarming on his screen down here. They were using Internet Explorer. Oh, also, he posted his password on a post-it note onto the screen, which is what people do when they have trouble remembering passwords. So here’s our change. We’re moving from this very academic, bureaucratic view of what creates a strong password, and we’re moving to much more practical methodology. And the guidance we’re getting from very formal, authoritative institutions is reflecting that.

Troy: 00:19:21.614 So, for example, NIST a few years ago said, “Verifiers should not impose composition rules to memorize secrets. Don’t force upper case, lower case.” What if I want to have a passphrase? It’s four words. It’s all lower case. They’re random words. By any dictionary definition, it’s very, very strong. Don’t impose a composition rule. The NCSC in the UK — another great body that provides lots of really practical advice — says, “You should only ask users to change their passwords on indication of suspicion of compromise.” Don’t force them to change the password. Could you think about the logic of it, right? Imagine a hacker gets your password. They’ve got your password. Do you think they sort of say, “Well, I’m kind of busy. Life is hectic at home. I’ve got to look after the family. Got some chores to do. I’ll get around to using it in 90 days from now,” or do they take it and just use it straight away?

Troy: 00:20:21.413 To be fair, though, this is not all about just throwing out things that we held dear for so long. I don’t know that any of us really held a dear, but it was something that we just did for years and years and years. It’s not all just about throwing that out. It’s about recognizing that the landscape has changed, the risks are different, and we also have other mitigating controls now. Really good example. We have near-ubiquitous transport-layer security now. Now when I say near-ubiquitous, we’ve got 80% plus of all requests over the web are over HTTPS issued portion of internal network traffic inside organizations is encrypted. So where we had risks at intermediary points before, switches, routers, things like that, encrypted traffic now. That risk goes away. We’ve also got great alternate controls for things like authentication. We don’t just have to compare two strings anymore. These days we’ve got things like user behavioral analytics. Bob normally goes into work and he fires up his computer, he logs on, and he does his work on the sales Excel spreadsheet. One day, Bob logs in from Beijing, pulls down five gigabytes worth of data. Maybe it’s not Bob. We can be much more intelligent about the way we do authentication.

Perimeter security

Troy: 00:21:34.424 A lot is changing within the organization and particularly these classic bureaucratic views of security. I’ll give you another good example. I took this in the Australian outback last year. And when I saw it, I was like, “This is going to be a really, really good slide for talking about perimeter security one day.” Where is the perimeter of the organization? Because somehow this gate is meant to keep something on one side and out of the other side. It’s not entirely clear to me what it is. But as I was walking around the desert in the 40-degree Celsius heat with all the flies, I was thinking, “It’s like the good guys are over there and the bad guys are over here and somehow this gate is going to keep everything separated.” So I thought I’d give you two examples of where our traditional views of perimeter security are just completely falling apart. And those two examples are Zoom, everyone’s very familiar with Zoom, and CloudPets Teddy Bears, which I’m going to come back to in just a moment. But look, let’s touch on the Zoom thing. Because one of the things that’s been fascinating about the entire Pandemic situation is just how rapidly we have moved to being reliant on external services, services outside the corporate domain, outside our walled garden, outside our firewall, and suddenly all the things that we used to do internally within the network started flooding externally.

Troy: 00:23:02.062 Everyone would have seen many examples of Zoom bombing over the last couple of years. Some of it kind of funny, if I’m honest. Some of it very damaging as well. Corporate secrets being discussed effectively in the public domain because people don’t know how to use the tooling. So suddenly this idea of all of the good stuff being on the inside and the bad stuff being on the outside completely falls apart. Now, this is at the end of a very long train of this happening as well. We got floppy drives. Floppy drives could take stuff external. We got USB sticks. USB sticks could take a lot of stuff external. We got cloud. A lot of our things started going external. Many organizations use all sorts of cloud services which are hosted by third parties. But the bit that I find most interesting is the things that we brought the other way on the inside. This is not just about the good stuff going out, it’s about the bad stuff potentially coming in. How many of you worked in companies where they glued up USB slots on devices because they’re worried about people bringing something in from outside? Or who can remember being in an organization when iPhone started coming out and there was always someone who is a manager that wants to join their iPhone to the network, and they were saying, well, hang on a second. Like this is an untrusted device coming into our beautiful, pristine network. So we’re worried about the introduction of externalities into our environment." And this perimeter keeps eroding and eroding and eroding. This is what’s brought us to talking about things like zero trust. Well, why don’t we start taking the view that, instead of all the good stuff being that side and the bad stuff being this other side, that it could actually be a bit of a mix?

Troy: 00:24:45.101 Now I promised to talk about the teddy bears because it’s something I had firsthand experience with, teddy bears. And it’s also a really good example of how we introduce risk into an environment that didn’t have them before. So these are CloudPets. These are Messages You Can Hug. [laughter] That was their tagline. And what they are, in essence, is a teddy bear with a microprocessor that has a little flashing light. You can see the light on the chest. A little button in the paw. You can just see a little symbol on the unicorn there. It also has a speaker and a microphone. So it is a listening device that you put in your children’s beds. This is the way they used to represent it. This is the old website. And the bit that I always just love with this is the promotion to active-duty military. It’s like, “Here’s a listing device. Let us ship it to you really, really cheap and you can put it in your home. It’s going to be fine. Trust me. Kids will love it.”

Troy: 00:25:46.107 The value proposition of the CloudPet was that it would connect to your mobile device and your mobile device was then connected to the internet. So what it meant is that if your child — let’s say they want to speak to a grandparent on the other side of the world. And phones are boring. So your child holds down the poor talks into the microphone of the CloudPet and that goes to the parents’ phone and then up into the cloud. Remember that’s where all of our private data’s now going. And then the grandparent also pulls down that recording and listens to it on their computer. And then it goes back the other way and the light flashes when they have a new message. Kind of cute. Nice idea until all their data got ransomed. So they had all their data in a MongoDB. It was left publicly facing. Someone went and deleted all the data and left this message. “Your DB is backed up on our servers.” So basically, it’s a backup strategy. But the data was deleted from the MongoDB. So all of the teddy bears just died. Metaphorically died. Don’t worry kids. The teddy bears were fine. But they couldn’t talk anymore. “Send one Bitcoin to this address, then send your IP address to this email address.”

Troy: 00:26:58.768 Now when this happened, someone sent me all the data. And I was trying to establish the chronology of what actually happened, so I went back through the Shodan logs. If you’ve never seen Shodan before, it’s a search engine for the Internet of Things. And I found this message, and I found two other messages where people have been found the same open MongoDB and just change the Bitcoin address to their own, which I thought was kind of ingenious because you haven’t done the actual crime, but you’re going to benefit from some of the proceeds of it. So these teddy bears are being brought home and put inside networks on the inside of the network. Yet they have fundamental flaws and they’re collecting data which then goes externally out to the cloud. And we come back to this whole concept again of, where’s the boundary anymore? Where’s the perimeter? There is no definition of the perimeter anymore, not the way it was before. So we’ve got to come back to this concept of zero trust. We’ve got to expect that everything’s going to be bad.

The increasing number of connected devices

Troy: 00:27:54.667 It’s an interesting thought exercise, not just for your organization, but even for your home. How many things in your home that are connected? Mine is crazy. I’ve got all these IoT things and stuff like that everywhere. I’ve got so many connected things. And quite often I will look around my network and I’ll go, “Okay. Well, what if one of these is rogue? What could it do? What’s the worst that can happen?” Because I’ve got to work on the assumption that I cannot trust any of them. Part of the reason that we need to keep thinking this way is because we are connecting everything possible, including children’s tracking watches. So let’s talk about these. I like to talk about examples I’ve had firsthand experience with and give a little bit of insight into what’s actually happening behind the scenes. So there is this movement of putting tracking watches on children. And often the parents that do this are referred to as helicopter parents. So parents that are so worried their kids are going to be kidnapped or run away or something like this, they need to know where they are at every single moment.

Troy: 00:28:54.065 So they go and get watches like the ones you see just here. Now, there was a long history of these watches having really egregiously bad security flaws. And a friend of mine is a guy named Ken Munro. He runs a penetration testing company in the UK called Pen Test Partners. And Ken had been very good at finding really serious flaws in these watches. And a couple of years ago he popped up and said, “Look, there’s a new company called TicTocTrack that’s selling a watch. They’re Australian-owned and operated. You’re Australian. Why don’t you go and get one of these watches and we’ll just pull it apart and see what happens?” So I thought, “Okay. That sounds fun. Let’s do that.” So I went to the TicTocTrack website, and I knew it was going to be secure because it had a padlock. I literally wrote a blog post yesterday about how imagery like padlocks and seals and things like that mean absolutely nothing and they really do nothing for you.

Troy: 00:29:51.863 So I’m reading the description of it. TicTocTrack’s software is custom built and securely hosted in Brisbane, Queensland." That’s about an hour that way from me. I am a very proud Queenslander. We have the Great Barrier Reef. We’ve got the deserts. We’ve got beaches. We’ve got awesome stuff here. But I don’t for a moment sit here and think, “Gee, we do data security.” Like, “This is our thing, data security.” We’re not Silicon Valley. This is not that. And what we actually found is this organization was putting a lot of emphasis on the Aussiness of the service because somehow that would give parents more confidence their data could be trusted. I like the start of the byline. “We take the security of your data seriously.” The reason I find this funny is that normally when I see an organization say this, it’s right up the beginning of their data breach notice. We take the security of your data seriously. By the way, we lost your data. No. Wrong term. By the way, there are now a lot of backups of your data. [laughter] It’s like the MongoDB thing with the CloudPets before. I’ve written a blog post about this. Every time I see one of these data breach notices, it’s like, “We take the security of your data seriously, however —“ so I’m reading this and I’m thinking, “You’d be crazy to have one of these and actually put it on your children.” So anyway, I went and bought one and I put it on my child. This is Elle. This is my daughter. And she was about six years old at the time. And you’ll see, it actually says Gator. So the Gator watch is made by a company in China, which is fine. I’ve got a watch made in China. It’s just got an Apple logo on it. That’s not the problem. The thing is that they build the hardware and then they ship it out and then someone else creates all the software. So they create the APIs that the watch talks to and they create the mobile app as well.

Troy: 00:31:44.084 So we got this watch and Ken and I started having a look at how the watch and the mobile app communicate with the backend services. And we found an API. Looks like this. Now, the whole point of this API was to pull back information about the family. You can see it goes, “API users filter equals family identifier 3497.” Now, this number was very important because this was the Identifier on my family. Now, just to give you an example of the sophistication required in order to find a serious vulnerability in this, it worked like this. Ken’s a smart guy. Ken can count. Therefore, Ken can hack. So he saw this number, and he’s like, “I wonder what would happen if I misused one off this number?” And we found the last subscriber. And that was all it took. We took one off the number. And this is the sort of data that came back. This is obviously my data because you can see me all over it. But by subtracting one off that number, we were able to find other subscribers of the service and their personal identifiable information.

Troy: 00:32:55.163 Many interesting/ entertaining things happened as we went through this disclosure process. The company actually did a pretty good job in so far as they took the service offline immediately when we reported the vulnerability. But clearly, this was not an organization which was ready to deal with a security incident. They hadn’t thought this through. It was a very small shop run by a mom with no technical experience but wanted to build a device that tracked the kids. During the process, reporters spoke to her and said, “We’d really like to know how many Aussie families have been impacted by this incident.” Reporters always want to know that. How many Aussies? It’s like, “You know there are other people out there possibly impacted by this too.” And the CEO said, “That’s commercial in confidence. We don’t share that information.” And then Ken and I are just like looking at this API request going, “I reckon I know how many Aussies are out there who probably have one of these.” So yeah, that’s another tip. If you’ve got commercially sensitive data, don’t put it in your API request that everybody can then see. So disclosure to the organization was problematic. They did get it fixed. They got it back online. Several months later, someone else dumped their entire database because they had a regression bug and the same problem came back.

Explaining breaches

Troy: 00:34:15.172 I spend a huge amount of my time in the process of operating Have I Been Pwned trying to talk to organizations and explain that they’ve been breached. And it’s where probably the bulk of my processing time goes just trying to say, “Look, I’m just trying to help you. You’ve got a massive, massive problem. You’ve got to sort this out.” And I thought I’d leave you with a really nice illustration of just what that challenge is like for myself and so many others in this industry. I saw a video recently from a guy called the LockPickingLawyer. Now, if you’ve not seen the LockPickingLawyer on YouTube before, he is awesome. It’s basically physical penetration testing. And he gets any lock you can think of and two minutes later, he’s managed to break into it with a toothpick or something like that. And he was doing this review on a biometric padlock. And the value proposition is you don’t need a pin. You don’t need a key. You just need something that’s always attached to you, being your fingerprint. You put it on the little sensor there, padlock unlocks. But he notices there’s a screw on the side of the padlock, and he takes out his screwdriver. And exactly what you’re all thinking happens. The padlock comes apart.

Troy: 00:35:27.268 That’s not the best bit. The best bit is because he’s a responsible guy, he says, “All right. Well, I now need to disclose this.” This is all we want to do in this industry, whether its digital security or physical security. You find a vulnerability, you contact the organization privately, and you give them an opportunity to fix it before going public. So he gets in touch with them, and he said, “Look, I got one of your biometric padlocks. I took my screwdrivers out, undid the screw, the lock came apart.” And they responded in the most epic fashion, which perfectly illustrates the problem we have in this industry. They said, “The lock is invincible to people who do not have a screwdriver.” And that is a perfect summary of the challenges we have on the cybersecurity side of things as well. So that brings me to the end of the presentation. I’m going to throw back over to Ev and hopefully, we have some questions.


Ev: 00:36:18.846 Troy, there’s a question there about what was the YouTube channel for this lock guy?

Troy: 00:36:24.599 LockPickingLawyer.

Ev: 00:36:26.257 LockPickingLawyer. Yeah. I see someone actually typed the answer.

Troy: 00:36:30.223 There we go. Yep. Few people found it. I seem to end up watching him almost every day. And what I love about it because I’ve got a short attention span now because of the internet is all these videos are two minutes because that’s how long it takes to break most of the locks. [laughter] There you go. Got a link to it. I’m looking at the comments. Awesome.

Ev: 00:36:44.621 Yep. Something I was thinking about when I was listening to the kind of first half of your talk when you were basically mocking passwords. So I think it’s pretty clear that we, not just the industry, but all people, realize that passwords are just basically a bad idea. And even examples that were given where someone can steal your password to log in some system as you, the consequences of that are way more catastrophic if someone gets into the infrastructure of a SaaS company. Because in this case, they get access to everyone’s data. But I do have a question for you, which I like to ask every security person I meet. But what prevents us to move completely away from authenticating with a secret? And a password is just one form of a secret. A secret is something you know. Yeah. Sure. Password. Because I’m actually getting surprisingly different answers. Some people say that “Well, we just need to invent new technologies.” Like, “We really don’t have a better tech than passwords right now.” I personally disagree with it. So Teleport, for example, we’re moving forward towards completely passwordless access to computing infrastructure. But some people also say it’s a user experience issue. Just like you said earlier, passwords are very simple. They survived for so long because it’s an easy concept to understand. And interestingly, a couple of weeks ago, we had Bruce Schneier on this very same podcast series. By the way, it’s recorded, so anyone can go and watch it. And his point was that no one actually wants to pay for better security. And passwords are cheap simply because they’re — if you’re an engineer, password management is built into Ruby on Rails or numerous software development frameworks. So he’s basically advocating that government needs to step in and introduce more regulation. So you see, we have multiple points of view. It’s a user experience issue, it’s a technology issue, or it’s an economics issue and free markets not going to help, so we need government regulation. What’s your take?

What’s keeping passwords alive

Troy: 00:38:58.087 Yeah. So what’s stopping the password adoption or what’s keeping passwords alive? My mom and dad. That’s what it is. And I’ll expand on that because it’s a little bit of a joke, but there’s a serious side to it. So my mom and dad understand very well how to use passwords. I try to get them using things like two-factor authentication. It’s not particularly easy. If I had to try and get them using, say, a U2F key, it would be a bit of a nightmare. There would be all sorts of challenges, mostly because of their demographic. They’re in their 70s. They are the stereotypical people of that age that didn’t grow up with computers like we did. So the thing that passwords have going for them that nothing else has been able to challenge yet is a combination of simplicity, low cost, and recognition. So everybody knows how to use a — if I go to mom and dad and go, “Hey, I’ve got this U2F key in my drawer here. You can have one. You can log on to things,” they’ll be confused. If I go, “Do you know how to make a password?” They’ll be like, “Yeah. Yeah. No problem. I can do passwords all day long.” So that’s our challenge. And I think the question then is, how do we transition from that into some other, more sustainable, more secure model?" Because arguably, passwords are woefully secure. And one of the sort of counterarguments I always put out to people who are very adamant that passes have to die, it’s like, “Well, they haven’t. Why do you think they’re still here?” It’s not an accident. There’s a reason for it. So I think the answer is going to be a combination of the fact that the demographic that is less familiar with newer technical paradigms will move on I hate to say it. And also, the newer demographic, people like my kids who have grown up with mobile devices on their wrists and secure cryptographic devices in their hands, they will start to embrace these technologies more and more. And we are seeing a shift. We’re definitely seeing a shift. Like you said, everything you’re doing in Teleport is passwordless. It would have been hard to say that probably only five years ago. But it’s going to be a slow shift because we’ve got so much out there that is dependent on passwords.

Ev: 00:41:06.329 Yeah. True. And so you don’t think that government needs to be looking into this? I’ll give you one example of let’s just say it’s a regulation. Say we already have existing standards like SOCs and PCI and FedRAMP. So they might introduce — again, it’s a random idea that just popped into my head. It’s like if you want to be — if you want to meet this compliance requirement, all of your engineers they need to have laptops. They have biometric authentication, and that needs to be the only method how they’re getting access to any kind of production data or any data at all within an organization. So then you won’t be able to use a random ThinkPad that doesn’t recognize your face and your fingerprint and a hardware token that you stick into your USB. So that would be one example. So what are your thoughts on that?

Troy: 00:41:56.874 Well, the first thing that came to mind when you were talking about SOCs and PCIs, I’m like, “How many people actually like dealing with SOC and PCI?” [laughter] Because it’s normally not a very fun thing for the very fact that it is a lot of regulation and burden and red tape. And this is the challenge we’ve got where we could say, “Should government regulate this more? Should we force people down a certain direction?” Well, the concern here almost here, is that going to stifle innovation or is it going to add costs and deny us of things that would have otherwise been produced? Look, inevitably, the answer is partly yes. And you’ve given some examples here of regulatory bodies that do set forth standard for us to follow. The thing that just immediately came to mind here, my fiancée’s Norwegian, and apparently in Norway, there’s this one banking authentication system. So you can be using totally different banks, but there is a standard, unified way of authenticating to banking. Now everyone seems to be pretty cool with that. It’s been around for years and years and years. That might be a good example. That brings with it other challenges. But if you want to log on to your bank, you need to use this system. If you want to log on to like catforum.no, well, then you’re probably just going to have a normal password and username. So I think it is part of the answer. But again, it’s something that’s going to be very slow-moving because we just have so much stuff to move.

The need for security solutions that don’t impact cost or usability

Ev: 00:43:18.800 Thank you for this. And what are your thoughts about using another technology that we already are implementing — well, we have been for quite some time — more like on a services side? So if you go to do online banking, for example, or shopping, every website actually authenticates itself to you quite well using certificates. So your browser is trusting Bank of America or it’s trusting Wells Fargo because the website is secure with the certificate and the browser trusts that certificate. So it’s extremely hard, although people do manage to pretend to be Wells Fargo, to pretend to be Golden Sachs or whatever. So what if we had a system where each individual, let’s say, gets a certificate from one central authority, and then the rotation of the certificate is automated in a certain way? So shouldn’t that make the authentication problem easier and go away? So stealing someone’s identity would become impossible because getting a hold of a certificate is not easy. So have you thought about this kind of interesting angle, that especially now with the fact that we have crypto, which is kind of similar technology, that you could maybe authenticate with your Bitcoin wallet?

Troy: 00:44:28.592 I think what’s fascinating about the discussion is that it’s very easy to come up with very solid technical solutions that work extraordinarily well, certainly much better than the current password mechanism. It’s very hard to come up with technical solutions which don’t have serious impact on either cost or usability. If we think about sort of the client certificate kind of model, great cryptographic solution. Fantastic for identity verification and authentication. What happens when you lose your certificate? What do you do next? What happens when you’re on a device without the certificate? What happens if you have a compatibility problem with your certificate? We used to have digital certificates for a bunch of our online banking services in Australia. Not banking services. Government services. So if I wanted to lodge my — we have to do a quarterly tax return here called a Business Activity Statement. And I always had to log on with the client certificate. And that did end up disappearing years ago in favor of not just passwords but also authenticator apps. So effectively, to a FA model. And largely, I believe, because of the fact that whilst it was very, very hard to take over an account with that, it did also create a lot of friction. And the challenge now is saying, “Okay, what do we do with people who then lose the certificate or don’t have access to it will have compatibility problems.” And that’s a difficult one because we keep sort of balancing up show do we make something more secure, but — it’s a little bit different for government because government — I was going to say, how do we also make sure you keep your customers? I’m not sure that I’d refer to myself as a customer of the government. [laughter] I feel more like they’re working for us. But if you’re an organization, you’re trying to constantly lower the barrier to entry and reduce the friction and save milliseconds here and there. And then you get these smart security people come up and go, “I got a really good solution that’s technically beautiful, but it creates the friction.” And there’s a balance there. There’s a balance. It’s going to be different in different scenarios.

Ev: 00:46:23.784 Yeah. It’s almost like centralization and decentralization. They’re constantly working on establishing equilibrium. So if everything is centralized, and all of the internet, for example, just shrinks down to, let’s say, Facebook — I know it’s a horrible picture, but [laughter] — then securing it is probably easy because you have a single authentication, single authorization, single identity source, single everything. But we don’t want that. We want to be distributed. We want to have our own data. We want to have many services, devices. Which is a great thing, but now it means that your attack surface area to go and attack you, Troy, is sprinkled all over the world. And this centralization versus decentralization idea kind of popped into my head when I was listening to your example of having a central banking service. I think it would be a beautiful thing, or maybe, if we did indeed have one global identity source and more and more services would learn to trust it. Kind of similar with your driver’s license. You lose a driver’s license; you go to DMV to get another one. Yeah, sure it’s inconvenient and you have to renew it every once in a while, but at least it brings same order. Let me also get back to we have some questions here and comments. So Forrest just posted. The situation is similar to COVID vaccines and mandates that we have tools in most part of the world to handle the pandemic and we know how to do it. It’s just that people don’t want to be required to do all of these things. Yeah. I guess it’s the argument for regulation. That’s basically what Bruce was talking about, that it’s unrealistic to expect kind of the market forces to kind of force everyone to do the right thing.

Troy: 00:48:06.280 I mean, I think Forest’s point is good, too, insofar as it’s friction, right? I know for me — I think we’re all like a little bit COVID tired — and it’s like I’m walking into a shop now and three-quarters of people are wearing a mask and I’m like, “Have I got to do this? It’s getting in the way of me having fun. Do I really want to have to do this?” I think it’s a little bit the same with a lot of InfoSec stuff where we almost become fatigued. I keep hearing this term being used where people go, “Oh, there’s data breach fatigue.” People are just over it. It’s like, “Another data breach. Who cares?” [laughter]

Ev: 00:48:37.630 Yep. But I will say about the usability issue, I just recently upgraded to recent MacBook with the M1 processor and the fingerprint. That works beautifully. Yeah. It’s gotten a lot easier for me to use a lot of services because I try to connect everything to that SSL method.

Troy: 00:48:54.108 So that’s a good example of where I don’t think security necessarily has to be friction. I love biometric authentication. From the perspective of now, we can put a very strong, robust means of identity verification on things like my iPhone with Face ID. And it’s easier than what it was before because I’m not trying to tap — okay. Other than when I’m wearing a mask, which brings us back to the previous point. But it’s actually improved the user experience as well as the security. I’m overjoyed when I see both of those things achieved.

Perspective on biometric authentication

Ev: 00:49:23.727 Yeah. I just want to make a comment for people who are wondering if biometric authentication is going to, or if it has the potential to replace the passwords. Our opinion is that by itself it probably is not enough. But for those of us who have a smartphone, that’s everyone, like Apple phone, in addition to recognizing your face and your fingerprint, it also has a TPM module on it. So there’s a device attestation going on. So it’s a combination of your finger and also this particular iPhone, not any other iPhone. So if you combine the two, that is actually a pretty good method of moving forward. We also have another question just popped up in chat. So Ali is saying, “Thank you for the presentation. It was fun. And Troy, by the way, I agree it was. Problem is, after some point, trying to secure your data makes you paranoid, almost. Where to stop?

Troy: 00:50:23.412 It reminds me of this old —

Ev: 00:50:24.320 It’s a fun one. Yeah.

Between being cautious and being paranoid about security

Troy: 00:50:26.772 This old saying. I keep clipping my fingernails, but they keep growing back. Where to stop? It’s like, “Well, you’re maintaining equilibrium.” And I think this is the way to look at it. Maybe it’s as I get older and more mature; I find that a lot of these discussions become a lot less binary and a lot more nuanced. So it’s no longer a question of, “Is it secure? Is it insecure? Is it good? Is it bad?” It’s like, “Well, under the certain circumstances and so on and so forth.” So I think here the bit about the paranoia and where to stop, let’s say from a personal perspective, I use a password manager. I generate all my passwords randomly. I use multifactor in most places. I try and reduce the amount of data that I share with organizations. Where to stop? I don’t really lose sleep over it. I don’t not use services which I think will give me — will give me some sort of benefit in life because I’m paranoid about what might happen with the data breach. I use Facebook. A lot of people get upset with me. Like, “Aren’t you a security guy? Why do you use Facebook?” “Well, my friends are on Facebook. That’s where I see my friends, particularly over the last couple of years.” So that balance is different for everyone. And where you stop is going to be different. But you never completely stop trying to protect yourself. The same way you never stop cutting your fingernails, even when they keep growing back.

Ev: 00:51:48.593 All right. So I have one last question, Troy. This is time for you to get vulnerable here. Have you ever been Pwned?

On being Pwned

Troy: 00:51:58.179 Yeah. Yeah, totally. I said earlier on the talk, actually. I mean, Have I Been Pwned? About 25 times? And what I find fascinating about that is that some of it I know. I’m in Dropbox and LinkedIn for obvious reasons, but then I’m in all sorts of things that I couldn’t even remember. So usually, I load a data breach. I set it, sending notifications to everyone. I go, “That’s it. Job done. Now, I’m going to go and have a beer and relax,” or something like that. And a little while ago, I loaded one called Housz. I think it’s H-O-U-S-Z. I go off to have my beer. And then I get this email from myself, and I’m like, “Oh, not that guy again.” Because I’m in the Housz data breach. I’m like, “What’s Housz? I have no idea.” And I run back up to my computer and I pull the data out and it’s something to do with interior design or something like that. And I’m like, “What on Earth is this?” And I go back to the date and then I find an IP address and I look it up and it’s an IP address in London. And I go to my TripIt and I was in London right at that point in time. So I’m so breached in so many things I just can’t even remember where. And maybe this also speaks to Ali’s point as well. It’s like, “Where do you stop?” Well, I still want to sign up for these things. But I got breached on Housz. Strong, random, unique passwords on everything, minimal data provided, the impact actually isn’t that bad.

Ev: 00:53:14.751 Actually, I just noticed we do have a few minutes left. And there was a question asked by Ben, which I think has a practical implication for our audience today. If you run a large organization, what extra steps should you take if you see an employee email in a data breach dump on your website?

Troy: 00:53:33.156 Well, it depends.

Ev: 00:53:33.655 Because you mentioned earlier that people use their work emails. So you’re an employer and you see that your employees ended up in the database, what should you do?

Troy: 00:53:42.246 It depends on a lot of factors. So let me give you — let me give you one example which made many organizations feel uncomfortable and then I’ll tell you about multiple ways it was dealt with. Everyone knows about the Ashley Madison data breach. Massive incident in 2015. 30 plus million people. Significant mostly because of the nature of the service being there for people to go and have affairs. There are a lot of people in there with their work addresses or with their government addresses. Now, when I sent domain notifications to organizations and they got emails saying that "You’ve got multiple staff members in this data breach," there were lots of different ways of dealing with it. And these are all true stories. Some organizations said, "This is within the scope of what we would consider acceptable use of a computer system. We consider this a relation site. Yes, it’s a very specific kind of relationship site, but this is within acceptable use. No action required." Other organizations, particularly where there were higher-level executives involved, were really worried about things like the risk of blackmail. So we saw a lot of blackmail after Ashley Madison. So in those cases, they had HR speak to particularly impacted individuals who were at that high level. The person delivering the mail? Probably don’t care too much. The person who’s a C-Suite executive? Well, that’s a very different discussion. So it is enormously contextual and it differs a lot between organizations as well.

Ev: 00:55:08.421 Thank you for that. And one last question. So this presentation has been enjoyed with the audience. Do you have any other topics that you’re looking forward to speak about?

Troy: 00:55:18.735 Yeah. You know what? The topics that I have actually I think enjoyed speaking about the most are the ones that are more soft skill topics. There was a talk I was doing a few years ago called Hack Your Career. So how did I end up from a normal employee through to the life I have today. And if anyone wants to have a look at that talk, if you go to my blog — /recorded-talks or just Google it — there’s a couple of talks out there. So I’ve been working on a successor to that, which I only want to do in person on a big stage somewhere in the world. So I’m looking forward to doing more of that sort of thing in the future as well.

Ev: 00:55:55.777 Well, thank you for this. So we have a few more questions coming in, but we also need to watch the time, and we need to be respectful of Troy’s time. So thank you, everyone. Thank you, Troy, for a fantastic and entertaining presentation. I don’t think I laughed as hard ever watching a security talk. And thank you everyone for attending. A quick reminder that this webinar is sponsored by Teleport, and Teleport is open source, easiest way to access securely your computing infrastructure. So we’ll see you again soon.

Troy: 00:56:29.785 Thanks, everyone.

Ben: 00:56:32.203 Thank you for joining for the second talk in Teleport's Security Visionaries 2022 series. We’ve gathered the world’s foremost security researchers, practitioners, and thinkers to see what's on the horizon for 2022. Next up, we’ll have Michael Coates. Michael will be talking about balancing security and agility while scaling your company. Please join us on March 15th at 10:00 AM Pacific. To register, please go to goteleport.com/security-visionaries-2022.

Join The Community

Try Teleport today

In the cloud, self-hosted, or open source
Get StartedView developer docs