Transcript:
RSA Breach: The Untold Story, Pt. I
FIRST NEWS
It’s the week of February 14th, 2011. We’re at the big, beautiful Moscone Center in San Francisco.
Walking through a truly giant expo floor, hundreds or maybe thousands of people pass by, chatting and visiting presentation booths set up by big tech companies from around the world.
Nearby, in a giant conference hall, row after row of portable, semi-comfortable chairs are lined up with TV monitors above, so that people in the back can see what’s happening all the way up on stage. Among the speakers lined up for the week are computer expert and author Bruce Schneier, physicist Michio Kaku, Deputy Secretary of Defense William Lynn, and the 42nd President of the United States, Bill Clinton.
This is RSA Conference 2011--one of the biggest industry events of the year, the place to be if you’re in cybersecurity.
[Sam Curry] I’m Sam Curry. I’m the Chief Security Officer at Cybereason and a Visiting Fellow at the National Security Institute.
Sam Curry was at the 2011 conference, naturally: as a Sr. executive at RSA at the time, he was busy organizing and shaking hands--probably too busy to actually enjoy any of the speeches. Among the wheeling and dealing, the snack bars and chatting with other CTOs and CSOs and listening to talks about this and that, there was hardly any time to think about anything else.
It especially wasn’t a good time for bad news.
[Sam] The first time I got wind something was happening, I was at RSA conference [. . .] and we had an incident.
An incident. Inconvenient, for sure, but probably nothing serious.
[Sam] We have incidents all the time. And it was late night on the phone – being on the phone constantly with all the chaos of a show with some colleagues and we pretty much contained the incident
With the incident contained, Sam went back to his more important business.
The conference soon ended, and a couple weeks passed. But not everything was completely back to normal.
[Dave] We had missed some code delivery for a client that was very, very out of the ordinary.
Dave Castignola, today the CRO at Bugcrowd, was a VP at RSA in 2011.
[Dave] And then I had seen some network issues and some servers being taken offline and that just immediately sent alarm bells up.
Lots of employees started experiencing these same issues--issues accessing data, communicating on RSA’s network. It was more than your typical outage.
[Sam] You couldn’t get out if you were on a corporate network and the VPN wasn’t – probably wasn’t deliv w ering the content you were used to.
[Dave] And on the Friday night, the week before we announced, I had just had a terrible feeling all weekend long. And then I received the phone call Monday morning about what was happening.
That phone call was the beginning of the most chaotic month in Dave’s career.
INCIDENT RESPONSE
Hey listeners, welcome to Malicious Life, in collaboration with Cybereason. I’m Ran Levi.
If you’re a regular listener of this podcast, or if you follow cybersecurity news, you’ll know the kinds of things that we typically focus on when a big hack occurs.
The first question, usually, is: what’s the damage? What machines were brought down, what data was taken, who’s affected and how?
After that, we ask two questions, in no particular order: who did it, and how’d they pull it off? Was it Russian cybercriminals, a phishing email and lateral movement in a corporate network, the Chinese military with a zero-day vulnerability, or Colonel Mustard in the hall with a knife?
Far less emphasis is usually given to another component every hack: the incident response. How the hacked entity deals once they realize what just happened to them.
It’s too bad, because incident response is just as important, just as dramatic as any other aspect of a hack. Think about Yahoo and Target, ignoring repeated red flags. Think Equifax which, after losing the largest trove of personal data in history, offered just one year of free credit monitoring to Americans, and put up a website which purported to tell you if you were affected but didn’t actually work. On the opposite end of the spectrum, think Maersk during NotPetya, or Saudi Aramco which, after 30,000 of their computers were destroyed by Shamoon, simply went out and bought 50,000 new hard drives on the spot. Nearly the entire world’s available supply at the time, right off the factory floor.
There are a lot of different ways an organization can respond to a hack but, when you’re in the war room, it’s hard to make the best, most logical decision. For the moment, you’re at the center of the world, with media attention and jobs and money and a lot else on the line. It’s this kind of pressure that can destroy the weak, or bring out sides of people they didn’t quite know were there.
In early 2011, Sam Curry and Dave Castignola had no clue that they’d be thrust into one of these situations. That the kind of security professionals, the kind of men they believed themselves to be would be put to the test..
3/14: FIGURING IT OUT
It all began on a Monday evening, when an employee noticed sensitive data exiting from RSA servers.
[Dave] He saw this thing in transmission. The profile of the data stream didn’t look right. It wasn’t what it should have been on that court. So he hit the big red button and shut down production, which was a brave move in itself.
The whole company was put on pause, just in case.
[Dave] And that happened three or four days prior.
The next day, there were lots of questions and few answers. Evidently something was wrong at RSA Security, but exactly what was still unclear. Unclear, until they discovered what data had left the system.
[Dave] We then hunted it down and we found an encrypted object. OK. We opened it up within one of our encrypted objects. We opened it up and we saw exactly what it was. It was a secret material related to SecurID.
SecurID, RSA’s flagship product. The exfiltration had been proactively interrupted partway through, but this was bad.
[Sam] We had three days to think about what if, what if, what if, and hopes rose and fell in those three days. [. . .] It was really the Wednesday morning that we determined yeah, beyond the shadow of a doubt that had happened. [. . .] It took us nearly three days to come to the conclusion that we had had a breach.
Sam and Dave, upon hearing the news, had the same thought.
[Sam] I believe we are going out of business.
BACKGROUND ON RSA
Sometimes, data breaches cause millions and millions of dollars in losses for a company. Usually, though, they don’t cause an otherwise solid business to go under.
RSA was not Yahoo, not Mt. Gox. They were a solidly-run company with a good reputation and steady revenue. Their breach threatened to take that all away--not because they’d lose money, or data, but because they’d lose trust. RSA were in the business of trust.
[Sam] RSA is a company that has been acquired twice in its history. It stands for Rivest, Shamir, and Adleman, the three people who came up with the RSA algorithm that really enabled internet commerce in the late ‘70s
RSA Security specialized in public key cryptography--algorithms that ensure the right machines are talking to each other on networks. Take, for instance, their most well known product: a little USB key fob with a screen called SecurID.
[Sam] I used to joke I was the, at one point, the CTO of the world’s largest keychain manufacturer.
SecurID was a hardware two-factor authentication device, with a screen that displayed numeric codes.
[Sam] The numbers on there changed synchronically with the server in the background because of a shared secret. And so, when you go to log in to something, you can enter something you know, something you are, a password for instance or a fingerprint or whatever. And then you could add this thing which represents I have the token, the unique piece of hardware in the world that has the same numbers that matched with the server on the backend it’s expecting.
And so, the machines and the systems can be reasonably sure it’s me. And that was a little over 50% of the business at the time.
SecurID was the heart of RSA’s business, but also a key component for the cybersecurity of organizations worldwide.
[Andy] Everybody who cared about multi-factor authentication had these RSA tokens. I mean, they were just totally ubiquitous.
That’s Andy Greenberg, journalist at Wired Magazine. This week, we’ve partnered with him and Wired to co-release the RSA story. He spoke with our Senior Producer, Nate Nelson.
[Andy] So if you're a SecurID customer, you trust those little tokens. You believe that they offer a trusted... And I mean, you assume almost unhackable layer of protection. And yet, something that happens entirely outside of your view, off of your network, this hack of SecurID on RSA's network, has totally compromised a really crucial layer of security on your network.
[Sam] And this breach said everyone using that from governments to financial institutions and secret organizations, everyone using that suddenly couldn’t trust who potentially was connecting to them.
MYSTERY OF THE KEY
There was just one bit of information to save them. A detail that wasn’t clear yet, and made all the difference in the world.
[Dave] The question was, did they have the key? Could they have decrypted it, that inner container?
[Sam] Because if you got something encrypted, no breach.
The attacker had stolen a safe, but needed the combination to crack it. If they had the combo, they had the safe. If they didn’t have the code, the safe was useless. As if no breach had occurred in the first place.
[Dave] A lot depended on whether or not they had that key.
To be clear, RSA didn’t have evidence that their attackers had stolen the encryption key. They didn’t actually know if they were compromised--and thousands of companies around the world were compromised along with them--or if nothing of consequence had actually happened at all.
SCHRODINGER’S HACK
It left them with an incredibly difficult choice to make, where each choice carried immense but completely different risks. As an analogy, consider our Malicious Life Senior Producer, Nate Nelson.
Nate is ugly.
[Nate] Wait, really?
It’s too bad. He’s just got the kind of face that makes you want to look away.
[Nate] Gosh...good thing I got a job in podcasts, then.
Is Nate better off now that I told him about his weird face? It probably made him sad, but maybe now he can address the problem: get plastic surgery, at least invest in some ski masks. Or was Nate better off before? He was probably much happier that way, going through life in blissful ignorance of why he can never get a girlfriend.
[Nate] All these years, I just assumed it was because of my terrible personality!
RSA Security may or may not have lost the keys to SecurID. It was Schrodinger’s Hack. Two, equally plausible alternate realities were ahead, and they didn’t know which was true. That meant they had a choice.
In timeline number one, they’d assume the worst. They’d disclose the breach, and face the consequences.
[Dave] let’s start with the media, the media is unforgiving, and rightly so in many cases. It was not going to go well. It was going to look extremely bad. That would have affected every deal, every renewal, every financial metric the company dependent on. But we were part of a bigger company. It was going to affect them too. So we were like a billion dollar entity, in fact, just shy of that in a larger over $20 billion company, EMC. This was ugly. This was really ugly.
It’d be a nightmare. And what if, a week after disclosing to the public, they discovered that the hackers hadn’t actually obtained the SecurID private key? All that for nothing.
On the other hand, if they didn’t disclose what’d happened…
[Dave] The risk associated with it though was potentially enormous because something like 80 million people who had highly sensitive and important positions used us to prove they were who they said they were in their own operations. The fear that it could cause and the genuine risk, that was terrifying.
3/16: DECIDING WHAT TO DO
It was a chilly, cloudy Wednesday morning in Bedford, Massachusetts, on March the 16th. In a conference room at RSA HQ, Sam Curry and the execs at RSA convened to consider the decision in front of them.
[Sam] Senior management were sitting around and the question, what do we do, came up. And I’m not going to out anybody. One person said, “We don’t have to say anything.” And it was because there were no contracts, no rules, no laws that said we have to.
Could RSA have ke pt the whole thing quiet? Most cybersecurity disclosure laws are pretty new, yes, but it’s difficult to imagine that working.
[Andy] It's possible they could've gotten away with it. It's clearly not the right thing to do.
[Nate] Right.
[Andy] And I think also, not the right business thing to do. And they would have gotten slammed for it, sooner or later. [. . .] I think they would have been facing serious criticisms or possibly a big lawsuit.
It was at the suggestion of a cover-up that an older gentleman--parted grey hair, a little hunch in his shoulders, butted in. As CEO of the company, Art Coviello had the ultimate say as to how RSA was going to respond to this crisis. What kind of company they were going to be. Employees later referred to what he did next as an “Art Attack.”
[Sam] So he swore, he said, “We are going to do the right thing.” And we shifted gears.
The decision was final. Every individual in that room realized what the implications were. What they were about to do would initiate a chain of events that would very possibly cause the end of their company--a company that’d stood for 30 years.
Sam was as scared as anyone. But he understood what had to be done.
[Sam] I mean I have to share this. Twenty years prior, there were two brands I wanted to work for one and thought were unattainable, and one of them was RSA. And I spent 16 years trying to get there. And eventually, I came on board and ran the product organization and went through a few roles over my time there and ended with CTO. And I was incredibly proud to have made it there and to be among the best. Art used to say, “We stand on the shoulders of giants.” I felt like I was responsible for that. And I remember when we went to disclose, I felt, “OK, if we are going down, we are going down right.”
…
21 HOURS
[Sam] I should tell you, Nate. It was 21 hours from the decision that we had had a – from the moment we knew we had a breach to decide and go public was 5 minutes. And it was 21 hours from that moment until we did. So we can tell you what it was like to suddenly rush people to the battlements but it was less than a day.
[Nate] And why did it need to be less than a day?
[Sam] There was a sense of urgency about this. There was – maybe the adrenaline and the energy we were just talking about, maybe the terror around it. It wasn’t like something that we felt – there was no rules that said we have to but it felt like you don’t dither over stuff like this.
[Nate] True. Although when you write a podcast about enough corporate hacks, this kind of thing stands out as a bit unusual. So was it by virtue of leadership of the company, the culture of the company?...
[Sam] The Art Attack helped. I think part of it was culture that we were a security company and there was nothing like this before and arguably, very little exactly like this after. Later I think we came up with a whole bunch of like how should you do this stuff but even now, I’m amazed that look, Dave got the call on Monday something was wrong. You were there, what, Wednesday, Dave?
3/17: DAVE’S FLIGHT
[Dave] I flew in Thursday morning, Sam.
[Sam] Thursday morning.
Dave Castignola, in all this, was half a country away.
[Dave] My phone call on Thursday morning, March 17th, which was, “We need you. We are announcing this today. Bring a big suitcase. You’re going to be here a while.” [. . .] You’re going to be involved in the response with us and we can talk more when you get here.” But I just remembered the words in my head, failure is not an option.
As head of the communications arm of the company, Dave flew to RSA HQ in Massachusetts from his home in Michigan.
[Dave] I knew that this was dire. In fact, I called my father-in-law up and said, “Richard, I need you to come to the house and you need to help the family. I’m going to be gone for a long time.” And I said to him, “When I come back, I’m not sure if my company is going to be in business.” It was – I could feel it to my core.
[. . .]
I didn’t even have a chance to say goodbye to my wife. I left and she was out someplace. I had to rush home, pack my bag, and I was off to the airport.
[. . .]
And in that time between getting to the airport on a plane from Detroit to Boston, I download and read three books on crisis management. I didn’t know what I was going to do but I knew this was a crisis beyond anything I had ever dealt with.
The flight was only a couple hours, the drive out to Bedford 30 minutes more. He arrived at the office late in the afternoon.
TERRIFIED
The energy in the building was unlike it had ever been before.
[Dave] My first meeting in a very – not very big conference room full of many people, we had to stop the meeting after a few minutes because there were a few people weeping and crying and they were shaken. And I said, “Hey, let’s pause real quick. Get some waters. We’re going to be in here for a while this evening.” I had people out in the hall just like grabbing me saying, “Are we going to survive this?” And I didn’t know if we were going to survive but of course, I told everybody, “We are going to get through this.” We had to keep the strong face for everybody but everybody was beyond rattled.
Employees made last-minute scrambles, but there was only so much they could do. The announcement was about to occur, in just a few minutes.
[Nate] You guys had all of these, very sad meetings and-...
[Sam] By the way Nate, I don’t think sad is the right word. I think terrified. It’s not sadness. When I think back not just how I felt afterwards but how I felt in the moment [. . .] it was terror.
THE BIG MOMENT
As the big hand passed ‘12’, and 4:00 pm became 4:01 pm, they released the news. The whole world would now know what happened at RSA. Everyone in the building braced themselves.
[Sam] There are a few times in your life when you realize you’re staring at a monster. It’s bigger than you, way bigger.
It was time.
Sam Curry: It went crazy.
[Dave] it was just mayhem.
