Malicious Life Podcast: Operation Soft Cell

Malicious Life Podcast: Operation SoftCell Transcript

Hi, and welcome to Malicious Life, in collaboration with Cybereason. I’m Ran Levi.

You know, I always open the episodes with this line: ‘in collaboration with Cybereason.’ Well, two days ago this collaboration thing became very useful in a rather surprising way.

If you were to read the WSJ on June 25th, you’d come across an interesting headline about something called “Operation SoftCell.”

I know what you’re thinking. Tainted Love, right? Soft Cell were an English new-wave duo that were pretty successful when I was a kid, in the 1980’s. That was a long time ago, sure, but hey – Freddie Mercury is back in fashion these days, so maybe they’re having a comeback… No, that’s not it. The ‘Cell’ in this SoftCell stands for cell phones, and Operation SoftCell is a very dramatic discovery announced just a few days ago by Lior Div, Cyebereaon’s CEO and co-founder.

This discovery, as I noted earlier, made headlines in many major publications around the world. Since I’ve been working with Cybereason on this podcast for a few years now and interviewed many of their researchers on various occasions – it gave me the opportunity to give you, the listeners, a rare inside look on how this discovery came to be, from the actual people who made it. Let’s jump right in.

Unusual Alerts
MOR: So, everything started almost a year ago, when on boarded just another customer.

This is Mor Levi – VP of security practices at Cybereason. And no, no relation – Levi is somewhat of a generic name for us Israelis, sort of like Smith or Johnson in the US.

The client Mor was working with was a big Telecommunication provider: a company which operates a cellular network. A year ago, the Telcomm’s IT people noticed something strange going on. It looked as if someone was stealing information from their network, but none of their existing tools could detect anything malicious.

MOR: So after a few days, we started to see a few alerts in their environment. And our SOC, they are the ones actually analyzing these alerts.

SOC stands for Security Operations Center, the people monitoring networks.

MOR: They are the first responders to the alerts that we’re seeing in our platform. They started seeing these alerts, and already started to realize that something is not normal. Its not the regular alerts they see on a daily basis. They decided to perform an escalation to the research team.

RAN: What was different about these alerts that seemed off?

MOR: First of all, within a few days they received several alerts – one about a web shell, and others tied to this web shell, which usually means that there’s an active attacker in the environment.

A Web Shell is a script that is installed on a server and enables remote access to that server. Having an unknown Web Shell sitting on one of your servers is a sure sign that something is very very wrong. It means someone has a backdoor into your network.

MOR: That was the abnormality of these alerts, and this is where our research story begins.

Team Nocturnus
The team decided it was time to call in Nocturnus. Nocturnus is Cybereaon’s Research team. It is made of some of the company’s brightest and most experienced researchers. One of them is Amit Serper. Some of our listeners might recognize the name: we had Amit on our show before, and he also made some headlines in 2017 when he single handedly managed to stop the notorious Not Petya attack. He’s an ex-Israeli Intelligence project leader and an expert in malware analysis and reverse engineering.

AMIT: So I got involved I think a few days after we – and by we, I mean the company started to look into the environment of that company we were looking into. The team saw some anomalies over there in the environment, large amounts of data that’s being transferred, all sorts of weird-looking commands running on various servers.

One of the team members asked me to help with some reverse engineering of those files. So I think at this point, we were perhaps two days into the investigation.

What Amit found was actually a legitimate software: a trusted and signed application by Samsung. This application, however, loaded into memory another file which based on its activity seemed highly suspicious.

AMIT: the code that we saw had the ability to upload files, download files, change files in the file system, change registry keys. It had the ability to take screenshots to do key logging. Basically it was a full-featured RAT.

Why would a legitimate software load and execute malicious code? Digging deeper into the code, Amit found his answer. The attackers were using a technique known as DLL side-loading. What is DLL side-loading?

It’s a surprisingly easy technique to explain. Say you’re visiting a friend. You open his refrigerator and you see a tasty looking chocolate bar. You eat the chocolate, and then a few minutes later your friend opens the fridge and asks – ‘hey, guys, did anyone touch my Chocolate Laxatives?’.

As part of its normal behaviour, the Samsung application needs to load a file named ssMUIDLL.dll, which it expects to find in a specific folder. When it finds the file, it loads into memory and execute its content. Except that just like the chocolate bar you’ve just eaten, the file it actually loads only looks like the real thing: the crooks replaced the original DLL file with a new one with the same name. The Samsung app loads the malicious file and runs its code.

AMIT: And I think after a couple of hours, we have determined that this is a RAT that’s called Poison Ivy.

Poison Ivy is a very well known Remote Access Trojan, first identified way back in 2005. Although ancient in cyber security terms, Poison Ivy continues to be a popular choice for APT groups around the world because it’s a very sophisticated and effective malware.

AMIT: We then determined that OK, this is not something that we can brush off or this is not like some cases of commodity malware. The fact that it’s a telco and they have large amounts of data ex-filtrated out of them by a program that appears to be a legitimate Samsung application, something is off.

We then saw another indication on one of the machines that had Poison Ivy running on. It was another program that’s running and we didn’t understand in the beginning what it was. We just saw its – we saw that it’s running. We saw that it’s accepting connections from one network and that it’s sending packets to a different network. When we tried to find what this program was, we couldn’t find anything. Its file hashes return nothing on virus total. In was completely unknown.

Hashes And RATs
This requires some explaining. Remember that the attacker’s goal here is to evade the automatic detection systems protecting the network. One of the ways these protection systems detect malicious software is by analysing the files on the network and computing their hash values. What’s a hash? You can think of hashing as a kind of a meat grinder: a machine that takes a piece of meat as an input, and outputs a blob of ground or minced meat. In our case, the input is a file and the output is a number – and most importantly, the number at the output is unique for each file: If you change even a single character of text inside the file, for example, the hash number at the output will be different. It’s as if changing some small parameter in our piece of meat – say, its size or thickness – causes the ground meat at the output to be in very different color.

Detection tools use this mechanism to pinpoint malicious software. When some new malicious software is discovered, its hash is calculated – and when some suspect file is discovered in the future, its hash is compared to the already known malware. If the hashes are identical, we can be sure that the files are identical as well.

In this case, when Amit compared the new malware he found on the server – the one moving packets between different networks inside the client’s IT environment – to previously discovered malware, he found nothing. It was a completely new malware, and so he didn’t have a clue as to what it was doing.

AMIT: After again doing some analysis, I got a copy of this program and I ran it in one of my test machines and it was just spewing out really weird debug messages. Like you couldn’t really understand. Like it showed on the screen “CFE!”.

Debug strings are just simple text alerts that a program can print to the screen, usually to notify its developer of certain events going on inside the software such as “Connection Error” or “Connection Established”.

AMIT: and I got all of those weird, obscured messages that I didn’t understand what they were. So I started reverse-engineering the program in trying to understand what those messages mean. So I found in the disassembled code, I found all of the functions that are printing those messages and then I had to figure them out.

So for like I think a good two hours, I spent like a good two hours in trying to figure out what it means and I kept sharing my notes with the rest of the team and with Asaf of course.

Assaf is Assaf Dahan, another member of the Nocturnus Team, and an experienced offensive security expert. Assaf was in Japan at the time, while Amit was in Boston. Despite the extreme time difference, the two team members worked side by side, trying to figure out what this new malware was doing inside the client’s network.

AMIT: Then Assaf says, “Oh, you know what? It looks familiar. Hold on.” He sends me a link to a GitHub page with a project that’s called hTran.

hTran is actually a well known tool, that allows attackers to easily bridge connections between different networks. Using hTran, the attackers were able to connect to servers deep inside the client’s IT environment, like explorers using suspended bridges between tree-tops to reach deep inside a thick jungle.

The question is – how did the attackers manage to make hTran, an already familiar malware, invisible to the detection systems monitoring the network? The answer: by simply changing the debug strings.

Changing these strings doesn’t change the actual functionality of the program in any significant way – but as I explained earlier, it does change the calculated hash of the file. It’s like changing the licence plate number of a stolen car: it’s still the same car except now, if you’re a cop looking for this particular vehicle, it’s much harder to find.

AMIT: So I then started comparing the disassembled code that I have on my screen and I compared that to the source code that I was looking at on this page that Asaf shared with me and I realized that it’s the same thing. So Asaf actually – Asaf was spot on. All of those weird messages that I saw, when I saw CE, CFE, all of those weird messages that had no meaning, were actually the error messages that were changed by the attackers in the source code.

So various antiviruses wouldn’t be able to catch this program and various researchers would be thrown off. They removed all the other letters in the word and just used the first word. So connection established is CE.

Having discovered the attacker’s modus-operandi, Amit and the Nocturnus Team managed to stop the attack and remove the malicious code from the servers.

Playing Cat & Mouse
A few months went by, and nothing happened. At this point in time, the researchers had no idea what would happen next. Maybe the attackers had abandoned their scheme. Maybe they’d already got what they were after, who knew. Mor, Amit and their colleagues moved on to other projects.

But the attackers, it turned out, were not so easily scared.

MOR: This is the second wave and this time, they’re getting a bit deeper into the network. It seems like they know exactly what they’re after.

AMIT: In the second attack, in the second wave, we saw the attackers coming back and it has been a while. It has been a few weeks. It has been like two months in between waves and the attackers came back and they’ve used a modified version of the web shell that they’ve used before. Again for the same reasons that I mentioned before about HTran. They modified some things.

This time, they were starting to compromise more servers. So they’ve already had a way in through that web shell. So they would get in and they would use a customized and modified version of Mimikatz. Mimikatz is a tool that extracts authentication tokens. So that could be passwords, hashes, Kerberos tickets, pretty much almost any form of authentication that Windows supports. Mimikatz has the ability to extract these authentication tokens out of memory.

They then scanned the network around that web server to see which machines are accessible. Once they’ve had a list of machines, a list of IP addresses, they started trying to authenticate into these machines using those credentials that they have dumped from the web server.

Once they have authenticated, they basically repeated the entire process. They dumped the creds and then they scanned all the machines around the newly hacked machines. Got a list of more IP addresses and then lather, rinse, repeat.

By that time, they had quite a nice amount of usernames and passwords. We saw it as it was happening, almost live. I think we saw it like minutes after it happened. We were following the incident as it happens. We’ve seen the attackers typing the actual commands.

This cat and mouse game went on for several more months and two more waves of attacks. Each time Amit and the Nocturnus team managed to detect the intrusion in the network and remove the malware – the attackers backed off, laid low for a few months and then returned equipped with better tools and better knowledge of the network’s structure and weakness.

Amit: Eventually down the line, they did get a domain admin hash. One of those machines that they were targeting had a domain admin logged in. Now if you have a domain admin logged in and you gain access to their privileges, it’s pretty much game over because you can do whatever you want with the network because you now possess the highest privilege possible.

That was the point where they stopped just like moving around from machine to machine. But they – OK, they – it’s like they said, OK, now it’s time for business.

A Huge Surprise
And the ‘business’ here was the attacker taking control of a major database inside the client’s network, and extracting hundreds of gigabytes of information. But what kind of information? The researchers didn’t have clue: The stolen information was heavily encrypted. So the Nocturnus team scoured the compromised network, found the actual component of the malware that did the encryption – and reverse-engineered it to find the encryption key. They decrypted the stolen information, fully expecting to find a dump of credit card numbers, IDs and all the usual financial data that cybercrooks often exfiltrate from big organizations.

But here they had their first huge surprise. It turned out the information stolen had nothing to do with financial records. In fact, it didn’t come from the client’s business network at all: the information actually came from the Telco’s operational network – the IT network in charge of operating the cellular communication network, the actual cellular antenna towers. The encrypted information contained hundreds of gigabytes of CDR data.

What’s CDR? Mor Levi explains.

MOR: So CDR stands for call details record and this type of data is data that every telco company in the world has. It basically stores the originating phone number that called you and the destination phone number, the duration of the call, the cell towers that you’re connected to, any text messages that you send and so on.

Many people got confused. It’s not like they’re listening to the content of the calls. It’s more of metadata on the traffic of the call.

With this CDR metadata, one could track the Telco’s costumer’s every action on the network – and learn a great deal about each customer’s daily routine.

For example – say you were tracking my CDR data. Tracking my CDR information would tell you that I often like to write in a small coffee shop not too far from my office. Knowing this, you could concentrate your efforts on compromising this coffee shop’s wifi network, and use it as a stepping stone for future attacks. By the way, don’t bother – I never use public wifi networks, and if you’ve listened to enough Malicious Life episodes, you probably don’t either.

But why would anyone be interested in the daily routine of millions of cellular customers? I mean, if your’e Google or Facebook, this kind of information could be valuable for advertising and such – but it hardly justifies the risk and effort involved in a hacking operation of this magnitude. We’re talking about attackers who over the course of months, or even years, went through the trouble of creating a “Shadow IT” for the actual IT network of the Telco: an elaborate infrastructure that gave them complete control over the Telco’s network. Why would anyone invest so much time and effort? It just doesn’t make sense.

The answer became apparent when the researchers started examining the actual content of the CDR data. It turns out our attackers were not after millions or even thousands of customers. They only tracked 20 people. Yes, that’s right: 20 people, that’s it.

This was the moment when the alarm bells started going off in Amit, Mor and Assasf’s heads. Only nation states would invest so much time and effort in tracking down 20 people. This was not your run of the mill cybercrime operation. This was a cyber espionage operation. An APT – advanced persistent threat attack.

A Matter of Life & Death
LIOR: Hi. I’m Lior Div, the CEO and co-founder of Cybereason. So usually they’re super resourceful and usually they’re not coming to me. When there is something that they believe and like they have such an experienced team and when they believe that something of this magnitude is going to happen or they have a hunch that something is – that this is starting to develop, then I am engaged as well.

The fact that Operation SoftCell target only 20 individuals, tracking their daily routine over a long period of time, clued Lior that his company was dealing with a major event. As CEO, Lior Div probably does little actual hacking these days – but years ago, he was a decorated officer in 8200 – an elite israeli Army Intelligence unit- and led many a secret intelligence operations. He knew from experience that when someone tracks the daily routine of an individual in such depth and scale… well, let’s just say that it doesn’t usually end well for that individual. It could be a matter of life and death.

Lior: Usually they count the first wave and the second wave. They dealt with it and they really – they don’t need my help. When they see that you are talking about – in this case, it’s actually an espionage case and there is – people’s lives may be in danger, this is were I’m getting involved.

The software tools used in the SoftCell attacked pointed at one clear potential threat actor: APT10. APT10 is a very familiar name for cyber security researchers: it is a chinese cyberespionage group that over the last ten years or so has targeted many American, European and Japanese construction and engineering firms, mainly for purposes of military and industrial espionage. APT10 is also known for its close ties with the Chinese government.

But although it certainly seemed as if APT10 was behind the attack – it doesn’t mean that APT10 were necessarily the culprits. Lior and his people thought twice, even three and four times, before placing blame on any one particular party.

Lior: It’s super sensitive and we are trying to be super responsible when we are to do something like this magnitude to a country or to a specific group. We managed to see that all the indication is driving us to the conclusion that it is APT10 but in every conversation, we added another caveat with this and we said the conclusion and the correlation was so good that this is APT10 to all the information that exists out there that or APT10 didn’t care to be discovered or somebody basically used their tactic and technique and tried to disguise themselves as APT10.

While all this was going on, the Nocturnus Team was conducting another investigation in parallel – this time, an intelligence gathering operation. They had in their possession the actual malware tools used by the attackers, and so they started looking around for other places where these tools were used.

MOR: So we started to look them up everywhere across our other customer base to see if they’re affected as well as online to see which information exists on those artifacts online and we were able to identify similar tools which were slightly modified. That’s how we revealed the additional companies that were involved.

And when Mor is talking about additional companies being involved, she’s talking about another major surprise for the researchers. When Cybereaon’s investigators tried matching their findings with other threats and malwares encountered elsewhere in the world, they discovered the actual control infrastructure: the proxy server used by the attackers in Operation SoftCell. By analysing this infrastructure, they came to realize that the attack they uncovered was actually only a part of a much larger cyber espionage campaign against many other telecommunications providers around the world: 12 global Telcos, with hundreds, maybe billions of customers world wide. What seemed like a big cyber espionage operation, turned out to be an espionage operation of massive scale, one of the largest ever unearthed in the history of cyber security.

Now they had a different kind of challenge. Here was one single cyber security company out of Tel Aviv and Boston, that had to notify all those huge telecommunication providers, most of which they had no connection to whatsoever – and convince them that they are the victims of one of the largest cyber espionage campaigns in history. Sounds simple? Not at all.

RAN: And then I guess you reached out to them, right? What was their response?

MOR: Of course. It was a very mixed response. In some cases, people just ignore that because they didn’t want to believe or they just didn’t believe. In other cases, we got on a call. We provided them with a full analysis report and never heard back from them. In other cases we got yelled at. So …

RAN: Why would anyone yell at you for actually giving them valuable information?

MOR: Put yourself in their position for a second. Think about it that someone that worked for a cyber-security vendor reached out to you and we reached out through LinkedIn and whatever we could find online because we didn’t have a direct connection with people.

So we reached out to them and for them it was a very strange call because they do not know us first, I believe. This is the first time we’re talking to them. So for them it could sound like someone is trying to do fraud or to blackmail them. It’s not a tactic that people didn’t use for social engineering. So you can understand why people are a bit anxious about it.

APT10
Clearly, APT10 – or whoever was behind the operation – were not going to give up. Lior knew that the only way to stop Operation SoftCell once and for all was to expose it to the world – which as I said in the beginning of the episode, he did last week, on stage in a security conference in Tel Aviv.

RAN: Once you announced that case, Operation Softcell, what were the responses around the world that you received?

LIOR: The amount of feedback, good feedback that we got from everywhere, all the way from starting from different countries all over the world that contact our us with their CERTs and their cyber unit, all the way to – I believe that by now, we debriefed probably more than 70 different telcos in the world and we’re still counting to give them kind of the information and instructions on how to deal with this type of situation. So I believe that many of the people that I discussed with were very grateful that we shared the information and very grateful that we help them in some cases to find the attacker in their nets.

If there’s something Operation SoftCell shows us, it’s how extremely vulnerable our current cellular technology infrastructure is. We know that this critical system is under constant threat: roughly a quarter of cellular providers around the world have reported being the target of APT attacks in the past. Here we see exactly how dangerous these attacks can be: the attackers had complete control of the compromised network, down to the last authentication credentials. Had they wanted to, the attacker could have shut down the entire cellular network with just one press of a button.

Operation SoftCell is also a powerful demonstration of one of the basic assumptions in cybersecurity: no one, no matter how powerful and diligent, is immune to APT attacks.

MOR: At the end of the day, I do believe that if this is a nation state behind this, they will get the information no matter what. They will find their way. It could be in the cyberspace world and it could be in the human intelligence. I think that it’s very hard to protect against a nation state.

LIOR: I think it’s important to emphasize that this is a situation of somebody decding in a very systematic manner to create a capability, almost an asset, to track people. This capability to track any person, anywhere in the world, this is something unheard of. Nobody built something like this. So we’re talking about the situation that a foreign country has an ability to track every civilian in another country, and have a big reach to everywhere in the world.

With 8 billion cellular customers around the world, almost no one is protected against this kind of cyber espionage.