Delayed Detections in MITRE ATT&CK: What Do They Mean for a Business?

 How much data can be exfiltrated in an hour?

That is the question one must ask when considering the value of a delayed reaction by a security product.

During the recent MITRE evaluations, it became apparent that many security vendors, while able to detect threats, were doing so well after the fact. This isn’t an easy topic to broach for many vendors affected by delays, so it remained relatively undiscussed. However, It's important to consider what these delayed detections would mean for a SOC experiencing a real breach.

Without addressing the challenge of delayed detection, or referencing the severity of delays, what are consumers meant to learn? Reportedly, it can take a Russian APT a mere ten minutes from the moment of initial foothold until domain admin privileges are gained. It is worth noting the reasons behind delayed reaction, and the change that they represent in the security industry as a whole.

Our MITRE ATT&CK evaluation results are live. Check them out!

Read Results →

The MITRE Evaluations Shine a Light on a Troubling Trend

Delayed detections for each vendor in the MITRE ATT&CK evaluation.

When viewing the graph of delayed detections across vendors, it’s easy to see the difference in vendors when it comes to delayed detection. Every vendor (except Cybereason & Carbon Black) that received more than just a couple of alerts had many delays. This calls attention to the varying effectivity of the products tested.

According to MITRE, a delayed detection is an alert that is not received in real time, or near real time. This means the detection may come many minutes or hours after malicious activity occurs. When this happened during the MITRE evaluations, it usually meant that a security product required a human analyst to confirm certain malicious activity due to the software’s inability to do so on its own. The security product would send suspicious data to the analyst team, which in turn analyzed the data and alerted the customer when needed. Many critical parts of this process are done manually.

This is in contrast to a real-time alert, which comes as a result of the platform making decisions in real time without any human intervention. This difference means that not all detections are created equal. Alerts created by the security product arrive instantaneously, while alerts coming from an MSSP or services team often arrive delayed.

The Mechanical Turk Problem

The Mechanical Turk was believed to be a fully-autonomous chess-playing machine in the late 18th century. It was able to defeat many impressive challengers, and was heralded as an automaton until it was eventually revealed to be a hoax. A human chess master was hiding inside and operating the machine the whole time.

Put simply, some vendors use human services to make up for product limitations, even though the problem is better solved through automation. These decisions are often critical to an organization’s successful security operation. Unfortunately, the line between what is automated and what will be done manually on the backend isn’t always made clear to consumers.

What’s worse, it’s a difficult solution to sustain, especially when considering the growing amount of data being processed. Humans will never scale as fast as computers will. As data repositories grow bigger and the number of customers increases, vendors that depend on services will need to find new ways to react to security events without increasing delays.

The potential for human error is also worth considering as part of the Mechanical Turk problem. Emerging computers and software are designed for large amounts of traffic and are not affected by fatigue. The human analyst is irreplaceable in certain circumstances, but may be better complemented by an automated solution where available.

Modern advances in technology have allowed security companies to develop products that automatically run through decision-making processes in order to replicate human behavior. Ironically, rather than develop their underlying technology further, vendors that rely on services are opting to use human resources to emulate what a modern security product should do in order to remain competitive.

The Importance of Services in the Right Place

None of this is meant to suggest that services have no place in this equation. This only means that services shouldn’t be forced to make up for the limitations of a particular solution. Humans are best used in roles that require complex and creative decision making. Machine thrive when presented with a finite set of rules. When possible, a security solution should analyze incoming data and run through common decision making to produce the highest quality alerting and reporting. Human services can then be used to compliment and build on these abilities to create an even more secure environment.

Humans should complement the technology, not replace it. Threat hunters and incidents response analysts can better perform their duties when a solution has laid out the basic facts quickly and with correlation, so that the full attack story is available.

The Technology that Drives Real-time Detection 

An overview of the Cybereason Defense Platform.

The Cybereason Defense Platform is able to achieve real-time results using a Cross-machine Correlation Engine. Cybereason’s Cross-machine Correlation Engine is designed to automate many manual and time consuming analyst efforts, proactively identifying malicious activity. It associates malicious activity across the network and incorporates threat intelligence to deliver context-rich alerts. Behavioral detection across all stages of the attack lifecycle provides valuable, insightful alerts immediately. We incorporate machine learning to reduce unnecessary alerts and give defenders the information they need instantaneously.

Our services are then used to monitor and hunt in customer environments, providing an additional layer of security and guidance, addressing threats that require human thinking and experience to match. The value of such an approach is easily demonstrated in some of our latest threat reports.

How Much Data Can You Lose in an Hour?

Our security researchers have seen a real-life WannaMine attack infect hundreds of machines in twenty minutes. This is a password-stealing, lateral moving, cryptomining threat, and a nightmare to anyone not prepared for it.

With a delayed detection, a defender using an EDR could be waiting hours for an alert on an attack like this. Attackers leverage that time to exfiltrate data, spread the infection, or escalate privileges.

With a real-time detection, the defender is alerted immediately as the attack begins and is able to remediate with no exfiltration of data. Our customer response to the Astaroth Trojan is a good example for this. This is the bottom-line difference between delayed detections and real-time alerts.

Beware the Mechanical Turk

Security teams need immediate knowledge of an attack so they can empower their team to resolve incidents before anything serious happens. Unfortunately, that just isn’t happening with the majority of vendors, as seen in the MITRE evaluations. Anything less than real time can be catastrophic for a company that relies on being able to defend attacks as they are happening, instead of hours later. It is ultimately up to the practitioner to decide whether or not delays are acceptable, and at what cost.

Ready to check out our MITRE ATT&CK evaluation results? 

LEARN MORE »

Eliad Kimhy
About the Author

Eliad Kimhy

Eliad Kimhy is on the Cybereason Marketing team, leading production of the Malicious Life podcast.