Introduction to cybersecurity threat detection analytics

by HPE Enterprise Security Team

Many IT security organizations are in the midst of considering how big data can assist in their plans to detect advanced cyber adversaries. Many teams are starting to build big data infrastructure and feed it both structured and unstructured data. But few have determined exactly what they will do with the data after they collect it.

This tutorial offers a vision of what you can do with all of that security data, a vision for detecting advanced adversaries by pairing big data and data science. 

Why it's so challenging to detect breaches

The security industry is not catching enough bad guys. According to the M-Trends 2014 Annual Threat Report: Beyond the Breach, by Mandiant, the time between a breach occurring and its detection averages 229 days. In fact, the majority of breaches are reported by external parties and law enforcement agencies after stolen assets show up in the underground economy, according to Verizon's 2014 Data Breach Investigations Report. These facts underscore what the security community knows to be true: You need to consider new ways to detect your own breaches more quickly.

Currently available cyber security tools are pretty good at detecting known attack patterns. If an attack matches a signature, talks to a known bad place, uses unencrypted protocols, or happens within the infrastructure that you closely monitor, you can reliably detect it as it occurs (if the technology is set up properly). What people struggle with is detecting unknown attack types, new malicious behaviors, and insider threats.

Security professionals also struggle with attackers hiding within a bell curve. In the past, many attacks occurred at 3 pm on a Friday just before a three-day weekend. This allowed plenty of time to break in, ransack the place, clean up, and install a back door for persistent access. Today, things have changed and we now often see attacks on Wednesdays at 10 am, because our adversaries understand that we're sensitive to volume, and that is the hour of peak network traffic based on well-understood behavior. Our adversaries know how to hide in our normal bell curve of network activity.

A “river delta” analogy for detection analytics

When envisioning the future of detection analytics, think of a river delta as analogous to the current enterprise security landscape. The delta consists of streams, rivers, and an ocean, as shown in Figure 1.

Figure 1. The modern threat landscape and the geography of security detection as analogous to a river delta.

Endpoint and other security devices produce small streams of operational data. This data flows downstream, where it's aggregated into rivers of enterprise log and security data. The rivers include business operations context, IT operations logs, and information security events. When you aggregate and monitor these using real-time correlation in a security information and event management (SIEM) system, they anchor the modern cyber defense center (CDC), which monitors real-time correlated security events to detect indicators of potential attacks in progress.

Given the modern threat landscape, you can now picture how real-time capability requires a correlation and longer-term analytical capability as a supplement. You need to expand operational post-hoc analytics to the data “ocean,” by assigning this work to a newly formed “hunt team.” There should be an important operational link between the hunt team and the CDC, especially when an unknown attack takes place. Once an attack type is detected, it is then converted into automated (and hopefully) real-time detection, so in the future you can catch it in real time.

In terms of geography, the tactical technologies for breach detection and prevention are in the “streams” of data (e.g., intrusion prevention), operational monitoring capabilities sit across the rivers of data (e.g., SIEM), and any strategic data analysis for breaches resides in the oceans of data (e.g., hunt teams). People and processes are a critical link between these levels.

All data is not equal

One counter-argument to using big data is that all data is not equal. In the data analytics community, this has led to what's called the “small data movement.” The argument is that you should find the application or use case for the data you collect before you collect terabytes of it. The conventional wisdom, “collect it all and figure out what to do with it later,” is both false and expensive.

Every bit of data collected must be processed, transmitted, normalized, analyzed (which is labor-intensive), stored, and managed through a complete lifecycle. For an example of wasteful data collection, consider the typical router log collection. Some of the common logs seen from routers include route up, route down, and route flap. Those three messages are not often useful in enterprise security detection, if ever,  and having a terabyte of such data will not magically make it useful. There is a strong tendency in enterprise security monitoring to collect the information that is easiest to get at scale. But the better way to approach big data and data science is to first find an application for the data, and only then collect it on a large scale.

Security analytics is not a product

You can use commercial products and open-source tools to perform analytics, but you can’t get the full value of enterprise security analytics by installing hardware or software only. Organizations are building systems that ingest hundreds of megabytes of security data per second, but analysts can only read a few bytes per second. Even with great visualization tools, analysts will be able to mentally process only a few kilobytes of data per second. Tools can help pare down the dataset to a smaller size, but analysts also have to know what questions to ask.

As an enterprise security expert, you need to learn to think with an analytical mindset. You need to be curious, explore the data you collect, find patterns, and follow the trail of an investigation wherever it leads. Looking at individual events or correlated events is not sufficient anymore. While the hunt team is a separate role for cyber defense analysts in larger organizations, smaller organizations do not have this luxury. Existing security analysts should be trained to look at their data with an analytical and curious mindset.

A practical note on retrieval speeds

Poor retrieval speeds prevent enterprise security teams from optimally exploring data. The big data space now includes technology, such as columnar data stores, that can quickly retrieve results, provided that you store the data in an optimized fashion (structured data is the fastest). As you build out plans, consider not just how fast you can place data in your ocean, but whether you can get the data out of that ocean, and how quickly you need to be able to retrieve it if it is to be useful to your security teams. Generally speaking, exploration environments require results in seconds (not minutes) to capture the creativity and the “aha” moments of a hunt team.

A vision for the future

To apply data sciences for efficient security detection in the future, you need mature capabilities as you work to detect, explain, explore, and understand security events in your environment. Figure 2 below provides a vision, mapping, and strategy for your current and future detection analytics needs.

Figure 2. A vision for detection analytics.



Existing detection

This is the base detection technology stack commonly in use today.

  • Detect: Current detection methods include real-time correlation and log aggregation. This is the single pane of glass and the ability to highlight the events of interest.
  • Explain: Explaining often results in reporting. This includes threat, vulnerability, and compliance reports. These are topics that need careful construction and effective communication to leadership.
  • Explore: Exploration requires the ability to query data in an ad hoc manner to produce small datasets on which you can conduct basic, formal analysis.
  • Understand: Context is an important component of detection analysis because it can be tricky to decide whether unusual activity is truly a breach or simply a benign event. Having context is a critical component in that decision. To gather context, you can collect information about assets, networks, and identities that you are protecting. This context will be critical in deciding whether an event is malicious and should be escalated as a breach, or if the event is simply a compliance scan.


Emerging detection

This analytical capability is available either as a piecemeal component of an enterprise security program, or as an individual, point solution. It is not yet available as a coherent technology stack.

  • Detect: Consider historical analysis and imagine the ability to conduct long-term correlation as an immediate follow-on to real-time correlation. You could escalate an event of interest and then immediately search for every historical instance of that correlation event. This kind of cyber epidemiology makes it possible to make policy decisions about your enterprise security. This approach is based on an analysis of a “patient zero,” rather than just cleaning up an infected host
  • Explain: Emerging scoring requires the ability to profile IP addresses, users, systems, servers, and other elements of the risk picture in the enterprise at a much higher level of fidelity.

There is a real danger in “feel good” data science that mixes quantitative elements with qualitative descriptors. This can water down the value of the measurement.

  • Explore: Advanced searching assists with exploration, and the ability to pivot through security data is particularly useful. For example, if when exploring a user’s behavior you see the user interact with a particular server, you can pivot and see what else that server has done subsequently. In this way you can conduct effective searches of enterprise security data when looking for known lists of indicators of compromise (IOCs).
  • Understand: True understanding of a breach often requires advanced context that includes such things as internal application error logs—for example, collecting application logs from an application security testing tool such as the HPE Fortify runtime agent so it can report into an SIEM tool such as HPE ArcSight. Another key to understanding a breach is network flow and deep packet inspection data. This allows deeper root-cause analysis so you can understand whether an ongoing attack deserves to be escalated.


Advanced detection

Advanced detection capabilities represent the current frontier of development, both in the vendor community and in advanced enterprise security programs.

  • Detect: Advanced detection includes advanced statistical analysis, the ability to train models and detect outliers across any of your enterprise security data feeds. Even an insider has to deviate on at least one measurable parameter in order to conduct malicious activity.
  • Explain: Here we find one of the greatest assets we have available to us as a community to date: data mining. The main disciplines include:
    • Clustering, which at first glance might not appear to be that valuable for security. However, when you cluster security data, you almost inevitably identify large numbers of false positives. When you clean up these false positives on your security devices, you make the deep, muddy river of data that you're monitoring clearer and shallower.
    • Classification, which is used to classify data into types for comparative analysis and cross-correlation.
    • Correlation, which has been around a long time and provides the analytical engine for modern enterprise SIEM deployments.
    • Aggregation, which seems less powerful than it is. Imagine an aggregate profile of a server, a user, or an IP address based on long-term historical behavior information; this type of aggregation can easily profile an attacker, allowing you to identify similar profiles with the addition of a scoring algorithm.
    • Affinity grouping, which is best articulated in an anecdotal retail use case called the “beer and diapers analogy.” When retail companies analyze what you buy and demographically categorize you, they uncover interesting insights. For example, when men buy diapers, they also tend to buy beer. Thus, keeping beer en route to diapers increases the sale of beer.

In a security context, malicious command-and-control infrastructure within your environment has a higher affinity for itself and its peer nodes than for the normal infrastructure surrounding it. This allows for a significant advance in detection capability. This type of lesson is called a “domain transfer,” and there are a number of effective lessons you can learn from other domains in data science (such as marketing) that you can apply to information security.

  • Explore: Analytical query takes your big data and turns it into small data, with which you can actually do something. Big data management takes large amounts of data and turns it into analytically tractable small amounts of data that can be retrieved in a reasonable time frame. These are often called queries or data marts, and these are where you apply your advanced analytical capabilities.
  • Understand: Technical intelligence in the “understand” row is the concept of producing your own indicators of compromise. Currently, the majority of the industry purchases this intelligence. It is generic intelligence aggregated from open sources and occasionally augmented by honeypot collection projects. The ability to detect new malware in your environment, detonate it, and produce indicators to be shared across your enterprise begins to simulate an immune response, and that is clearly a desired direction for the security industry.


Target detection

In target detection, the ultimate endgame is behavioral detection of advanced and insider threats. There’s a deeper focus here on baselining normal and abnormal behavioral profiles.

  • Detect: Establishing baselines as much as possible in your environment is helpful in advanced detection. The smarter you can get at baselining, the better you'll be at identifying abnormal and noteworthy activity. A behavioral whitelist, where you deny all traffic and explicitly allow only some traffic, is still far off for most enterprises, but it's a long-term target.
  • Explain: As data matures, you want to increase your depth from data mining and move into machine-learning algorithms. There may not be a clear differentiation, but one big impact of machine learning is the automated classification of data types for further analysis. Often, these types define a norm that you can use this in your dataset for analysis.
  • Explore: You need true exploratory visualization for security data. While visualization tools are being effectively applied to information security problems, currently there aren't any dedicated security visualization tools. The detection capability made possible by visualizing large amounts of data and being able to cascade disparate visualizations through a selection processes allows you to perform root-cause analysis and remove data that's not of interest. You can then dig into the remaining data, which will be richer in malicious activity.
  • Understand: Human intelligence advances you beyond technical intelligence. You can understand and gather human sentiment and motivation indicators based on observed activity (e.g., Twitter scraping or IRC monitoring). While this could be a boon to your work, it is an unstructured data problem and quite complex to address effectively. Imagine the ability to put human sentiment under the same pane of glass as enterprise log data. This may be one privacy bridge too far, but it describes a powerful capability for detecting pending, campaign-based attacks.

Shift your outlook

As a cyber security professional, you need to catch more attacks earlier. To do that, you need to shift how you look at data. Simply collecting large amounts of big data is not useful in and of itself. You need a guiding vision as described here and a plan if you are to build systems that will grow with your needs. Only then can you achieve your ultimate goal: the reliable behavioral detection of advanced attacks and insider threats.

Additional reading