How to Hunt: Finding the Delta

Identifying outliers or anomalous behavior depends heavily on a robust and credible understanding of those baseline characteristics within a network. Normal behavior and attributes vary significantly and are unique to each environment. Any efforts to structure the baseline may include essential factors such as temporal and geographic considerations, number of users, file types, approved applications, and so forth. In the past, we have referenced baseline and gold image usage to reduce false positives. Since so much of outlier detection and hunting rests on a solid foundation of understanding these baseline characteristics, it is useful to devote some time to this and how to use gold images for hunting. When available, this is extremely important and impacts every aspect of the hunt.


Baselining the Hunt

So what is the most useful baseline for hunting? Most IT staffs start with an OS standard build, and expand it to include additional applications approved for internal company usage. The gold image is defined as this clean slate prior to any user interaction. By comparing workstations to the baseline image, you can simply perform the delta analysis to help hone your hunt. This is especially important if your users are unable to install new software or don’t commonly do so, as anything that deviates from that baseline gold image might be an anomaly worth investigating. Even if this analysis determines the file or application is benign, it may surface a policy violation. This sounds pretty simple, but like any hunting approach, there is a level of grooming that is required to reduce false positives in your environment. Baselining is straightforward in principle but can be very difficult within heterogeneous environments.


Tips to Help Baseline Your Environment

While there are numerous ways to baseline your environment, SANS wrote a great white paper on it which is still useful today. However, you can’t keep the same baseline forever, and it is imperative to keep this image updated to stay current and avoid noise in the analysis. For instance, if you aren’t updating your baseline with routine OS updates, you could end up comparing different Windows machines with varying updates – which will cause many false positives.

In the end, if are you able to keep an actively up-to-date baseline image, then what comparisons should be prioritized between the baseline and active image? It is far too complicated and inefficient to start comparing every system file and running process.  These comparisons are riddled with false positives, so it’s important to only compare artifacts worth investigating. As we’ve previously written, any discussion of malware quickly turns to the challenges of persistence. So why not compare only persistent files? This is a much more manageable approach which entails only collecting persistent artifacts in our baseline and then comparing differences. This will more broadly illuminate potentially suspicious persistence items. Even if you are worried about user installed applications, outlier analysis will assist you, assuming that there is some consistency in different business units at your company. As always this consistency may not exist, and introduces an additional factor for consideration when building your baseline.


From Baseline to Hunt

Now that you have established your baseline, there are many open-source methods for conducting in-depth delta analyses to help guide the hunt. While the list of approaches is long, it’s most useful to start with Powershell and SysInternals Autoruns. To do this, simply collect persistent files using sysinternals on your gold image and store those results. Next, you could execute remote scripts to collect additional autoruns collections in your environment (check out our previous blog for more information on Hunting on the Host). Compare the results of these scripts to your gold image and highlight the differences. You might get lucky and find something interesting without using any logic!

Here is a quick example of how to compare MD5 hashes from persistent files:

  1. On the gold image execute: PS>.\autorunsc.exe –a * –h | select-string MD5 | sort-object -unique > baseline-autoruns-output.txt
  2. On the target host execute: PS>.\autorunsc.exe –a * –h | select-string MD5 | sort-object -unique > target-autoruns-output.txt
  3. Compare: PS>compare-object -ReferenceObject $(get-content .\baseline-autoruns-output.txt) -DifferenceObject $(get-content .\target-autoruns-output.txt

To optimize your chances of success, you should put some logical thought into what you collect. If you store too many variables, it simply becomes too unwieldy for manual analysis, while too few risks omitting some key concepts. In the above example, we simply looked at MD5 hashes and only examined unique occurrences of a hash, but you could expand this logic. Starting with the persistent location is a great first step, but you should also consider expanding to hashes, filenames, signer information, etc. as additional useful data to enhance your comparison. If you need to utilize outlier analysis, you may want to include counts as well. There are plenty of third party applications that may not be in your baseline, but will be installed by your users. Let’s just hope malware isn’t installed broadly to ruin our outlier analysis!


Baselining with Endgame

Manually constructing that gold image can take significant time, especially at the enterprise scale and if there are different environments across business units. Using Endgame, you can create a baseline by investigating a clean image. As our previous posts and videos have shown, an Endgame investigation is an aggregation of hunts or surveys to include persistence, process, network, applications, user surveys, and more. Now with this baseline investigation, we can survey those production workstations and compare.

Rather than using our UI, in the referenced video I’ll show how we can do all this using our RESTful API in a few simple steps. As we hunt with Endgame, each investigation is given a unique UUID. Using the UUID from the baseline investigation and our target system investigation, we can compare all the tasks and collections that were executed through the Endgame platform. All of our collection data is stored as JSON, which enables these simple comparisons. For instance, you can look for those pesky persistent files that were not in the baseline. Any differences you find may indicate something malicious.




A solid baseline is necessary to executing hunts based on finding deviations from an established baseline. This approach can quickly help identify those key areas that look suspicious and reduce false positives in relatively homogenous environments. Unfortunately, most of us don’t have the luxury of obtaining a snapshot of an enterprise environment, so we need a shortcut to creating this baseline quickly and precisely to get into the more interesting aspects of the hunt. Endgame’s Investigation feature provides this shortcut, intuitively allowing you to compare the baseline investigation against an investigation with the new tasks and collections, providing a quick means to exploring any differences. For us, the API turns into an analytic haven, helping us structure that baseline and quickly leading us to potential malicious activity.