How to go threat hunting: The thrill of the hunt

Let's go threat-hunting. When the hunters and tools have been assembled, let us explore the thought processes that prepare them for a successful hunt, as well as a proven methodology called the hunt chain. This depicts the entire threat hunting process. 

The hunters may be familiar with operating system (OS) internals, and been inside the machine. Maybe they have written some of their own tools and exploits. They're reading about cyber crime and are no longer going to take it! Time for the chase and putting cyber criminals on the defence.

The threat hunter knows how systems work, how attackers think and act, and how to use tools to find them and kick them out. An organisation has its weak spots and this might give cyber criminals an easy way in. But the hunter knows where they'll strike and lie in wait for them.

The primary objective of threat hunting is asset and information protection through:

• Knowledge of systems, networks and exploits
• Knowledge of the enterprise applications, how they work, where the treasures are, and how the data flows
• Knowledge of endpoints, how they work and how they're used.

A threat hunter continues to stockpile his / her knowledge, skills and tools. With the right tools, each new query becomes another automatic threat detector, so the hunter slowly gains ground and denies attackers access to more and more attack surface. This way, a threat hunter never needs to hunt for the same thing twice. Yet attackers' tools improve, and more exploits are discovered, so it's a tug of war between threat hunters and their adversaries.

Constantly reading and learning about new exploits, threat hunters test out new hunches and see whether attackers are trying these new techniques and, if so, what they look like.

Planning the hunt At first a threat hunter becomes oriented to the environment and masters the tools used and how they're configured. Soon it will be time to venture out on individual campaigns — probing deeper and further than before.

While threat hunting is continuous, it is broken up into individual missions called hunts. A hunt can last a few hours to several days, depending on the objectives. A hunt should have one or more objectives, narrowly focused at times but not too broad either. Hunt objectives might include:

• Hunting for specific exploits: A threat hunter may have read about a specific new exploit such as Locky, and will look broadly in the environment for signs of it.
• Attacks against specific vulnerabilities: A threat hunter dives into high‐value systems with one or more known unpatched vulnerabilities to see whether attackers are attempting to exploit them.
• Attacks against specific high‐value targets (HVTs): Here the threat hunter dives deeply into the operation of a specific asset (or a small number of them), learning more about how it operates and looking for signs of reconnaissance or intrusion.

Threat hunters generally focus their attention on endpoints with tools that provide detailed forensic data on these. Depending on the hunt's objective, the threat hunter may be triangulating attack evidence by using additional tools, such as an intrusion prevention system (IPS), web proxy filter, or next‐gen firewall to identify signs of compromise.

As well as detecting malware, threat hunting also tracks abnormal usage of legitimate tools (such as PowerShell and EMET) and accounts.

Keep notes on threat hunting experiences. Over time, hunts may become a blur, but with good records you can go back and familiarise yourself with past hunts. The records might be highly structured and include hunt objectives, logs, traffics, activities searched for, and analytics. Or they might be more like a narrative describing a hunt.

Launch a similar hunt in the future, and your records can be used as a springboard.

The hunt chain The hunt chain comprises a series of activities that constitute a formal threat hunt. The overall chain is depicted below.

Where to start A threat hunt starts with the collection of data that's directly or indirectly related to its objective. When developing an objective, the threat hunter needs to know what data will be mined in order to achieve the hunt's objective.

Define objectives and the scope for a hunt to quantify success and know when the hunt is completed. Without clear objectives, a hunt is more of a fishing trip that could go on and on.

As threat hunters begin observing the target environment, they begin noticing activities. By using their knowledge about the OS and application(s) in the target environment, they begin to filter out legitimate activity, leaving only anomalous activity to investigate. One by one, as those activities are explained, all that remains, if anything, are attackers and their actions.

Suspicious activity During the hunt, the threat hunter observes data and filters out known legitimate activity. Anything that remains could be suspicious. For example, an organisation might utilise PowerShell as a part of its endpoint management tools. PowerShell is a command line shell and scripting language; you could liken it to the new and improved version of command line and batch files.

A threat hunter can use this knowledge to filter out all the organisation's legitimate use cases for PowerShell. If any uses of PowerShell remain, they either belong to additional legitimate use cases, or attacks. Remember that threat hunts don't always turn up activity indicating intrusion.

Deeper investigation Activities that remain unexplained are investigated further. The threat hunter may need to seek help from experts on the OS, applications, data flows, use cases or other aspects of the anomalous activity. Often, the hunter discovers previously unknown aspects of legitimate activities.

Sometimes the threat hunter discovers aspects of an environment that represent improper implementation of a system. For example, he/she might find persistent temp files containing credit card numbers, where the files were supposed to be encrypted but weren't. This may have been considered an artefact of an attacker scraping credit card numbers out of an application.

This portion of the hunt chain is iterative; as threat hunters investigate anomalies, they filter out legitimate activities and then resume hunting for illegitimate activities.

When anomalous activity is observed and confirmed as an attack, the threat hunter continues to investigate to see where and how the attack originated and proceeded. This is essentially a root cause analysis which, depending on the attack, may narrow into an initial intrusion, but it may also branch out into an investigation into what could be a broader attack on more systems.

Remediate After the full extent of an attack is known, the threat hunter, often together with appropriate colleagues (system engineers, network engineers, security engineers, software developers, and maybe others), contributes to the remediation effort. The specific activities vary, depending on the nature of the attack, but the general principles are:

Remove malware and restore all altered and removed files to their original state Update configurations, permissions, and software versions to prevent a similar attack in the future Apply security patches to prevent similar attacks Update defences.

The organisation needs to update its defences so that similar attacks require greater effort. Updating includes automating systems to look for what you found. The range of activities may include:

• New or updated firewall and IPS rules
• New or updated alerts in a security incident and event management (SIEM) system
• Improved incident response procedures
• Updates to infrastructure, application, or security architecture
• Changes in application development, testing, quality assurance (QA) or quality control (QC) tools, and processes
• New alerting rules in endpoint detection and response tools.

The investment in threat hunting tools and personnel is mostly wasted if there isn't a feedback loop incorporated that illuminates lessons learned and updates defences. A threat hunt can discover inside threats as well as outside attackers.

Threat hunt results will give the hunter a pool of ideas for future hunts. If you're fishing in a pond and find a spot where fish are biting, you will be going back to that spot next time.