In the Trenches: Working as a Security Analyst

by woland

Since we can't seem to go one day without reading about a ransomware onslaught, a supply chain compromise, or a database dump of private user info, I wanted to write an article about one of the more under-appreciated roles in the amorphous blob that is the information security industry.

Yes, I'm talking about security analysts, the people/hackers who work on the front lines with the common goal of detecting and preventing attacks.

What Do Security Analysts Do?

In the most basic terms, security analysts review and investigate alerts relating to potential malicious activity and determine if those alerts are legitimate concerns (true positives) or not (false positives).  In a way, it's like being a digital security guard.  It can sometimes be a thankless and tedious job, but that doesn't mean it's not an important one.

Most analysts work in a Security Operations Center (SOC), a (usually 24-hour) base of operations used to monitor digital activity.

Some SOCs even track physical events.  Enterprises and government agencies normally have SOCs in-house or they pay a third-party to monitor events for them.  Depending on the company and the usage, SOCs might be set up in an office environment or, as the trend grows, operated and staffed remotely (especially after China COVID-19).

While all SOCs are different, they often function in a similar way.

Security Information and Event Management (SIEM) software is a common tool of the trade.

When deployed, it collects logs from multiple hosts on an organization's network.  The SIEM collects different logs, and engineers program rules for those logs in the SIEM that determine what alerts the analysts will see.  For example, if an admin user on the network has 50 instances of a log showing an authentication failure to the same server within a specified time period, that information could be turned into an alert that is sent to the analysts in the SOC.

Analysts can then review the details in the alert and use the SIEM to examine the log data and, based on predetermined SOC triage procedures, they decide if there is a security threat.  If a threat is found, they will escalate the alert to a higher level in the organization or to the client they're working for.

Logs, Logs, Logs

One of the best ways to learn how a system works is to read the logs it creates.

If you're an analyst, you're probably going to be exposed to a shit ton of logs, especially if you're working for multiple clients.  It's entirely possible for an enterprise SIEM to work through 100,000 plus logs a second and have more than 30 GB of data each day.

On any given day you may investigate logs from firewalls, operating systems, endpoint detection/anti-virus software, VMs, routers, containers, proxies, SNMP traps, authentication frameworks, or anything else living in a network.  This provides an analyst with a keen perspective and, as you gain experience, you begin to understand what specific attacks look like based on the logs alone.

Of course, that's if everything works like it's supposed to, and it often doesn't.

You can only see certain logs if they exist.  Maybe debug logging for the Netlogon service hasn't been enabled on a domain controller.  Maybe verbose logging hasn't been permitted for that one Linux server.  Some organizations have terrible inventory management, and don't even know what devices are on their network.

There are times where you won't be able to tell what is going on because there's just no visibility with the logs available.  Then there are times when the SIEM won't parse logs correctly, which can lead to confusion.

With so many logs being correlated with SIEM rules to create alerts, tuning out false positives and expected activity is also critical to preserving the analysts' sanity.  It's a SOC team effort to find non-threats and silence them from alarming.  Otherwise, this leads to a deluge of useless alerts and noise, which creates alert fatigue and a delayed response to legitimate threats.  If that happens, attrition will take a toll on the analysts and security is eroded.

Threat Intelligence Is Key

Since a SIEM is dependent on programming rules for log data that will trigger alerts, it's important to emphasize just how important threat intelligence is for creating those rules.

Not only is it crucial to know the common tactics used by attackers to exploit a system and move across a network, but you must continuously stay on top of all the new attacks that are out there, and the Indicators of Compromise (IOCs) that offer clues to identifying those attacks.

Some SOCs have their own threat intelligence teams, and some will pay third-parties for the intel.

You could be working as an analyst and receive an alert that a user account on a host is using a command line interpreter to execute a process injection technique.

Then you may get an alert that the same host is calling home to an external IP address that is believed to be a command and control server that has deployed ransomware in the past.  If you act swiftly, you might be able to block that traffic before anything bad happens and quarantine the host, and that's a wonderful feeling.

The reason you saw the alerts in the first place was because of threat intelligence that was added to the STEM.

To make things more complicated, threat intelligence is constantly changing.

A parked domain or random IP address can go from innocuous to malicious in a short period of time, and back again.

Threat intelligence is its own industry, but as an analyst you still have to understand its value.

Real Talk

No security apparatus is foolproof, and a SOC with analysts is no different.

Attackers can evade detection.

They could use a trusted binary file on an operating system as a proxy to execute a malicious file, not triggering any alerts.  They could obfuscate or encode a malicious command that will go unnoticed by analysts.  Maybe an attacker does their research and knows when the shift change from night to morning occurs, and they choose that time to strike.

There are several techniques that can be used to do things quietly, and there is no one-stop solution for security.  If you are a human making countless judgments on hundreds of logs each day, it is inevitable you will make a mistake.  The first few might hurt, but you learn from them and move forward.

Most alerts you see are going to be false positives, and many analysts tend to spend hours looking at their monitors, so the work can be draining and repetitive.

Even when you do escalate something that is a valid threat, there is no guarantee that it will ever be fixed or that you will get a response back.  It could take months of escalating the same issue before it's acknowledged that it was malicious behavior and stopped.

Despite these hurdles, analysts trudge on.

You live for the infrequent moments where you are, in some small way, responsible for stopping what could have been - those moments where all the stars align, and you can put out a fire before it becomes a headline.