If you are an SE like me, then you have probably experienced a 'priority one' incident response with your customer. Things are on fire and you call in all the reinforcements you can. If you are an IT or security guy, then you have probably placed the call for help. Either way, you will understand.Here's the customer scenario. It's fire drill time. Internet connectivity and applications are going down and everyone is panicking. Your organization has been either compromised by malware or you are being actively attacked. Now is the time that all of your security products need to be working, and working well.
This happened at one of our customer sites recently. The customer was infected by malware and multiple infected endpoints were unknowingly participating in a distributed denial of service (DDoS) attack on a popular search engine. The search engine then blocked all access from our customers' location. This meant webmail, searching, news and other services to this company stopped. Users complained. IT escalated the incident and vendors were called in to resolve a P1 urgent case.
What we discovered is that some of their security systems were not deployed or not configured in a way that would help triage the problem. The IPS system wasn't configured to see internal host IP addresses – it was deployed on the Internet side of the NAT function, which means all the behavior by internal hosts detected by the IPS had the firewall's NAT IP address. The SIEM server didn't have the correlations written to figure out what's going on – even assuming all logs were being updated – so there was no indication of which endpoints were part of the DDoS attack.
I was called because the customer is using one of our Vectra X-Series platforms at another location, though not at the one experiencing the problem. The customer asked us to turn on a second platform at the site experiencing the incident to identify the endpoints involved in the DDoS attack.
With a new Vectra X20 platform up and running in less than 15 minutes, we immediately started seeing detections for internal hosts. We quickly found patient-zero – it exhibited behavior that correlated with the outbound activity seen on the IPS, which means we now knew the responsible internal host. The host was relaying connection requests from a command and control server on the Internet to the DDoS target's IP addresses based on instructions it received. The Vectra X-series also detected other abnormal web behavior for the first host and detected several more infected hosts within 20 minutes of being powered on. See how it works. »
Our customer was able to quarantine the infected devices, the company running the search engine unblocked their domain and business went back to normal within a very short period of time.
What is the lesson in all of this? Make sure your security devices are seeing the appropriate traffic and more importantly, can easily correlate intelligence across your security systems. Stuff will get through your perimeter defenses and being able to determine which internal systems are misbehaving has to be something your security operations team can do every day. I mean, you spend all this money on threat intelligence systems, so you should get their full value.