On May 23rd, 2023, Barracuda announced a vulnerability (CVE-2023-2868) in their Email Security Gateway appliance that was being exploited in the wild as far back as October of 2022. On June 15th, 2023, Mandiant released a detailed analysis of the activities and threat actor involved in the campaign that was actively exploiting CVE-2023-2868. Subsequently, Proofpoint’s Emerging Threats team released a Suricata detection rule (SID 2046280) for detecting attempted exploitation of the reported vulnerability. During a meeting with a Vectra AI client, a concern was raised that the existing rule may not be performing as expected. After conducting initial testing, we found the rule failed to alert on a specific proof-of-concept exploit, despite successful delivery of the exploit payload.
We analyzed the rule and related exploit and identified the specific detection gap, which was caused by a non-deterministic ordering of exploit related content within the exploit payload (more details are provided in the “A brief look at the proof-of-concept exploit” section below). With the gap identified, we were able to rewrite a rule to provide the necessary detection coverage. After submitting our findings to Proofpoint’s EmergingThreats team they released a new M2 detection rule. This report provides an explanation of the analysis and rule writing process used for the M2 detection rule that eliminated the confirmed false negative.
TL;DR
Proofpoint's ET rule is flawed due to a built-in assumption that everyone follows rules, even those who don't. In this analysis, we encountered a manipulated TAR archive that slips a payload past ET's detection rule for CVE-2023-2868 because the exploit author didn’t follow the rules. And even with an improved detection rule in place, the concern remains: if we encounter another manipulated file structure tomorrow, could it bypass ET’s detection rules again?
1. Anatomy of CVE-2023-2868
Mandiant’s report provides an explanation of the vulnerability but for convenience, we’ve summarized it with the illustration below. As you can see, certain versions of the Barracuda Email SecurityAppliance are vulnerable to an injection vulnerability.
In order to identify this malicious payload, the Emerging Threats Research Team developed a detection room that searched for the “’`” (single quote, back tick) and “`’” (back tick, single quote) in particular packet types. As we will see, their rule construction method allowed for some Proof of Concepts to go undetected.
2. A brief look at the proof-of-concept exploit
Turning our attention to the proof of concept vulnerability (PoC) exploit, available here, we started to identify what underlying payload is used and should trigger the Emerging Threats Detection Rule. As shown below, the “CMD” variable uses the same basic command structure that was described in Mandiant’s report:
Note that the variable “PAYLOAD” starts with the single quote and back tick and ends the“CMD” variable with the back tick and single quote. This is what will trigger the command injection.
A look at the PoC over the wire in hex
Here’s the output of a hexadecimal representation of the generated tar archive containing the payload shown above:
“PAYLOAD”, which are the injection patterns, are highlighted in red. The command string assigned to the “CMD” variable is highlighted in blue:]
Observe at the very top of the archive that some left over of the CMD variable are present, as well as the injection patterns \x60\x27 (back tick, single quote). We will return to this important part later.
A normal Tar archive structure
Before diving into the defects of the detection rule, it is important to understand the structure of a regular tar archive. Tar archives are organized into blocks of 512 bytes. Basically, the format of a tar archive is:
- “Header” contains the name of the file, plus some additional metadata.
- “Content” contains the actual data within the file.
A peek into a normal Tar archive over the wire in hex
What’s considered a “normal” Tar archive is a file that has been created by a utility that follows the rules and respects either the TAR format from UNIX systems or the USTAR format defined by the POSIX 1003.1 standard. This is a hexadecimal dump of a normal tar archive containing three regular files named 1, 2 and 3, using tar utility:
We can see the file named 1, followed by some byte codes and then the file content “hello 1, ”which is followed by the file named 2 and so on. Clearly, tar archive creation utility puts the header and content data in descending order based on the names of the files. In other words, it is mindful of the rules dictated by the file format standard and respect them.
Of note, the file structure is different from the one generated by the exploit and presented in the first section. The PoC exploit contains a series of commands in the header instead of files names as expected. To make this possible, the exploit writer had to get a bit crafty.
Dissection of the R7 exploit
The exploit written by the Rapid7 Security Researcher team has a dedicated part, with comments, on the “TarWriter” Ruby class:
This exploit overrides the filename size checks that are originally done by the "split_name()”function here:
3 “if” statements have been removed from the original function. These 3 statements raise“TooLongFileName” to verify if:
1. The file path is too long
2. The file name is too long
3. The file’s base path is too long
Looking at the payload of the exploit, and counting the characters gives 129 bytes:
Based on the code in the PoC, if the command to execute is more than 100 bytes, the command will be split across multiple “file entries” in the tar archive. We put “file entries” in double quotes because the malicious tar archive doesn’t actually contain files – it contains an escape sequence and embedded commands that the attacker wants to execute.
Revisiting the hexdump of the malicious tar archive, we can now understand why the first entry in the tar archive is a dangling `’ (back tick, single quote) and the end of the CMD variable, which should come at the end of the command.
Since the command exceeds the 100-byte limit imposed on the file name structure for a file’s header, the command was split into two file entries with the second file entry containing just the ending p”`’ (letter p, double quote, back tick, single quote). That file entry appears at the beginning of the tar archive because the tar archive’s contents are sorted in descending order and a p”`’ (letter p, double quote, back tick, single quote) string comes before a ‘`s (single quote, back tick, letter s) string when sorted in descending order:
While the ordering within a tar archive is consistent, the ordering of the escape sequence and command contents of the exploit payload will be non-deterministic from the detection rule’s perspective because the order will vary depending on the combined length and characters used in the commands embedded within the payload.
3. High level explanation of the detection rule:
Once we had a better understanding of the vulnerability, what it takes to exploit it, and confirmed what patterns need to be discriminated against within the exploit, we turned our attention to the detection rule from the emerging-exploit.rules file:
To simplify, this rule looks for 3 patterns:
1. Any SMTP packets coming from anywhere, destined for any SMTP server contained in the$SMTP_SERVERS mapping. Note that by default, Suricata maps $SMTP_SERVERS to $HOME_NET so any internal SMTP server will be covered by the rule.
a. That contains a valid tar archive attachment; Attachment must match the file magic numbers of a TAR archive.
b. That contains within it these two patterns “|27 60|” and “|60 27|”.
The searched keywords are the hexadecimal representation of the following ASCII characters“’`” (single quote, back tick) and “`’” (back tick, single quote). The characters “’`” (single quote,back tick) are used to trigger the command execution as explained previously. Below is the hexadecimal representation of the exploit with the 3 highlighted patterns that the detection rule is looking for:
Looking back to the rule, we see that there is also a “distance” and “within” parameter that can be paired to fine tune a rule. The parameters are used as follows:
- Distance: sets cursor to where to start to analyze.
- Distance: 0 means the pattern can be anywhere after the previous match +0bytes. So right after \x27\x60 in our example above.
- Within: Limits how far after the last match the rule needs to look forward.
- Within: 500 means there will only be a match if the content matches with the payload within 500 bytes.
The within and distance parameters are explained in the Suricata documentation.
4. Defects observed in the original rule compared with our exploit
As the adage goes, “a picture is worth a thousand words”, a dump below shares how the detection rule behaves into the Rapid7 exploit:
This snippet of the rule:
The dump above easily depicts why the detection rule never triggers: it will, in our sample, never find the other pattern as it is positioned in the beginning of the archive.
5. A variant M2 detection rule for this undetected exploit
In light of the above findings, a new detection rule M2, revision 2, has been published on September 29th by Proofpoint’s Emerging Threats team as SID 2048146:
Below is a rundown of what the M2 detection rule checks, using the same hex representation of the same tar archive exploit as before:
6. Resolution
In light of the evident shortcomings of Proofpoint's original ET rule (SID 2046280), it becomes clear that relying solely on fixed rules and standards has its limits in the ever-evolving landscape of cybersecurity. General MacArthur's timeless wisdom, "Rules are mostly made to be broken,"serves as a stark reminder that malicious actors will exploit any vulnerability, including rigid rule sets.
Through this analysis, we have witnessed a prime example of the limitations of this approach, via the manipulation of a TAR archive structure that allowed a payload to slip past ET's initial detection for CVE-2023-2868. This event underscores the urgency of adopting a more adaptable and dynamic security strategy.
As we look to tomorrow, it is imperative to shift our focus away from rigidity and embrace a more holistic, proactive security paradigm. Instead of merely pondering the odds of another rule breach, we should actively strive to enhance our cybersecurity measures by constantly evolving our detection and defense mechanisms. By acknowledging the limitations of fixed rules, we can better prepare for an uncertain future where cyber threats continually evolve. In doing so, we can strengthen our security posture and better protect against malicious actors who persistently seek to bypass the rules. Adopting a strategy that goes beyond reliance solely on rules-based technologies not only aligns with the multi-layered approach of Defense-inDepth but also beckons the reader towards embracing the currently most effective strategies, including the Zero Trust Model.
For the sake of providing an immediately actionable recommendation, we offer the following guidance:
For Suricata deployments that are using the emerging-all.rules file and have automated management of ruleset updates, the revised detection rule should already be in place, assuming the updater has been executed after 9/29/23. To verify that the currently applied ruleset contains the revised detection rule, search your local ruleset files for “2048146”. If SID 2048146 is not present in any of the currently applied rulesets, update the rulesets to the latest available source from Emerging Threats.
Timeline:
- 9/19/23: Submission of the findings via the ET portal. Did not receive anything back to acknowledge submission.
- 9/21/23: Sent an email to support@emergingthreats.net in order to log the submission and inform the Emerging Threat team of the Google responsible disclosure policy that will be followed here.
- 9/21/23: ET Security Researcher emailed me back and agreed to release a M2 version of the detection rule as part of their daily publication, today 7 PM ET.
- 9/21/23: Noticed that the newly published M2 variant detection rule doesn’t trigger an alert against R7 exploit.
- 9/21/23: Emailed ET Security researcher about the failure to detect the exploit payload with the newly released M2 detection rule.
- 9/29/23: A revision 2 of the M2 variant rule released at 7PM ET and successfully detectsR7 exploit payload.
Reference:
Suricata documentation: https://docs.suricata.io/en/suricata-6.0.1/rules/payload-keywords
html#withinhttps://docs.suricata.io/en/suricata-6.0.1/rules/payload-keywords.html#distance
Github POC exploit: https://github.com/cfielding-r7/poc-cve-2023-2868/blob/main/poc_cve_2023_2868.rb
Exploit explained: https://www.mandiant.com/resources/blog/barracuda-esg-exploited-globally
ET Rule: http://rules.emergingthreats.net/open/suricata-5.0/emerging-all.rules