In this three-part series, ParaFlare’s Director of Incident Response, Shanna Daly, and Cyber Security Analyst Aaron Williams detail the importance of using multiple approaches for detecting adversaries and performing forensics on systems affected by web shells, because relying on one method or tool is not enough.
The threat of web shells continues to grow as organisations rapidly shift commerce online and leverage growing numbers of platforms and technologies that are digitally enabled and served over the web.
To assist in the hunt for web shells ParaFlare has released a Windows compatible tool that assists in automating the detection of web shells.
In Part 1 of our web shell series, Shanna muses over the idiom ‘having more than one string in your bow’ and how that sums up not only the highs and lows of Digital Forensics and Incident Response but the importance of using multiple tools to identify and investigate the growing threat of web shells. In Part 2, Aaron details two common web shell hunting methods – string matching and entropy and in Part 3, Aaron expands on more complex methods including de-obfuscating strings, long lines, and over used variables.
HAVING MORE THAN ONE STRING IN YOUR BOW.
Incident response is a journey, where often things do not work quite the way you want them to, when you need them to. Each job presents its own challenges, requiring you to adapt your approach and leverage different skills and toolsets to achieve your objective. This article is about a recent web shell investigation and illustrates the importance of ‘having more than one string in your bow’ when undertaking digital forensics and incident response (DFIR) engagements. We discuss the complexities involved in DFIR work, the tools of the trade and our decision-making process regarding why we chose to use specific tools at the time.
The year 2020 will go down in history as a year of many challenges for most people, but for us at ParaFlare it will also be remembered as the year of the resurgence of the web shell.
For the uninitiated or curious, a web shell is a malicious file that is loaded onto a public or external facing web server (though not always externally facing).
This malicious file generally allows backdoor functionality to the web server enabling an intruder to perform many administrative actions such as browse directories, download files, upload files and pass commands to the web server and potentially other connected backend servers. With more and more organisations pushing for greater digitisation of their products and services, web shells present significant privacy, commercial and reputational risks for businesses as they connect front-end web sites to back end systems.
In early 2020, ParaFlare assisted an organisation with investigating a breach of their public facing web server after a company employee identified their website had been defaced while performing server updates. A malicious file was also detected by their Anti-Virus (AV) tool around the same time. A web site defacement is similar to writing graffiti on a wall, a hacker had gained access to the web server and changed the contents of the web site to brag about their skills. The web server had definitely been compromised.
Our team was asked to find out what happened and how. So we began our analysis, starting with the web server and available file system logs that thankfully covered the time of the breach.
Our initial approach for the investigation was to fire up X-Ways Forensics – an extremely powerful forensics tool. Using X-Ways’ timeline feature we were able to quickly identify the malicious files including where and when they were loaded onto the web server. In total there were five web shells and hundreds of web pages loaded with defacement messages. The analysis gave us the file system names of the web shells, their location, the timeframe they were loaded onto the server and when they were accessed by an external party.
Worryingly, this analysis showed us that the original breach happened in late 2019 – approximately five months before it was detected by the organisation. This meant that the nature and scope of data requiring analysis expanded significantly and we needed to use a different tool for the next stage of our analysis to enable us to efficiently identify pertinent information and anomalies in large volumes of log data. To gain a deeper understanding of what had happened we would need to use the Elastic/LogStash/Kabana (ELK) trio, a toolset often used to examine and draw meaningful insights from log files.
As a practitioner of DFIR I have come across and used a plethora of tools, but my ELK skills are not the strongest. Fortunately, my colleague Daniel Eden (co-director of Digital Forensics at ParaFlare) is a whizz with ELK. So, we dumped the web server logs and parsed them in ELK with relative ease. Using a basic log stash configuration file, we were able to input the web server logs into an ELK instance that we ran up internally.
The beautiful thing about ELK is that it easily allows you to search, correlate and graph data to identify common names, user string agents, time frames and the amount of data sent and received from the web server for each connection. Sure, you can see all of this in grep too, but ELK makes it easier to spot anomalies through data visualisation.
By using ELK for this stage of the analysis, we identified an illegal streaming service that was running on the server based on the incredible spike in data transfer beginning in November last year. Now we had identified what happened, when it started, the malicious files, a list of IP addresses used by the unauthorised actors, geo locations of those IP addresses and most importantly we were able to get a good idea whether any data was taken from the server.
As a combination of good practice and curiosity we decided to run Anti-Virus on an image file of the server. Yes, we still use AV scans in Digital Forensics.
There are multiple approaches to mounting an image for scanning. Tools such as FTK imager or Arsenal Image Mounter allow you to mount the image within Windows for scanning. What this means is that all the files that are available on the image will be viewable as files within the host Windows system. The downside of these tools is that when they mount the image files, the permissions for the mounted files remain active and certain files and directories may be omitted from being scanned as they are protected by the operating system.
A better way is to mount the image as a drive letter using X-Ways Forensics. This method makes a copy of the files available in a directory as the mounted image, overcoming the permissions issue and therefore makes every single file available for scanning. If you do not have X-Ways, you could consider performing an AV scan against an image mounted in Linux.
The exercise of performing an AV scan against the mounted image demonstrated how ineffective AV tools can be at detecting web shells. In this case, the AV scan found zero web shells in comparison to the five web shells we had identified during our analysis.
Our customer had AV and commercial Endpoint Detection and Response (EDR) solutions in place at the time of the breach – which unfortunately had not retained the logs from the period in question so we weren’t in a position to confirm if it had alerted to the issue or not. To add heat to the burn, the client also had a Managed Security Services Provider that had not alerted them to any issue.
Luckily for the client no critical information had left the server and we were able to identify, contain and close out the issue quickly.
If someone has more than one string to their bow, they have more than one ability or thing they can use if the first one they try is not successful. – Collins Dictionary
Several lessons were taken away. The first being that as investigators it is important to have multiple avenues, skillsets, and tools available – often we have no starting point in a breach and potentially we have no idea of the scope, time frame or what road blocks or issues we might face in obtaining a forensic image file, analysing it or extracting relevant artefacts from it.
The increased frequency of web shell incidents that we have investigated prompted ParaFlare to develop a web shell detection tool (see Part 2 of this blog series for more details) that we are sharing with the community to help improve the detection and eradication web shells.
We have added YARA and NeoPi to our toolset. YARA is a malware detection tool that relies on searching for common characteristics of malware over the usual signature-based approach. The benefit of YARA is that it focuses on searching for malicious intent within combinations of files and their contents. NeoPi is a tool that uses statistical analysis of files to determine what might be a likely web shell.
For those who are looking after security the lesson is similar – the combination of multiple tools and techniques is important to provide holistic and in-depth detection and security.
Should we look back at maybe what would have either prevented this incident, or sped up the detection, we could consider the following:
Centralised Logging and Analysis – Ensuring that logs are stored centrally and monitored. This should include web log analytics as the spike in web traffic we observed would have stood out like a sore thumb and possibly led to earlier detection.
File Integrity Monitoring (FIM) – A FIM solution could prevent or at a minimum detect malicious files loaded into directories within a web server file structure. This may not always work for every directory if the contents is dynamic or files are uploaded by third parties, but extra scanning of those directories with a tool like ParaFlare’s web shell hunter would provide an additional layer of detection capability.
Endpoint Detection and Response (EDR) – EDR should in theory be able to pick up web shells. We could not verify in this case; however, it may be that the alerts were missed. Though, what we do know from our experience is that without a person assigned responsibility for writing detections for the EDR tool, and looking at the resulting alerts and conducting threat hunting, an EDR tool on its own, out of the box may not provide detections for an incident like this.
Cyber security is extremely dynamic, with constantly changing environments and situations and this is why we love what we do here at ParaFlare – but it is also why having more than one string to your bow is important.
