UnderDefense is a Security-as-a-Service (Sec-a-s-S) & Compliance platform that has been giving a stellar performance in security services, meeting and exceeding our clients’ expectations. 

Now we have indisputable proof of our consistent excellence and professional expertise and official confirmation of our risk managing efficiency.

With a sense of immense pride and a thrill of excitement, we would like to report that in November 2021 UnderDefense received ISO 27001:2013 certification which is easily recognized all around the world and increases business opportunities for organizations and professionals. 

After extensive audits carried out by Bureau Veritas Certification Holding SAS-UK Branch, the UnderDefense IS Management System was found to be fully compliant and able to meet all best practices for serving customers` needs.

This achievement demonstrates UnderDefense’s continued commitment to protecting customers’ most valuable assets because our target is proven excellence for all.

Why This Is Important

Ukrainian cyberwar has become a great platform where the US government and commercial sectors can learn the best protective measures. 

Since the Russian-Ukrainian war broke out, Russian hackers have been focusing their attention and cyber attacks on Ukrainian government institutions as well as on civilian targets, keeping Ransomware attacks more like a source of financing for their day-2-day life. 

Over the first six months of 2022 only, about 1350 cyberattacks have been detected by the Computer Emergency Response Team of Ukraine (CERT-UA).

The top sectors targeted by Russian hackers are the following:

• Government and local authorities
• Security and defense sector
• Energy sector
• Financial sector
• Commercial sector
• Telecommunication sector and developers
• Transport sector

The most widespread types of cyberattacks include:

• Malicious code and Implants
• Intrusion
• Intrusion attempts
• Violation of information properties
• Accessibility disruption
• Harmful (abusive) content
• Known vulnerability (this case)
• Ransomware

UnderDefense together with the CERT-UA team is releasing this Cybersecurity Advisory to give a deep insight into one of such attacks. In this material, we would like to share a fascinating story where EDR software detected an initial foothold but missed other TTPs which were later discovered by UnderDefense MDR Team (back then we had just started providing Pro bono services for that Ukrainian government organization) and the attacker was finally kicked out. 

This time, the attack was targeted at the mail server of the Regional Military Governance Organization, more precisely on Zimbra mail service 9.0.0 patch 23.

The attacker’s innovation was in their persistence and codebase, bypassing CrowdStrike and attempting to keep remote access as the target was very important for them (especially because it was a Mail server).

Malicious actors (in our case APT groups) were constantly scanning Domain and IP space for both well-known and recently discovered vulnerabilities.

Notable Facts & Adversary TTPs used in this Attack

• The active attack phase began at night, all noisy and risky actions were performed outside of the IT team’s working hours
• More than 40 IPs took part in the attack and multiple actors acted in parallel, cooperating in real-time
• A lot of defense evasion techniques were seen throughout the Kill Chain phases, with at least 20 backdoors left on the disk and in memory
• A really interesting and innovative C&C communication scheme was established, undetected by all of the tested EDR solutions. C&C communication through a legitimate PubNub proxy, messaging service based on the Publish/Subscribe model. This trick allowed the APT to hide their server’s IP and evade detections based on repeating outbound connections to suspicious domains

The attackers’ ultimate goal was to obtain confidential email communication data from the Regional Military Governance organization, namely employees’ credentials and mailboxes. No data wipe, drive encryption, or critical files integrity violations were performed since the adversaries knew perfectly well what they wanted to get – Military Intelligence and keep stealthy access to all the further mailings.

Attack Success & Failures

Undoubtedly, such a complex, multi-stage, and obviously targeted attack couldn’t have been prevented by a single security tool. Conversely, each security misconfiguration or vulnerability accelerated the breach, making the adversaries’ goal a lot easier to reach. Here are the main reasons why this attack was successful, at least in the initial phase:

• Legacy OS: Ubuntu 16.04
• Vulnerable Zimbra Mail Server: 9.0.0 patch 23
• No MFA enforcement, password-only access
• No 24×7 security monitoring established

And here are the main security tools and attackers’ time-wasters that stopped the breach:

• CrowdStrike EDR: real-time protection against malware
• Network Isolation: kill switch that stopped the attackers
• Web Logs in Splunk: the useful source for activity correlations
• Incident Response: manual threat containment, system recovery

Attack Breakdown

As seen in the diagram above, the attack references more than twenty MITRE Enterprise techniques. Following the timeline, the active vulnerability scan began more than a month before the attack. 

A lot of different IPs were probing the server for web vulnerabilities, hidden services, and open ports, trying password spraying, and sure enough, preparing for the attack.

Initial access

June 18: Password hijacking was performed via one of the Zimbra vulnerabilities (CVE-2022-27924). Exploiting it by sending a specially crafted HTTP request, adversaries were able to redirect all further login credentials to the controlled server. The screenshot below shows one of the crafted requests that forwards login data of admin@loda.gov.ua to the APT-controlled 45.14.227[.]108 IP address. 

While the concept is simple, the attack consequences were devastating. In just a few days, hackers obtained valid credentials for more than five active mail accounts, including the administrative one.

Execution & Persistence

June 19, Night: The password hijacking campaign is over and initial access is achieved.  A valid account is a required step for the following vulnerability exploitation – Zimbra unrestricted file upload (CVE-2022-27925). Interestingly enough, both of the vulnerabilities were publicly disclosed less than two months before the exploitation, indicating the importance of a rapid path management process. This is an indicator of a very good shape of Malefactor toolkit preparedness and awareness.

Using the mentioned CVE, the attackers uploaded three variations of JSP backdoors. Java Server Pages (JSP) is a technology that helps developers create dynamic web pages, basically by running an arbitrary Java code to build the page components. One of the malicious JSP files (originally named skins_bk.jsp) is shown below. All the observed backdoors required two values to operate and work as follows:

Zimbra operates a large number of JSP files in its work, and the adversaries knew it for sure. One of the most challenging tasks during the Incident Response was to distinguish between legitimate and malicious JSP files, mainly because of three observed defense evasion techniques:

• At least 5 insignificant Zimbra files like translations of help pages were replaced with the mentioned backdoors.
• A few legitimate JSP files were modified to include malicious obfuscated one-liners, which were hard to detect by keyword search.
• Backdoor’s original creation timestamps were hidden using “touch -r good.jsp malicious.jsp” command which copies the creation time of the first argument to the second one.

June 19, Morning : Soon after uploading the files, the active phase began. At least three IPs were seen using different JSP backdoors simultaneously, discovering the content of Zimbra folders, analyzing OS and network configuration, and looking through the service user’s permissions. In the meanwhile, threat actors were exploring Zimbra CLI – a set of tools to manage the entire server from the command line.

Credentials Gathering

June 20, Early morning: Threat actors managed to fetch all usernames and corresponding passwords from the mail server. Moreover, they created a few fake administrative users, changed passwords for some of the inactive ones, and generated an authentication token to retain at least programmatic access to the mail server even after the accounts’ reconfiguration.

The attack timeline is worth emphasizing – every noisy action was made at night. Threat actors were silent during the day, and stealthy at night, moving forward only when everyone was asleep. However, to achieve their primary goal, to dump and upload a huge database of historical mailings, a stable network channel had to be established from the target to the controlled server.

Exfiltration Attempts

June 21, 3 AM: The hackers chose one of the noisiest ways to offload their findings. 

A simple bash reverse shell that was immediately flagged and blocked by CrowdStrike XDR. After a few dozen attempts to create a reverse shell with Bash, Java, JSP, Python, Perl, Socat, and Netcat, adversaries finally gave up and uploaded a file called “watchdog”, the one that made this breach one of the most extraordinary cases compared to previous Incident Response projects. 

C&C Establishment

The threat actors shouldn’t be underestimated, especially considering the approach and the sophisticated methods they applied. A great example – is the above-mentioned “watchdog” file. It is a simple Python script, packed with Pyinstaller, that begins the C&C communication. Yet, there are two facts that make it “special”:

• C&C communication through a legitimate PubNub proxy, messaging service based on the Publish/Subscribe model. This trick allowed the APT to hide their server’s IP and evade detections based on repeating outbound connections to suspicious domains
• Packed with a well-known, open-source Python tool, the malware was undetected by all of the tested XDR solutions, with 0 VirusTotal score, even after the beaconing start and command execution were successful

Downloaded and immediately executed, the file bypassed CrowdStrike detection and began beaconing to the ps.pndsn.com domain, whitelisted by most of the Threat Intelligence services as a part of PubNub infrastructure (which is extremely popular).

Through the watchdog script, victims subscribe to a certain PubNub channel and periodically turn to it for new messages. Meanwhile, the hacker can broadcast commands to the channel at any time, forcing the victims to execute the payload locally.

From the initial SOC perspective, a new process is beaconing to the PubNub service. Hash is clean and the source file is located in the expected Zimbra folder, together with the legitimate “swatchdog” script used as a mail logging manager.

Moreover, as you may see on the screenshot below, the file was copied to more than 10 locations on the disk, masking as log files, zimbra updates, and legitimate binaries. Besides, it was persisting across sessions by means of the nohup technique.

 Consequently, the C&C communication was nearly undetectable without manual investigation.

Privilege Escalation

June 21, 6 AM: Multiple IPs were still hiding backdoors in the system, trying obfuscation techniques to actually open reverse shell, and, of course, dumping the mailboxes locally to the /tmp folder.

Yet, a massive dump of government mailboxes couldn’t be sent via PubNub channel, so threat actors had to search for alternative data extraction methods. Restrictions of the zimbra user rights forced them to try to elevate privileges to root by abusing an outdated Ubuntu 16.04 operating system. Having a wide choice of exploits, adversaries chose one of the most popular ones – CVE-2021-4034, also known as PwnKit.

Trying both GitHub-downloaded binaries and locally compiled exploits, attackers still failed to overcome CrowdStrike prevention. Meanwhile, our Incident Response team was already on duty reviewing suspicious CrowdStrike alerts and the investigation began.

Remediation & Containment

The first thing we noticed was the fact that there were more than 15 CrowdStrike detections regarding reverse shell initiation attempts, so network isolation approval was immediately requested. While the request was being processed, both our security team and threat actors were doing their jobs.

Luckily, we were a bit faster and managed to stop the breach before the mailbox dump was completed. Still trying to figure out the root cause of the attack, and still observing malicious actions in real-time, we were discovering and remediating the backdoors one by one: from the memory, hidden folders, legitimate Zimbra files, etc.

As it was previously mentioned, adversaries had a plan to deeply persist on the infected system, and after each new backdoor discovery, it was becoming even more clear that complete threat containment would take a lot of time.

At the same time, the IT team was asked to reset users’ passwords and update all Zimbra components immediately to avoid reinfection. After prompt patching, the system was finally isolated from the network, and our security team could breathe a sigh of relief.

Over the next few hours of Incident Response, the security team was cooperating with local administrators checking the system for file integrity violations, looking for threat actors’ traces, and security hardening on the fly.

The server became fully operational (and patched!) after three hours of Incident Response.

Hardening & Recommendations

Although some of the steps were made during the IR, most of the time-consuming, but still critically required actions were planned for the nearest future:

Issue 1: A list of all created mailboxes was compromised

Proposal: Focus on cybersecurity awareness and personnel training since targeted phishing attacks may be observed in the nearest future.

Issue 2: The password hijacking attack was successful

Proposal: Enforce multifactor authentication (MFA) on login for every mail account, configure a secure password policy and configure access controls according to the principle of least privilege.

Issue 3: Adversaries have accessed some mailboxes via UI

Proposal: Audit login and management activities, most notable logins from suspicious sources, configure access controls according to the principle of least privilege

Issue 4: Two-month-old critical Zimbra vulnerability was exploited

Proposal: Prepare and enforce patch management process for all used services, both for internal and internet-facing ones. Regularly back up critical data and make sure it is stored offline, securely on a separate server

Issue 5: Privilege escalation attempt was performed on Ubuntu 16.04 OS

Proposal: Install updates for operating system, prebuilt software, and firmware as soon as possible. Migrate to a vendor-supported OS like Ubuntu 20.04

Issue 6: Some actions were detected, but not blocked by CrowdStrike

Proposal: Configure the prevention policies, and utilize all the available features of an existing security solution

Issue 7: Initial access, as well as C&C communication, were still undetected by CrowdStrike

Proposal: Configure SIEM to ingest more data sources, enable generic correlation rules and think of the custom ones to cover attack vectors specific to the mail server. Setup real-time notifications about any abnormal activity.

Issue 8: Network-based protection was absent, critical services were publicly accessible

Proposal: Forward the incoming traffic through a firewall with configured IPS to block known attacks, limit outbound network access and think about WAF implementation.

Issue 9: A recovery plan was not in place. Most of the decisions were made on the fly

Proposal: Work on a plan that describes the needed steps for a rapid recovery from integrity or availability violations. Make sure that the workflow is clear and consistent for the IT team.

Issue 10: No 24×7 security monitoring was established, the threat could have been detected a lot sooner

Proposal: Begin implementation of a solid real-time detection, notification, and escalation workflow or entrust the available security tools to a managed SOC team.

Indicators of Compromise IoC

IP

5.8.41[.]132
103.186.159[.]133
45.14.227[.]108

Files	MD5 hash	Description
watchdog	838e313ec92350785f66945456836e26	Initiates connection with C&C the server executes received commands 0 / 58 detections on VirusTotal https://www.virustotal.com/
pwnkit	811069d057625523d8f3c550382062ec	Used for the elevation of privilege by exploiting a vulnerability in Ubuntu 16.04 OS 19 / 59 detections on VirusTotal https://www.virustotal.com/

Mitigation

• Implement a recovery plan that maintains and retains multiple copies of sensitive or proprietary data and servers in a physically separate, segmented, and secure location (i.e., hard drive, storage device, or the cloud)
• Regularly back up data and password protect backup copies stored offline. Ensure copies of critical data are not accessible for modification or deletion from the system where the data resides
• Install, regularly update, and enable real-time detection for antivirus software on all hosts
• Install updates for operating systems, software, and firmware as soon as possible
• Audit user accounts with administrative privileges and configures access controls according to the principle of least privilege
• Enforce multi-factor authentication (MFA)
• Focus on cybersecurity awareness and training. Regularly provide users with training on information security principles and techniques as well as overall emerging cybersecurity risks and vulnerabilities, such as ransomware and phishing scams

Additional Resources

• Read more about our Managed Detection and Response Service (MDR).
• Download MRD Service Datasheet.
• Read more about our Incident Response Service.
• Discover all features of our Security Automation and Compliance Platform WarRoom.
• Need help? Contact Us.

Malicious Actors have a new trend – brute-forcing various remote desktop SaaS services/tools like AnyDesk, GoToMyPC, Zoho Assist, RemotePC, VNC Connect, DameWare Mini Remote Control, etc. to gain unrestricted access to internal corporate networks without any obstacles. It is as simple as that, an IT guy who left a company a few years ago made that backdoor to avoid the hassle of supporting some annoying employees. This is our legacy, this is our harsh reality.

New ransomware-as-a-service (RaaS) teams like BlackCat (also known as ALPHV) target Remote Access Web vulnerabilities in unpatched or outdated firewall/VPN devices.

Here are a few things to keep in mind in order to Detect, Contain, and Eliminate such threats:

• An efficient MDR provider who can watch anomalies in users and access behavior and jump in whenever a response is required  
• Use the Single sign-on authentication scheme with mandatory 2FA 
• Utilize NTA or Cloud Access Security Broker (CASB) products like Accediant Interceptor, DarkTrace, PAN, Cisco Stealthwatch
• EDR

Additional Resources

• Discover the benefits of involving MDR Provider.
• Download MRD Service Datasheet.
• Learn more about our Security Automation and Compliance Platform WarRoom.
• Need help? Contact Us.

Why It’s Important

Email is one of the most valuable IT systems where organization share their plans, sensitive documents, chats….and even passwords.

UnderDefense, in cooperation with the Computer Emergency Response Team of Ukraine (CERT-UA) participated in a series of Incident Response cases in H1’2022 and noticed that Russian hackers and Ransomware groups shifted their focus to breaking into E-Mail Systems (primary on Exchange and Zimbra).

In this specific case, CrowdStrike EDR was in place and spotted an initial foothold but missed other critical backdoors and TTPs which were later disarmed by the UnderDefense 24×7 MDR/SOC Team. And the attacker was eventually kicked out of the network.

What You Will Learn

1. Risks for email system as document exchange and integral part of business workflow
2. Data theft via business email compromise in a targeted attack scenario
3. Recent technical vulnerabilities and risks
4. What data APT groups are hunting for in their targeted attack
5. Arsenal used in this case
6. Tools vs PPT
7. Case Details
8. BEC incident response playbook
9. Recommendations and takeaways

In the end of 2021, the whole digital world has suffered the new cybersecurity flaw named Log4Shell. A new vulnerability is considered to be one of the worst that have been discovered during the last years. It scored 10 out of 10 points on the CVSS vulnerability rating scale, and it puts countless servers at risk.

What is Log4Shell?

On December 9th, a critical vulnerability that allows arbitrary code to be executed was discovered. The exposure got the code CVE-2021-44228.

The Log4Shell is a vulnerability in the open-source logging library, Log4j version 2, which is used by millions of Java-based applications/servers to log error messages. Such digital giants as Tesla, Twitter, Apple iCloud, Amazon, and millions of other companies use the Log4j library.

There is a lookup substitution function in the Log4j library. Log4Shell vulnerability exists because lookup substitutions are not protected enough when dealing with user-controlled input. Unauthenticated users can exploit this vulnerability via a web request to execute arbitrary code with the permission level of the running Java process.

The first worldwide famous target was Minecraft. On December 10th, people started sharing videos showing that, while playing online, they could just insert code to chat on the server and seize power over the server. But most likely, everything started earlier. Cloudflare -Content Delivery Network and DDoS mitigation services provider – checked their systems and noticed that the first attack on their clients with Log4Shell vulnerability had been tried to conduct on December 1st.

What makes Log4j uniquely dangerous even though you seem protected

Exploiting Log4Shell vulnerability allows hackers to launch Remote Code Execution (RCE) and remotely take full control of the victims’ systems. Hackers are already actively exploiting this vulnerability. For the last week, Ransomware groups weaponized their toolset with this exploit and are using it to disrupt normal businesses operations, exfiltrating data & making affected servers unavailable for customers.

One more point which makes Log4Shell as dangerous as it is the simplicity of exploitation. Even “junior” hackers can use this exploit. To gain control over the victim’s system, a hacker inserts the code anywhere this library handles – fill the form the website, modify website URI or Browser user-agent, or text in the support chat – and it will lead to code execution.

The whole java-world is trying to deal with Log4Shell and emphasize that it is the highest possible priority for all-sized businesses. Cisco, Apple iCloud, Microsoft, and so many other huge technology companies have already stated that some of their systems were vulnerable, but they are fixing it. But for small-sized companies without a cybersecurity department, it might be quite hard to mitigate the attack independently.

Which Version is not affected?

Almost all versions of log4j version 2 are affected. On December 14th, version 2.15 was found to still have a possible vulnerability. And a few days later, a Denial of Service (DoS) vulnerability was found in 2.16 too. The developers have already prepared version 2.17 and, as of December 20th, recommend updating the library again.

How to Mitigate the Log4Shell Vulnerability? First aid actions

Put a high priority on your IT/DevOps on patching/mitigating this vulnerability. This is worth immediate effort.

Update

It was previously thought that to be not vulnerable to Log4Shell, it is enough to turn off the lookup substitution function. But after a few days, it came across that it doesn’t work like that. Generally, the main action now (on December 20th) is to update the Log4J library to 2.17, which is supposed to be safe and has lookups turned off.

“To my satisfaction, our programs are not written in Java,” – you might think. But the point is that you may have hundreds of different systems, and they most likely are not developed by the inside team but developed by third parties – as it usually occurs. Therefore, you might not even know what is inside these systems. In this case, you should look at the product’s website or contact support for instructions on what to do to be safe.

Constant Security Monitoring

Log4Shell vulnerability is one of many, critical vulnerabilities that were found during the past ten years. And the situation is constantly evolving. The only way to see what is happening inside your system is to have 24×7 security monitoring and threat remediation and response. It will help you identify your vulnerable internal and external assets, patch production, review your log files for any Remote Command Execution attempts. Security analytics can see attempts to exploit Log4Shell vulnerability in the logs and block them*.

*Only in one client, the UnderDefense Managed Detection and Response team blocked six attempts to exploit this vulnerability only a week after the vulnerability was discovered.

A firewall is not a panacea

A firewall can block the attempts to exploit Log4Shell vulnerability, but this is not a panacea because the firewall main task is “not to pass such text.” But the exploitation of this vulnerability can vary. Hackers can easily make it so that the text does not match 100%, writing the same code using different methods, but still works WAF bypass. Accordingly, WAF is not enough but still shouldn’t be neglected.

Enable blocking on Web Application Firewall through AWS WAF, Cloudflare, or any other WAF you have, or directly on your web-server, reverse-proxy, load balancer.

Penetration Testing

After remediating this vulnerability with your DevOps team, it is worth running a penetration test to ensure external and internal systems are patched correctly, and other old vulnerabilities are not exploitable. Generally, pentesters will do the same as hackers do – try to conduct an attack on the vulnerable system. But don’t forget about other vulnerabilities that existed before Log4Shell and didn’t disappear. It is the same as having 12 bad teeth, but to treat only 1 of them. So, conducting a pentest, it is better not to choose only one vulnerability test.

Conclusion

Since December 9th, developers have thought that user can just turn off lookups in the Log4J library to fix the vulnerability. But a few days ago came across that this method doesn’t work, and millions of systems still stay vulnerable. Developers told to update the Log4J v2 library to 2.16. And people did it. But recently, the vulnerability was also found in 2.16, and now there is a 2.17 version, which is supposed to be safe.

The situation is evolving. Log4Shell is something new, something dangerous, and something that is not studied enough. We recommend you to have your finger on the pulse and take care of your cybersecurity.

UnderDefense enters strategic partnership with Microsoft Azure

Companies in various industries increasingly choose cloud as a primary location to store data or plan to migrate to the cloud. Such infrastructure is much more flexible for data modernization and analysis. But also among the main reasons for such transformation in the first place is cybersecurity and data protection.

UnderDefense selected Azure as main cloud platform to support Enterprise customers and becomes part of MPN (Microsoft Partner Network) to deliver better Incident Response and Managed Security Services.

Current UnderDefense partnerships, certifications and competence include also GCP and AWS where we serve our SMB &customers from Technology domain. The hassle we are going to solve for our Enterprise customers as part of their Cyber Resilience control include:

• Onboarding new customers to the UnderDefense service in minutes
• Implement MITRE ATT&CK detection coverage in a mixed Cloud environment
• UnderDefense investigation of Microsoft Defender ATP, LogicApps SOAR and Sentinel SIEM for alerts and telemetry
• Minimization of false positives and Massive automation with SOAR Provide on-demand
• Inicdent Response & Cloud based Digital Forensics (C-DFIR)
• 24×7 Cyber Incident Response Team (CIRT)