Dealing with a cyber incident is a miserable situation for all involved. The financial costs, the time consumed, and the resources diverted to recover from such incidents are substantial. This reality underscores a critical lesson for organisations: investing in cybersecurity is not just a precaution but a necessity. This article encapsulates the message taken from Phil Cambers presentation at Securing The Law Firm.
What has become increasingly apparent is that many clients regret not having invested in security sooner or to a greater extent. Conversely, some organisations that had invested in the right security measures still found themselves compromised. However, their prior investments mitigated the damage, limiting the extent of the attack compared to those who had neglected to invest adequately.
While security tools play a crucial role, they are only part of the solution. We will delve into the multiple facets of comprehensive security strategies, including regular updates, employee training, and robust response plans, which are essential for minimising the impact of cyber incidents. By learning from these real-world scenarios, organisations can better protect themselves and ensure a quicker, more efficient recovery should an incident occur.
Phil Cambers presenting ‘The Recovery Position’ at Securing The Law Firm.
The concept of the ‘recovery position’ in medical terms refers to a body position used in first aid to keep the person still and open their airways. In short, it helps people by keeping them stable after a medical emergency until professional help arrives.
It’s no secret that medical emergencies can happen anywhere at any time. They can result from progressive conditions or random circumstances. When they happen, the first aid knowledge of the people on the scene can make a difference.
The same applies to your cyber security. Breaches can occur at any time because of progressive conditions or random circumstances. Especially with the prolific use of AI, the frequency of targeting has increased and become more sporadic. But when they do occur having the knowledge and experience to secure your business is paramount.
This article aims to be as transparent as possible while also showcasing our experience as an MSSP; specifically detailing the process of overcoming attacks and recovering a client’s business when they are compromised.
We all acknowledge that people are often the weakest link in the chain, whether wittingly or unwittingly. This is often because of being busy and sometimes because of shortcuts in management practices. However, the result is the same: the resources and money invested in your security do not perform as anticipated because they haven’t been implemented, updated, or supervised adequately.
2. How quickly can you recover my business if someone compromises it?
3. What security tools/services do I need to invest in to ensure I’m safe?
We will explore these varied questions in more detail below. At Trustack, we know that these concerns are common, and we have potential solutions. We address these by building relationships with our clients and understanding their environments.
Below are two recent scenarios that are anonymised to protect the client and their business identity. They outline the real impact of the cyber breach and highlight important steps in the recovery process.
Background with the client:
Attack and Initial Response:
An MDR (Managed Detection and Response) solution detected an unusual data exfiltration stream to an external cloud service. The MDR agents quickly severed the stream and isolated the affected devices after approximately 12GB of data had been exfiltrated.
Following the agreed playbook, the MDR provider contacted the client, who then involved Trustack. An incident commander was assigned, and a three-way call was initiated with the client, the MDR provider, and Trustack. The client’s cyber insurance company also became involved, invoking a third-party team to handle communications with the threat actors and work with Trustack.
Containment Measures:
The containment process took around eight hours, during which all external internet connectivity was severed. Trustack shut down non-essential systems and isolated compromised systems for further investigation. Our team also established secure communication channels outside potentially compromised systems like Teams or email.
The investigation revealed that the attack succeeded due to a brute force attack on an Azure Active Directory (AAD) account with a weak password. Trustack had previously recommended implementing MFA, but it had not been done due to business priorities.
Security Weaknesses and Mitigation:
The attack highlighted several security weaknesses: incomplete rollout of NGAV and MDR solutions, lack of consistent security policies, no MFA on AAD accounts or firewall VPN sessions, and outdated systems lacking proper patching. Within 24 hours of the attack, MFA was implemented for all user accounts, along with P1 conditional access policies. Within 48 hours, the old AV solution was replaced with a fully implemented NGAV solution, and comprehensive scans of all systems were conducted.
A new domain infrastructure was implemented, taking about a week, including restoration from backups dated five days before the attack. After approximately 1.5 weeks, the client was fully operational again.
Post-Attack Actions and Reflections:
Following the attack, the client implemented additional MDR modules for end-user security awareness and attended all MDR onboarding sessions. The client finally acted on security recommendations that had been waiting for 18 months. They realised that taking proactive steps would have been less disruptive than waiting for an emergency.
The attack was successful due to unmet basic security standards, lack of formal security responsibility, insufficient system monitoring and patching, underinvestment in technology and time, and a lack of understanding of vulnerabilities.
The client’s desire to save money often led to incomplete implementation of security solutions, and a reluctance to request expenditure from the board, which was a false assumption by the IT manager. The IT manager understood the need to outline requirements and ask for budgets so the board could make well-informed decisions.
The attack exposed significant security gaps because of the incomplete implementation of recommended measures. The swift, albeit pressured, response resulted in a more secure and updated infrastructure.
Moving forward, the client committed to ongoing security improvements and better preparedness for future threats.
Our Background with this Client
Attack and Initial Response:
The client’s EDR solution flagged suspicious data movement between virtual machines. Trustack joined a three-way call with the EDR support services team and the client, confirming a compromise.
All internal systems and services were isolated or shut down. We checked that the backup job was successful and that we could access the data. We also found that certain systems were maliciously encrypted, so we isolated them for further investigation.
By the following day, the EDR vendor’s global incident response team was involved in determining the compromise method using their retrospective tooling. Trustack provided log files and other data to third parties around the clock for 3-4 days, including firewall logs, domain controller logs, switch logs, and specific system event logs.
Containment Measures:
Trustack’s support service team worked in shifts alongside the EDR IR team, providing logs, scanning systems, confirming configurations, and testing backups for recovery readiness. The client’s cyber insurance company was involved from day one and wanted to deploy SentinelOne, but this was deemed unnecessary after the event.
On day four, the access point and method were discovered: a compromised legacy NAT rule allowed RDP sessions through a DMZ service, which was used as a jump box. A keylogger was dropped onto this DMZ device, harvesting credentials, and a switch with Radius authentication was used for internal network access with a legitimate account. The client later claimed they thought the NAT rule had been removed years ago.
Security Weaknesses and Mitigation:
For around 16 days, bad actors conducted reconnaissance using standard tools like Microsoft PowerShell, eventually exfiltrating small streams of data to Russian servers and starting the encryption process on internal systems. Despite being ready to recover the client’s systems after about five days, the recovery process didn’t start until around 11 days after the incident due to the client’s request for forensic root cause analysis.
Trustack recovered 110 systems in three days, including deep scanning each system before bringing it back online. The recovery point was roughly one week before the attack, resulting in a loss of three weeks’ worth of data. Two Trustack support team members and two client staff worked around the clock during the prevention and recovery process. They identified the exfiltrated data and deemed it of limited importance, so they did not make any ransom or extortion payments.
The attack succeeded due to weak security practices: still-active legacy NAT policy previously flagged by Trustack, poor admin practices (such as not deleting old accounts and ensuring strong passwords), misconfigurations between multiple protection systems, and the absence of an MDR solution which had been recommended by Trustack for the past two years.
The attack exposed significant security gaps, including legacy security practices and poor admin and system configuration management. The intensive response and recovery effort highlighted the importance of proactive security measures and the need for robust, ongoing security audits and oversight.
The incident underscored the value of listening to trusted security advisors and maintaining clear communication and accountability within the organisation.
In both cases which is fairly typical
In both, time and products
Either own it properly or outsource it to someone like Trustack
In that no one truly was accountable
The one that didn’t had a much more serious outcome in downtime and cost
Much more after experiencing a breach.
Were key in the recovery process.
Upon receiving the last invoice from Trustack for third-party incident response teams and new tools, some clients have jokingly remarked, “I might as well have just paid the ransom.” Recovering from a cyber attack can be expensive, not to mention the potential loss of revenue. However, giving in to ransom demands or doing nothing are not viable solutions.
Insurance companies increasingly require solutions like MDR as a condition for purchasing a policy, making a minimum security standard a common requirement in the industry. Overall it’s evident that most clients regret not investing sooner. Some of our clients have followed our guidance, avoided risks, and made substantial investments – and the best part is, they have not been compromised. The reason is they are not an easy target.
Does it constitute a failure if your business experiences an attack? As a transparent MSSP, we state that you can’t completely prevent a compromise. However, we can simply reduce the risk and make sure you have excellent recoverability options. The question is not ‘if’ but rather ‘when’ an attack occurs.
Preventing attacks in the current technological climate is almost impossible. There are various external factors to consider such as human error, unpatchable outdated systems, and the speed at which threat actors leverage AI. As well as the rate at which CVEs are disclosed compared to vendor response time and our ability to patch is extreme.
Limiting the extent and damage an attack does when it occurs is the focus. An attack occurring is not a representation of the failure of you or your systems. But rather how quickly we catch, contain, and resolve the threat.
Introduction of Incident Commanders:
After experiencing attacks of this nature, Trustack introduced the concept of an Incident Commander to manage incidents from a broader perspective.
This role focuses on resolving incidents and preventing executive sweep, allowing for more effective and streamlined incident management.
Proven Recovery Capability:
Trustack has successfully recovered every client affected by an incident without the need for ransom payments. Their layered protection strategies and practical experience in IT recovery have been key factors in their success.
The use of immutable backup protection has been particularly critical in ensuring business recovery.
Opt-Out Document: Trustack implemented an opt-out document for clients who choose not to follow their recommendations. This document, which must be signed by an appointed Partner or Director, serves as a disclaimer stating that Trustack is not liable for any issues that arise due to the client’s decision to ignore their advice.
This measure, though somewhat litigious, underscores the seriousness of Trustack’s recommendations and ensures higher accountability and attention in boardrooms.
Comprehensive Skill Set: Trustack differentiates itself from some security-only MSSPs by having in-house practical skills in networking, firewalls, VMware, storage, Microsoft, Veeam, and more. This capability allows us to rebuild an environment from scratch if necessary, providing clients with confidence in our ability to recover from attacks.
Unlike some MSSPs that merely resell security product licenses, Trustack offers a full spectrum of security and practical IT services, ensuring comprehensive protection and recovery solutions for our clients.
Get your business on the front foot