Recently the world has got to know the details of the NotPetya attack suffered by A. P. Moller – Maersk in June last year. The company chairman has recalled the events, listing the losses incurred by the company, but also praising human resilience which allowed it to recover relatively quickly from such a damaging attack.
Among the teams bringing Maersk’s systems back online were also several Predicans – and here is their story.
Discovering the problem
It was the end of June, a nice, summer day. The morning started out like any other. Two of our Predicans, Service Manager Fruzsina Zacs and Predica CEO Paweł Szczecki, were on site at the Maersk UK office for vendor meetings.
The planned meeting began and discussions about the new Managed Service operations had just commenced, when a large commotion spread across the office. Suddenly, the monitoring screens started to show all systems turning to red, and not long after all laptops started to reboot.
Meanwhile in Poland, the team had arrived at the office, made coffee, switched on their laptops and were checking for project updates or verifying the systems like they would on a normal day. But for some of them, something was different.
The team working on the Managed Service engagement with Maersk suddenly received a call from their Service Manager Fru. She stated that all connections to the company are to be terminated immediately. They asked how they were expected to perform their duties that day. The response was: “You are not expected to. Not today.” They moved on to other things.
How to avoid making your systems vulnerable to an attack like this?
Start with the basics – ensure that all your security systems are up-to-date and patched. This will limit the attack vector in an event of a breach. Your users should also be aware that attachments can carry devastating malware. It is best to erase attachments from your communications altogether if at all possible. Additionally, make sure you have a secure backup of your data collected on a regular basis.
At that point, nobody knew what had actually happened. The first assumption was that something went wrong with the monitoring systems. However, once the laptops had started to reboot, the suspicions of ransomware were raised.
The extent of the problem was still unknown. Our colleagues on site were told that the meeting would be postponed to the next day but there was nothing they could do to help and they should return to their hotels.
The next morning, as they made their way back to the client office, it was evident that the issue had not been resolved. As they entered, they noticed several war rooms had been established and the vendor’s Identity teams were in planning.
After getting a very quick update on what was known at the time, the team in Poland was once again informed that they should keep their laptops disconnected from the Maersk network. At the same time, a quick decision was made to bring in our first experts ASAP.
That same afternoon our Cloud Identity and Active Directory experts landed in Heathrow. In the evening, the news finally broke: “Maersk is down.”
Limiting the attack vector
Administrator accounts are vulnerable to attacks if you don’t manage your access privileges properly. Simple solutions like logging off remote sessions or separating admin and regular accounts and workstations will instantly improve your security. Additionally, there are many tools you can use to secure your accounts and resources, such as multi-factor authentication or temporary admin access which gets automatically revoked after a specified time. It is also important to manage your external users and their permissions properly – a dedicated tenant may be a useful solution here.
It was soon starting to become apparent that the situation was extreme. As our Team Lead Tomasz Gościmiński recalls:
“In the morning I emailed Tomek who was our AD expert on site, asking him what was going on. At 5 pm I finally got a reply: ‘Tom, in a minute!’”
Once we finally had a clearer picture of the severity of the situation, we’ve sent in more team members to help. In the end, they were around for four weeks on rotating shifts, helping and advising the various IT and support teams to bring the systems back to life.
Chairman Jim Hagemann Snabe said at the World Economic Forum that the “complete infrastructure” of the company has been shut down by ransomware. For 10 days, Maersk had to switch to manual operations to manage their ships which were docking and unloading their cargo worldwide at all hours. Employees, senior management, partners and suppliers all worked around the clock to reinstate Maersk’s systems.
What to do during a ransomware attack?
Act immediately. Shut down all your computers and systems to contain damages. Ransomware can spread within minutes across your network using Windows Credential Manager and SMB exploits as just some of the ways to infect the machines. Do not attempt to pay ransom to recover your data – it’s just a decoy and your resources are already gone. You can focus on recovery once you have contained the attack.
Powering through, powering up
Our team was present on the premises for four weeks to recover and stabilize the core infrastructure services. They also ensured a quick turnaround for requests during the reinstatement of critical business systems. The primary and key objective was to reestablish the Active Directory for identity management. This was necessary to reenable resource access which was the first step towards switching back to automated operations.
At the same time, we have enabled login to Office 365 via a web portal and upgraded the authentication solution. This way, while the company hardware was still unavailable, employees could communicate using their mobile devices or alternative machines.
While on site, we have also implemented cloud solutions which enhanced company’s IT security. These services will protect the company from damage to critical systems in case of any future attacks.
Our team was also there to provide general support to Maersk’s employees while introducing the new services. This helped to make the adoption process as easy as possible under the circumstances and guide users through any required processes.
How to plan for an attack such as this?
Focus on the critical systems. Identify the resources which are essential for the business to be operational, and prioritize your recovery strategy around bringing them back online most quickly. The strategy should also include a forest recovery plan for your Active Directory – something that is too often omitted. Additionally, ensure that your credentials are truly secure in your network to help contain the damage.
After 2 weeks of very intense focus, round-the-clock shifts and gallons of coffee, the core systems were operational and stable once again. Maersk employees were going back to their stations and returning to their regular duties. The majority of our team was able to leave the UK and come back to our Warsaw office.
But the work on Maersk’s infrastructure is by no means over. The introduction of new systems means that there are still challenges with adoption which need to be addressed. The disruption also caused long-term effects on the systems which are continuously being resolved as they arise.
The NotPetya attack on Maersk has been one of the most challenging projects we have had to contend with. However, our team, together with Maersk’s IT department and other partners, has worked tirelessly to overcome the difficulties, minimizing the losses and bringing the company back online as quickly as possible.
It took time to fully recover all systems, and new challenges still appear, but even so, the infrastructure is now stronger than ever – as are all the people who worked to restore it. It was the experience of a lifetime!
What are the key lessons?
- Expect the unexpected. New vectors of attacks constantly emerge, like the chain of supply attack vector. They require a new approach to overall security management and monitoring
- Observing best security practices is still the best defense. Real-time monitoring services and appropriate rights management (particularly for admin accounts) are the right places to start
- Service providers release security updates regularly to help combat threats. Be sure to keep your systems up-to-date with the latest patches
- Consider network segmentation or application whitelisting to limit the range of a potential attack
- Having a backup strategy is key. Follow the 3-2-1 rule (3 data copies: 2 on-site, 1 off-site) to make sure you can restore at least the basic functionalities quickly
- Check your ports and security protocols. TCP ports 445 and 139, and SMB v1 are known vulnerable points. If possible, disable their use in your network.
Securing the systems for the future
Our Digital Advisor and Board Member Andrzej Lipka added some final remarks on securing organizations against attacks such as this.
We were able to react really fast to the difficult situation at Maersk because we had a managed service agreement (which started just a few months prior) and acted immediately. Right now, we are doing a lot of work on fixing the issues arising in the aftermath of the incident.
However, we are also optimizing and preparing the infrastructure in the event of future incidents of such a scale, e.g. preparing and test running DR strategies, designing regular security reviews and ‘penetration tests’ of the infrastructure we support. We are also deploying new cloud technologies to help alleviate or at least minimize the risk of these events happening again.
To get more advice on defending your organization from ransomware, read our previous post on the subject. If you want to be sure that your security strategy is up to scratch, or would like to know more about our managed services offering – contact us now.