Mitigating the effects of ransomware on television post-production

Unlike a car crash, a ransomware attack is most likely uninsurable. So, the effects on a major media enterprise, production company or post facility can be totally devastating. Here’s why we should all be concerned, however big or small an organisation. 

We have seen very successful attacks on major broadcasters in the Americas, Asia, and EMEA – high profile events, which on occasions have caused irrecoverable loss or significant outage. These major organisations are well resourced and have large IT departments, with smart people and yet they have still succumbed to a ransomware attack. The only reason we knew these enterprise attacks were successful was because the effects were public facing through impaired service.

However, the prevalence of successful ransomware attacks in SMEs is as high – and probably higher – than the big enterprises. The SMEs don’t get or indeed want any publicity but the impact is just as painful.

The biggest mistakes are assuming a pre-existing business continuity strategy will protect against ransomware, plus the assumption is that data tape libraries are safe. This is also compounded by the widespread use of virtualised and often remote infrastructure. A business continuity (BC) strategy usually has some form of mirroring and or replication of business-critical components and often the concept of an automatic live fail-over in the event of fire or flood, etc.

Ransomware, by virtue of its design, does not result in a single point failure. It will destroy the redundant system just as quickly as the primary system as they are usually network-connected. So, a conventional business continuity strategy is not a fit form of mitigation. Let us consider some examples of successful attacks:

• A major US media enterprise succumbed to a ransomware attack which encrypted databases. In normal operation, their BC strategy required primary databases to be sync’d to back up databases. Unfortunately, one of the first databases to be encrypted and irrecoverably lost was the single sign on (SSO) database and its backup. So, no system admins could login to access any systems as the ransomware propagated through the infrastructure thus wreaking havoc.
   
• A major Americas media enterprise succumbed to an attack which was encrypting storage. Unfortunately, it penetrated the production networks and encrypted several Avid Nexis systems and their back-ups, totalling 1.5PB of data. The loss was total and unrecoverable.
  
  
• A major European broadcaster suffered a ransomware attack, again encrypting databases. The on-air service was severely compromised. However, the data tape-based archive irrecoverably lost the primary and backup databases. All references to tape IDs, slots, indexes, etc., were lost. The totally inaccessible library held around 2PB of data.
  
  
• A specialist documentary production company suffered a ransomware attack in the final pre-delivery days of post-production. Fortunately, the ransomware attack was only partially successful and they were able to deliver the production according to the contract. However, the owners of the business found that their insurance policy only covered reinstatement of affected equipment. Consequential loss was excluded in the case of a successful ransomware attack.
  
  
• A North American VFX company had their facility irrecoverably encrypted. They paid an undisclosed sum to the extortionists, thought to have been in the $M range.

Conventional disaster recovery systems may not work

Organisations usually have some form of protection using firewalls, anti-virus, anti-spam and ever-more sophisticated AI-based services to protect IT infrastructure. However, it’s the human factors which probably represent the biggest risk. Ranging from good old-fashioned incompetence and wilful ignorance, through random and malicious targeted attacks.  

Malicious attacks could be via current or former disaffected staff; the lone wolf approach. In this case there may be behavioural methods of detection. Also concerning is malign recruitment, whereby the extortionists actively target and recruit system admins via platforms like LinkedIn, with the reward being a percentage of the extortion. And just when you thought it couldn’t get worse: allegedly on the dark web, malign employees advertise access to otherwise secure corporate networks to insert ransomware for a fee.

So, from the above, we can conclude that the modes of ransomware attack are increasingly diverse. We naturally underestimate the number of on-going attacks as often they are hidden from public view. Of course, there’s a lot of stigma regarding payment of ransoms. To compound the situation, there is also a real risk that the decryption key may also not work.

Identifying the highest risk assets

At this point, it’s worthwhile considering which are the highest risk assets in the media production chain. Strangely enough, it’s not the finished masters, since these will be replicated in multiple libraries, but actually the unconformed project with all its media – in the worst case the day before the master is created.

A single project could represent months or years of work. If it is indeed encrypted the day before delivery, it could be an irrecoverable loss to the business, as well as an uninsurable contractual default. So, time to pay up, get your bitcoins ready!

In some regards we can’t assume anything within a facility is safe from attack. However, if there is one thing we should protect, it’s the work in progress – we can’t assume traditional business continuity policies, such as mirroring or tape back-ups, will help.  

Mitigating total loss

So, let’s assume a worse case attack and everything within a facility is encrypted – a total loss. How can we avoid paying a $1m ransom? Marquis offers a different approach, which provides ransomware mitigation and the ability to easily recover business-critical work-in-progress.

In this example, we’ll consider the total loss of an Avid NEXIS system and its mirror, through encryption. Unfortunately, a conventional backup of a NEXIS system will back up data in its native unintelligible form, making it impossible to quickly recover a business-critical project. Here, the facility has been encrypted, however the S3 back-up is fully recoverable because of the way it has been created by Marquis.

The Marquis approach features:

• Detailed analytics of the NEXIS workspaces
• Visualisation and management of the workspaces to ensure duplicates and orphans don’t get backed up
• Will not add encrypted media files into the backup
• Automatic backup of projects with version control to allow direct recovery at a project or bin level
• Independent recovery tools 
• Blind project recovery from the back-up
• An extensive storage connection library including on prem and cloud storage

Recovery

In the above example of a total facility loss, Marquis tools can be used to recover a business-critical project remotely and securely (from cloud backup, for example) to another facility or even a single Media Composer. A video of this ransomware recovery process working with Wasabi cloud storage is here.

A blind recovery to a free-standing Media Composer, so no reference to the original facility, is the ultimate ‘acid test’ recovery and unique to Marquis. Using this unique approach, recovery can be achieved in minutes (not days, weeks or months using conventional tools).

Conclusions

There are many vectors of ransomware attack; unfortunately human factors are extremely difficult to mitigate. Don’t assume a conventional disaster recovery or business continuity plan will work. Plan for the worst case attack, e.g., ransomware inserted into the production network. How will the business continue? How will you meet contractual commitments? Are you prepared to pay a ransom?

Marquis Workspace Tools provides effective mitigation against conventional disasters and ransomware. This software is used by some of the largest media enterprises using Avid NEXIS to provide effective recovery from a ransomware attack. For further information, please visit marquisbroadcast.com/workspacetools.