This report focuses on identifying the most important investment drivers in Manage and Support derived from a mixture of sources, including survey data on technology priorities, company announcements, and financial data.
View the interactive report below or click here to access the PDF.
Highlights:
View the interactive report below or click here to access the PDF.
By Duncan Beattie, GB Labs
When designing a storage system for a production or post facility, it is easy to concentrate on capacity and throughput, and think that is all you need. However, a very real issue is data safety and how to guard against it. That is a particular problem in our industry of media production. Once you start considering the issues around data loss, the potential problems start to escalate: a real domino effect.
Media production companies are faced with significant challenges: perhaps an order of magnitude larger than other industries. First, we tend to deal with large files, and therefore very large storage requirements. If you are shooting with today’s digital cinematography cameras you are generating many terabytes a day of raw content. If you are a busy post house with a number of edit suites, grading rooms and audio dubbing theatres, then you will have multiple simultaneous projects on the go. This will create a significant number of files with some being very large.
In parallel, and compounding the challenge, there are significant time pressures. Often you only get paid for a production once it is delivered. There are few things more disastrous in the production industry than failing to deliver a project on time: at best there is the risk of significant reputational damage, and there may well be a financial penalty too.
So we start out with data which has already had a significant investment. The costs of a shoot including cameras, crew, actors, locations and more. That data has to be secured to the highest standards: the loss of data requiring the recreation of an event is almost unthinkable. The cost of restaging a shoot will spiral: actors and key crew may be unavailable, and it may even be impossible to do it again. (Think live sport).
The next domino to drop is that while you are working out how to recover the lost data, you are not making any progress on current projects. Editors and others are standing idle and the delivery deadline looms.
In truth, few would think of any production set-up – even one based on a single edit workstation and its internal disk drives – that did not back up the data on ingest. But backing up takes a finite time, especially when dealing with large files, which means that there are inherent delays in the system.
Larger facilities will have central storage systems, serving multiple creative rooms. Here is another domino: a single storage system supporting multiple projects means that any data failure has the potential to delay many clients. And that enterprise-level data failure means there will be many editors, colourists and others standing idle, costing money and earning no revenue. Completed work will have to be remade, taking time and costing money.
RAID
Using a RAID-protected server gives some insurance against drive failures, although the performance of the server will be impacted while the RAID is rebuilt once the dead drive is replaced. But the whole server also needs to be regarded as a potential single point of failure, with consequent data loss or at least loss of productivity.
Again, you do not need me to tell you that you should be backing up your server. But now we are working in tens or hundreds of terabytes, so backing up is no trivial matter. And even if a good incremental backup system is implemented so there is relatively little overhead as backups are created, there will certainly be a long delay while the repaired server is rebuilt.
Some will choose to back up to LTO tape, which is simple and efficient, but very slow to rebuild: maybe a week to restore all the data. Tape archives should be stored off-site, so there is a physical time to locate and retrieve the tapes before restoration can start.
Others will prefer to back up to the cloud, but downloading many terabytes of data taxes even the fastest internet connections, not to mention the significant egress charges raised by the cloud provider. In either case, again you are facing expensive downtime.
You also need to consider how long it takes to write the data to the backup storage in the first place. In a production or a post house dealing in many terabytes of new data a day, the time to ingest the original material and to write the backups begins to represent a very significant overhead.
You may decide to run backups overnight when there is little other traffic on the server, but that risks losing a full day’s work on multiple projects. And when the facility gets busy there may be a temptation to stop backups to keep everyone working, which means you are back in the completely unsafe zone again, and running into a spiral of inaction from which you may never recover.
Nearline
An option which will reduce risk significantly is to have a second online storage pool that backs up the live work server. Sometimes this is called a nearline server, as it is designed to support as closely as possible the online server.
The advantage of a nearline storage pool duplicating all the data on the primary server is that should any sort of failure occur, there is a complete copy of all media available on the same network with no need to download or restore from tape. You can also arrange another layer of security – to the cloud or to tape – from this second server, without impacting the performance of the primary source.
At GB Labs we specialise in high performance, high security storage systems, particularly for media applications. Our very strong recommendation would be that the minimum you should consider is a second storage pool within the production environment. Ideally you should be able to work directly from it if necessary. This means a structure very similar to the primary server, but the benefit is in the secure knowledge of minimal downtime and delays in production deliverables.
Further, there should be the third level of security, either disk to disk to tape or disk to disk to cloud. With well-designed archive software, you will be able to incrementally add to the cloud or tape backup at any time over the 24-hour period, ensuring this third level of storage remains as close to synchronisation as possible.
A subtle point in the system design is that you might choose the second storage pool as your point of ingest, with automation pushing it to the live server. This gives production teams the security of knowing that if they can see it, there is already at least one backup.
Copy protection
You can also consider immediate copy protection within the primary storage. This is often achieved through clustered storage systems.
A cluster is a collection of servers with a management layer to spread the data across the pool. Should one node within the cluster fail, the way the data is copy protected means that there is no immediate loss of content and work can continue. Although as an entity a cluster is resilient, an entire cluster can fail, resulting in the same data loss as if you had just a single server.
The alternative is a failover server, a completely identical server maintained with precisely the same data at the same time. Should the primary fail for any reason, all the clients can be switched to the secondary storage without any interruption to service. This has the benefit of zero downtime and zero data loss due to primary failure. The two servers can be in different locations, although for optimum performance they have to be on the same network.
Adding it all together, we have an online or primary server delivering content across very high-performance networks to your team of creative artists. There is an exact copy of the primary server on a duplicate, with an automatic failover switch should there be a problem with the primary.
The secondary server also handles all ingest, ensuring that all data is backed up at least to another server before it is released for use. It is also responsible for off-site backups, to the cloud or to tape, scheduled to meet the practical data demands of the operation.
The storage management layer will also provide a dashboard of system health and escalating notifications should any problems arise.
GB Labs has developed the range of tools needed to implement the storage capacity, performance and resilience of your business needs, helping you to deliver on time every time, safe from the domino effect of data loss.
Synchronization is a fundamental requirement in all content generation and broadcast; used extensively both in SDI / HDI formats and in IP packet based systems. Synchronization signals are typically generated from equipment including sync pulse generators, master sync references, and for the IP format, Precision Timing Protocol (PTP) grandmaster clock generators. Most sync generators encapsulate a very accurate internal clock which is synchronised to an external reference, which for the vast majority of manufacturers is a GPS & GNSS satellite timing signal. Consequently sync generators are connected to an outside GNSS antenna via coaxial cable or fibre optic cable, like those supplied by ViaLite Communications.
Sadly however there are individuals around the world (including bad actors and cyber threat actors) that want to disrupt the GNSS users, including those in the Broadcast industry, for a wide range of malicious purposes. Security is not just in the domain of IP based internet traffic, preventative and protective measures also need to be taken in the non-IP world including radio, timing and synchronization type signals.
Most timing and sync generators will operate in a “locked” mode to the GNSS signal; when the GNSS signal is lost the generator will move into a “holdover” mode, where the internal clock will maintain an accurate sync for a set period of time. The accuracy of the holdover clock and the time for which it maintains this accuracy is dependent on the technology and quality of the oscillator. For example, a rubidium oscillator will have much greater accuracy and far less drift than a crystal based oscillator. At some point, however, the holdover period will end and the timing sync signal will no longer be in specification or aligned with other timing signals in a larger system.
In most countries around the world, producing and using a device which interferes with the radio spectrum and disrupts communication services in GNSS or indeed other services like satcom, cellular, broadcast television and radio is illegal. Authorities like the FCC in the USA and Ofcom in the United Kingdom, have capabilities to detect GNSS jamming events, locate the perpetrators and potentially prosecute them.
What are GPS Jammers?
GPS jammers are freely available on internet auction sites and typically quite simple devices from an electronics design perspective. Many of them are built from the humble ‘NE555 timer IC’ which is used to drive a Voltage Controller Oscillator (VCO) and antenna. This in turn creates a broadband radio interference. Other crude jammers are produced by filtering the harmonic of a fast rising clock edge oscillator coincident with the GNSS signal band, which is typically around 1.575 GHz for GPS L1 band. This type of interferer is more continuous wave-type and creates a single strong in-band interferer.
Figure 1. NE555 timer IC; a really good device sadly used for a bad purpose!
Accurate Timing for Broadcast
Since 1994 ViaLite Communications has been helping many broadcast customers design and install GNSS signal routing solutions across fibre optic-based systems. These systems start at the antenna which is located outside. The RF signal from the antenna is connected to a ViaLite RF over fibre (RFoF) transmitter which is an electrical to optical converter. From the transmitter a fibre is then extended into the facility and distributed as required to a single or multiple end-point. At each end-point a ViaLite RF over fibre receiver then converts the optical signal back into electrical RF, and is then connected to a device such as a PTP Time Server or Sync Pulse Generator (SPG).
The advantages of using RFoF for GNSS signal extension and distribution include: very low loss connectivity, electrical isolation, optical splitting capability, and (very important for cyber security) RFoF by design is a unidirectional technology. Furthermore, the GNSS signal is not converted into any other modulation or digital data stream and so very difficult to tamper with or damage the signal integrity.
How to protect Broadcast sync systems from GPS Jamming?
When a “jamming event” occurs, the broadcaster needs to try and prevent the harmful signal impacting their timing based service and ideally, they also need to know it’s happening so they can take action or inform the authorities. Avoiding harmful signals can be done in a variety of ways from specially designed RF antennas through to very sophisticated signal processor based firewalls.
ViaLite has teamed up with the GPS and timing experts at Focus Telecom Ltd to distribute and integrate their family of GPS Resilient and OtoSphere™ jamming protection products into ViaLite’s offering. Contained within the Focus Telecom products is a unique and patent protected interference filtering algorithm which combines the patterns from two omni-directional antennas. It can analyze the direction of the interferer and nullify it from the wanted signal. In addition, Focus Telecom can provide a cloud-based service for providing data and analytics around jamming attacks, when used in conjunction with the company’s GPS Sensor device.
ViaLite and Focus Telecom Integrated GPS Jamming Protection Solutions
The Focus Telecom GPS Resilient Kit and OtoSphere products can be integrated with all of the ViaLiteHD GPS RFoF product range. In a system, the GPS Resilient and GPS OtoSphere products are placed between the antenna and RFoF transmitter. They are powered from the ViaLiteHD transmitter through a DC feed on the RF antenna port, so there is no need for additional power supplies and the Focus Telecom solutions have DC pass-through to the antenna. If customers want additional RF signal gain, the OtoSphere can be used in combination with an additional external in-line amplifier.
These integrated solutions have already been deployed to users and are now providing them with additional protection from unwanted jamming events.
Contact ViaLite for more information at sales@vialite.com or visit www.vialite.com.
Christian Struck
Senior Product Manager Audio Production, Lawo
Axel Kern
Senior Director Cloud and Infrastructure Solutions, Lawo
Redundancy has always been a major topic for broadcast operations to ensure that the show goes on despite a defective power supply or other failure. While all eggs were in one basket—i.e. in one place and close to one another—this approach was certainly helpful. Calling such a setup resilient would nevertheless be a stretch.
Conventional solutions launched before open-standards-based IP came along may be redundant up to a point, but that doesn’t make the operation as a whole resilient. Most are able to exchange control, audio and/or video data over an on-prem network, but connectivity to a wide-area network (WAN) spanning several cities or continents is not on the cards for them.
While a WAN-based infrastructure has enabled operators to leverage processing resources in off-premise data centers—of which there may be two, for redundancy, as in Eurosport’s and many other cases—and while accessing these resources from just about anywhere are indisputable assets of an IP setup, a lot more is required to make an operation resilient.
Separating the mixing console from the processing unit and the I/O stageboxes was an important step. Building WAN-communication into all of these devices through ST2110 and RAVENNA/AES67 compliance led to pinpointed expectations regarding resilience, as a recently conducted trial confirms. The aim was to show the prospective client that immersive audio mixing remains possible even if one of the two processing cores is down. It involved one A__UHD Core in a sporting arena in Hamburg, and a second near Frankfurt, at the production facility.
The team successfully demonstrated that if the preferred core in Hamburg, controlled from an mc² console near Frankfurt for the live broadcast mix, becomes unavailable, the core closer to the console immediately takes over. The physical location of the second core is irrelevant, by the way, as long as it is connected to the same network. WAN-based redundancy is an important element of a solid resilience strategy, even though the unavailability of a processing core is only the seventh likeliest incident in a series of plausible failures from which operators can recover automatically, according to Lawo’s customer service. This degree of redundancy involves so-called “air gapped units”, i.e. hardware in two separate locations, to ensure continuity if the “red” data center is flooded or subject to a fire: the redundant, “blue” data center automatically takes over.
Strictly speaking, the five likeliest glitches—control connection loss, routing failure, media connection failure, control system failure, and power supply failure—require no hardware redundancy when the audio infrastructure is built around an A__UHD Core. That said, having a spare unit online somewhere is always a good idea. It is also required as fail-over for incident number six, DSP/FPGA failure.
A second important aspect is to decentralize what used to be in one box. Even some IP-savvy solutions are still supplied as a single unit that handles both control and processing. For maximum resilience, one device should do processing, while a COTS server or dockerized container transmits the control commands it receives from a mixing console to the processing core, and a switch fabric does the routing. Separating control, processing and routing, and making all three redundant minimizes the risk of downtimes. Plus, except for at least one switch close to each required component, all devices or CPU services can be in different geographic locations.
It doesn’t stop there. A redundant IP network with red and blue lines is built around a switch fabric. Without going into too much detail, certain management protocols (PIM and IGMP) may cause issues that could seriously affect broadcast workflows or even make them impossible. The first is related to situations where the red and blue lines are routed to the same spine switch. An issue with that switch means that this part of the network not only ceases to be redundant but may also stop working altogether: it is a single point of failure. The second issue is related to how switches distribute multicast streams over the available number of ports when they are not bandwidth-aware. In a non-SDN network, this may lead to situations where one port is oversubscribed, i.e. asked to transmit more gigabits per second than it can muster, which causes errors at the receiving end.
These and other topics are being addressed by companies like Arista and Lawo via a Multi Control Service routine and direct influence of the VSM studio manager software on traffic shaping. The goal is to avoid failures, oversubscription of network ports, and to allow operators of large installations to immediately confirm the status of their switching and routing operations.
Combining the above with the HOME management platform for IP infrastructures adds yet another building block. HOME not only assists operators with automatic discovery and registration, but also with controlling processing cores by hosting the MCX control software for mc² consoles either on networked COTS servers or directly in a virtualized environment—and to dynamically switch from one processing core to the other, one console surface to the next, or one MCX control instance to another if the need arises.
Resilience necessarily includes control. VSM achieves seamless control redundancy with two pairs of COTS servers stationed in two different locations and automatic fail-over routines. Hardware control panels are not forgotten: if one stops working, connecting a spare, or firing up a software panel, and assigning it the same ID—which takes less than a minute—restores interactive control. (The control processes as such are not affected by control hardware failures, by the way.)
As installs migrate towards a private cloud/data center infrastructure, provisioning two (or in HOME’s case, three) geographically distanced COTS servers with permanent status updates between the main and redundant units allows users to remain in control. If the underlying software architecture is cloud-ready, those who wish can ultimately move from hardware servers to service-based infrastructures in the cloud. Technologies like Kubernetes and AWS Load Balancer can then be solicited to provide elastic compute capacity that instantly grows and shrinks in line with changing workflow requirements. A welcome side effect is that no new hardware servers need to be purchased to achieve this kind of instant, high-level resilience.
After experiencing the benefits of resilient, elastic control, some operators may wonder whether a similar strategy is also possible for audio and video processing. The short answer is: “If you like.” Quite a few operators are wary of the “intangible cloud” and may be relieved to learn that the ability to architect private data centers in a redundant configuration already allows them to achieve a high degree of resilience.
A genuinely resilient broadcast or AV network is a self-healing architecture that always finds a way to get essences from A to B in a secure way. Users may not know—or care—where those locations are, but the tools they use to control them do. And they quickly find alternatives to keep the infrastructure humming.
The only remaining snag was to provide operators with an almost failsafe infrastructure. A lot has been achieved to make broadcast and AV infrastructures resilient by design while keeping them intuitive to operate.
John O’Loan
CEO at iOMedia Group LNS
John O’Loan is CEO at iOMedia Group LNS. After launching Sky News, he assisted in the launch and operations of more than 40 other news, sport and documentary channels around the world. He holds an Executive Masters Degree in Culture Change at HEC Paris and The University of Oxford. In this article, he discusses how companies can best protect themselves from cyber threats.
On September 25, 2021 an incident at a broadcast service centre run by Red Bee Media in West London caused significant disruption to several broadcasters’ operations. Just before 6.30pm a fire alarm system was triggered. It was designed to quickly reduce the oxygen level in a technical area through the rapid release of nitrogen gas stored under high pressure. No human would survive, but the technical infrastructure should have remained intact and undamaged. Unfortunately, on this occasion, the gas was released with such speed and force that it created a sonic boom, damaging many computer server components beyond repair.
What was designed in the mainframe age to create efficient and timely resiliency had instead created massive failure.
Of the Reb Bee clients at that facility, Channel 4 was the worst affected. It was not fully resolved until mid-November. In the subsequent investigation by the regulator Ofcom, which took another 7 moths to compile and release, Channel 4 said its own disaster recovery facility had not been ready to provide subtitling and audio description services.
The regulator further found that Channel 4 had breached another of its licence requirements, by failing to effectively communicate with affected audiences after the incident. Ofcom has now called on broadcasters to improve their disaster recovery plans, warning that it will be reviewing its ‘TV Technical Codes’ later this year.
No Fried Chips.
At 8.30am on April 4, 2022 staff at ITN production headquarters in London received an email telling them there’d been "significant power outage" at their building on Gray's Inn Road and ‘some things’ wouldn’t be accessible at their desks, or on their laptops at home.
45 minutes later the regularly scheduled morning live talk show produced in that building was replaced by taped programmes, and when the power problem had still not been fixed, lunch time and then evening news programmes had to be relocated. In the case of Channel 4 News, that was 200 miles away to the North of England.
The primary generator for that part of London had failed, as had the backup generator.
The bigger problem for ITN, as it had been in the past when news programmes had to decamp from Grays Inn Road to be produced across town at their Westminster bureau, was not so much the power, but their air conditioning.
Without adequate power, they couldn’t run enough of the air conditioning and without cool air, the computer chips would fry, and their mother boards would melt in the rack rooms in the basement. The racks would then become useless, and as with the Red Bee situation in West London 6 months earlier, it would have taken weeks if not months to clear out the old equipment and get in the new.
The lessons from both incidents, at Red Bee and ITN are clear – and quite simply point towards The Cloud. Never again would we build the same power hungry, demanding, manpower intensive systems using up premium floor space, designed for an earlier era.
Resilience and Security.
On October 17, 2021 TV broadcasts for Sinclair-owned channels went down across the United States. Originally described as “technical issues” multiple sources later confirmed it was a ransomware attack. The Sinclair Broadcast Group is one of the largest media groups in the US, controlling 294 television stations in 89 markets.
The incident occurred in the early hours, taking down the Sinclair internal corporate network, email servers, phone services, and the broadcasting systems of their local TV stations. The incident came just after Sinclair performed a company-wide password reset for IT shared by its stations, following what was described as a “potentially serious network security issue.”
As a result of the attack, many local channels weren’t able to broadcast lucrative morning shows, news segments, and scheduled NFL games. However, the attack did not reach the part of the Sinclair group broadcast system master control, which allowed the company to replace the scheduled local programming on the affected channels, with a national feed. This allowed at least some channels to remain on the air, but loosing potentially millions of local advertising dollars, which is Sinclair’s life blood.
Over the past 3 and a half years, several other TV operators have been hit by cyberattacks, including France's M6 in October 2019; The Weather Channel in April 2019; the Cox Media Group also in the States, just prior to Scripps, in June 2021; Portugal's SIC in January 2021; Iran's IRIB in February 2021. There are likely to be others who have not reported being held for ransom.
Resilience and Sustainability.
Such examples given here are likely to increase. It’s a growing problem, upon which billions of dollars are balanced. It’s also clear that demand for video is also increasing.
Clearly cyber-attacks are a big issue for media organisations for resilience and sustainability, but so is aging, rack based technology, requiring expensive floor space, machine age manpower levels and massive amounts of electric power.
International research organisation Statista estimates that we’re heading for 500 Billion hours of video to be live streamed globally over the next 12 to 18 months. The Carbon Trust’s calculates the emissions caused by the current level of global production and B2C streaming will result in the loss of an area of sea ice the size of Scotland - annually.
We won’t reduce demand driven consumption, so the only way to provide the content, security and the power needed, within increasing requirements to reduce the energy used, is to change our technology, which firstly requires a change of culture.
Culture Change and Security.
Before any kind of systemic change needed in these industries can take place, to adopt new levels of security, sustainability and reliability, there needs to be general acceptance that change is in fact necessary. Without that acceptance, change will fizzle out.
The examples above have certainly proved the need for change, and the Covid pandemic provided the confidence to adopt that change. It proved the value of using cloud technology for production and distribution/ transmission.
Subsequent developments in system cyber security, such as the proper deployment of Kubernetes has proved the cloud’s ability to resist outside interference.
Limiting the lateral spread of any attack is arguably the best shield against ransomware attacks. This practice, known as zero trust, limits admins or users’ access privilege to a range of APIs while restricting access to more sensitive and mission-critical data stores that might include direct object storage components.
One such access limiting system is Kubernetes. Also known at K8, it provides several concepts and mechanisms that can help secure clusters, in the cloud.
To The Cloud – and Beyond.
At the iOMedia Group Limited we can see the culture change required – and the safe technology to allow it to happen.
Our cloud native live news and sports system - LNS Cloud 9 – is an entirely cloud enclosed workflow system, from planning and post ingest to editing, production, sequencing, graphics control and multi-platform playout, all working within cloud native technologies which are engineered specifically for the cloud - not just retro-fitted from an outdated idea.
Cloud enclosed solutions require less bandwidth; run off any web browser; consume much less power; resulting in more sustainable workflows. Saving on emissions will save business costs, as found in an independent study of cloud native video editing, which can alone reduce total cost of ownership by 35%, on top of the gains in staff efficiency and multi-platform distribution.
For us, all of these advantages are important. But the reason we did it was to produce a system that made story telling more efficient and communication easier to build and understand; to make story telling more reliable and resilient - along with the technology.
Paul Glasgow
Managing Director, Marquis
Unlike a car crash, a ransomware attack is most likely uninsurable. So, the effects on a major media enterprise, production company or post facility can be totally devastating. Here’s why we should all be concerned, however big or small an organisation.
We have seen very successful attacks on major broadcasters in the Americas, Asia, and EMEA – high profile events, which on occasions have caused irrecoverable loss or significant outage. These major organisations are well resourced and have large IT departments, with smart people and yet they have still succumbed to a ransomware attack. The only reason we knew these enterprise attacks were successful was because the effects were public facing through impaired service.
However, the prevalence of successful ransomware attacks in SMEs is as high – and probably higher – than the big enterprises. The SMEs don’t get or indeed want any publicity but the impact is just as painful.
The biggest mistakes are assuming a pre-existing business continuity strategy will protect against ransomware, plus the assumption is that data tape libraries are safe. This is also compounded by the widespread use of virtualised and often remote infrastructure. A business continuity (BC) strategy usually has some form of mirroring and or replication of business-critical components and often the concept of an automatic live fail-over in the event of fire or flood, etc.
Ransomware, by virtue of its design, does not result in a single point failure. It will destroy the redundant system just as quickly as the primary system as they are usually network-connected. So, a conventional business continuity strategy is not a fit form of mitigation. Let us consider some examples of successful attacks:
Conventional disaster recovery systems may not work
Organisations usually have some form of protection using firewalls, anti-virus, anti-spam and ever-more sophisticated AI-based services to protect IT infrastructure. However, it’s the human factors which probably represent the biggest risk. Ranging from good old-fashioned incompetence and wilful ignorance, through random and malicious targeted attacks.
Malicious attacks could be via current or former disaffected staff; the lone wolf approach. In this case there may be behavioural methods of detection. Also concerning is malign recruitment, whereby the extortionists actively target and recruit system admins via platforms like LinkedIn, with the reward being a percentage of the extortion. And just when you thought it couldn’t get worse: allegedly on the dark web, malign employees advertise access to otherwise secure corporate networks to insert ransomware for a fee.
So, from the above, we can conclude that the modes of ransomware attack are increasingly diverse. We naturally underestimate the number of on-going attacks as often they are hidden from public view. Of course, there’s a lot of stigma regarding payment of ransoms. To compound the situation, there is also a real risk that the decryption key may also not work.
Identifying the highest risk assets
At this point, it’s worthwhile considering which are the highest risk assets in the media production chain. Strangely enough, it’s not the finished masters, since these will be replicated in multiple libraries, but actually the unconformed project with all its media – in the worst case the day before the master is created.
A single project could represent months or years of work. If it is indeed encrypted the day before delivery, it could be an irrecoverable loss to the business, as well as an uninsurable contractual default. So, time to pay up, get your bitcoins ready!
In some regards we can’t assume anything within a facility is safe from attack. However, if there is one thing we should protect, it’s the work in progress – we can’t assume traditional business continuity policies, such as mirroring or tape back-ups, will help.
Mitigating total loss
So, let’s assume a worse case attack and everything within a facility is encrypted – a total loss. How can we avoid paying a $1m ransom? Marquis offers a different approach, which provides ransomware mitigation and the ability to easily recover business-critical work-in-progress.
In this example, we’ll consider the total loss of an Avid NEXIS system and its mirror, through encryption. Unfortunately, a conventional backup of a NEXIS system will back up data in its native unintelligible form, making it impossible to quickly recover a business-critical project. Here, the facility has been encrypted, however the S3 back-up is fully recoverable because of the way it has been created by Marquis.
The Marquis approach features:
Recovery
In the above example of a total facility loss, Marquis tools can be used to recover a business-critical project remotely and securely (from cloud backup, for example) to another facility or even a single Media Composer. A video of this ransomware recovery process working with Wasabi cloud storage is here.
A blind recovery to a free-standing Media Composer, so no reference to the original facility, is the ultimate ‘acid test’ recovery and unique to Marquis. Using this unique approach, recovery can be achieved in minutes (not days, weeks or months using conventional tools).
Conclusions
There are many vectors of ransomware attack; unfortunately human factors are extremely difficult to mitigate. Don’t assume a conventional disaster recovery or business continuity plan will work. Plan for the worst case attack, e.g., ransomware inserted into the production network. How will the business continue? How will you meet contractual commitments? Are you prepared to pay a ransom?
Marquis Workspace Tools provides effective mitigation against conventional disasters and ransomware. This software is used by some of the largest media enterprises using Avid NEXIS to provide effective recovery from a ransomware attack. For further information, please visit marquisbroadcast.com/workspacetools.
Jonathan Morgan
CEO, Object Matrix
The sudden shift to remote working within the media industry saw an incredible turnaround, with workflows being instated quickly to ensure that quality content creation could continue. Existing media tools were adapted to enable workers from around the globe to access content and contribute to production, all whilst the industry came to terms with wider logistical challenges. A quick rollout of infrastructure saw big changes in how the industry managed their assets; suddenly, data that would have been very difficult to access needed to be available to workers from their homes.
As time has progressed, cybersecurity issues started being noted within the industry. Hackers realised that there was an opportunity to exploit the quick changes that had happened within networks; both online media systems and communications channels were an obvious target. According to ThoughtLab, the average number of cyberattacks and data breaches increased by 15.1% in 2021 from the previous year. Ransomware has been one of the most documented cyber threats in recent years; so how can media companies reduce the risk of cyber-attacks on their media assets?
Understanding the Challenge
It’s important to get to grips with how hackers attack media companies and their assets. Media assets are hugely valuable, and both gaining and restricting access to this data is extremely beneficial for hackers. The well-documented 2014 Sony hack is an example of the magnitude of damage that hackers can cause; unreleased films were leaked online, security threats were made, and emails amongst employees were shared to the press.
Ransomware centres around encrypting data to prevent access; this method became efficient in recent years, with hackers choosing to cause maximum disruption through only encrypting parts of files. This can include encryption of file system names and metadata indexes. This does not take long and can render data completely inaccessible. Spyware and viruses are also other common methods of causing upheaval within organisations and must also be considered when focusing on cybersecurity. With high-value media assets at stake, the industry is an attractive opportunity for hackers and, therefore, it is important to review networks and assess any potential access points.
Who is at Risk?
It’s important to acknowledge that every media company must review how content is being stored and how it is accessed; cybersecurity attacks should be expected, and organisations must be prepared. Security should be integrated within media networks and entire infrastructures should be centred around safeguarding content. The entire network must communicate securely, regardless of workflow requirements or integrated applications, and it is important not to make assumptions regarding the levels of security being provided.
Some adopted cheaper NAS services during the quick setup of remote working systems and moved entire libraries of content to them; in these instances, it is important to check the levels of security being delivered by the provider. Legacy platforms can also be left exposed to attacks and can often be overlooked when upgrading solutions. The value of content, both new content and old archives, is huge for broadcasters. Cybercriminals are looking to exploit this through routes into media storage systems that might not be immediately obvious. So, what can organisations do to enhance security within their networks?
Steps to Creating a Secure Media Network
There are several methods which can reduce the business’ susceptibility to the effects of a cyber-attack:
Digital Content Governance (DCG) utilises a mixture of these steps and is fundamental in providing layers of security, as well as auditing, to ensure that security is integrated throughout each layer of the system. Exposing all of your content via a file-based system leaves it vulnerable and, therefore, employing a purpose-built storage system that adheres to high standards of DCG can mitigate the damage caused in the event of a hack.
Integrating Security at Every Level
With hackers approaching cyberattacks with increasingly sophisticated and intelligent tools, media organisations who leave security as an ‘add-on’ are at risk of losing control of their assets. The inherent value of media is a highly coveted prize, and the reward has been clear for those hackers who have attacked major players within the industry.
Global working has opened-up the way we access content remotely and systems must have an integrated, multi-layer approach to security to circumvent the loss of data in the event of an attack. A single layer of security is not enough; criminals are experts in finding routes around seemingly robust methods. Building a system from scratch which utilises good DCG practices from its foundations, including its interactions with interoperable apps, will provide the best level of protection possible.
Every media business should anticipate a cyber hack and, with the correct tools, feel confident that they have enough mitigation techniques in place to stay in control of their assets. To future-proof the media industry for long-term remote and hybrid working, we need to manage risk effectively by building resilience from the ground up.
Davide Del Vecchio
Group Cyber Security Senior Director - Deltatre
Even with the global cost of online crime reaching $6 trillion by 2021.
Even with 50% more cyber-attacks per week on corporate networks in 2021.
Even with the world’s most influential technology leaders claiming cybercrime to be the greatest threat to every company in the world.
…the fact of the matter is most broadcasters are woefully underprepared when it comes to protecting their businesses from cyber-attacks. And this is a big problem.
First, some scene-setting. What is the current status quo? Most broadcast entities approach online security in an equivalent manner - preventing the disruption of service. If an attacker gets through the door, keeping the lights, camera and action rolling is the number one priority.
In practice this has meant an expensive mixture of temporary, secondary, or tertiary systems or locations, multiple backups/routes, auto failover, automation, move to cloud-based platforms, and so on. Re-routing or restarting from a broken or compromised system/location/feed to another with the audience none the wiser might have worked in the past but may not with be enough to deter today’s cyber bad actors.
Consider this. Many well-developed cyber-attacks today are not designed to break anything - at least permanently. The objective is frequently to delay, interrupt, interfere or subdue. And to do so temporarily or frequently in short periods, so that broadcaster’s ability to detect and respond is limited, but the impact, on content availability, monetization or performance quality is high.
If you are an attacker your aim is to create a window that you can exploit.
This is made easier by the fact that most broadcasters simply do not have a handle on what assets they have or how these can potentially be compromised when used in the field. The rapid move to automation, particularly with systems and applications deployed in cloud-based platforms, means that gains broadcasters can make through speed and flexibility are often offset by the loss of skilled intervention ability. Cyber security defence processes are often outdated and not aligned with the fast-moving threat vectors that attackers seek to exploit in broadcast environments.
It would not take an attacker too much effort to deploy a rouge wi-fi access point or APN (access point name) near frequently used broadcast locations or locations in remote areas to potentially access location staff devices.
Multiply this across hundreds of reporters, using thousands of pieces of equipment that are used for years and never retired. It is a hacker’s dream.
Why does all of this happen?
The reality is that most executive teams are focused on improving content impact and viewer consumption and - while they are keenly aware of increasing the resilience of their business against cyberattacks - the proliferation of attacks and breaches (according to the Digital Shadows Research Team there are 24.6 billion stolen credential pairs available for sale on the dark web, most of which were exposed in the past year) has led many to become ‘cyber-fatigued.’
This can often result in the view that cyber-attacks are a necessary cost of doing business in the modern broadcast age, and they should just stay focused on outperforming their competitors.
This is slowly starting to change but the industry has a long way still to go.
So - what is the solution to building a modern-day cybersecurity strategy? Our recommended approach is broken down into five steps.
First, accept that the default of secondary or tertiary systems/applications/routes/feed, etc is no longer by itself fit for purpose in establishing business resilience. As the sophistication of attacks evolve, so too does the mindset of broadcasters around how to prepare for them.
Second, conduct a robust threat modelling activity. During this process, it is important to identify what kind of cyber threats are a realistic risk, how well-equipped the business is to handle them, and where gaps exist in your response plan.
Third, audit your equipment. Catalogue the technology used to run your business, what it is connected to, whether it is patched and when it needs to be retired. Banks do this incredibly well.
Fourth, introduce a level of monitoring that can differentiate between the noise and the signals. Most of the clients we work with start with just focussing on the former with a single monitoring tool. But the key to a successful cyber security response plan is being able to connect the breadcrumbs across internal platforms, the dark web, external intelligence services and SOCs to identify what may be coming around the corner.
Fifth, develop an agile response plan around a security team that sits outside of the typical organisational structure. We have seen many instances of cyber-attack where the impact has been made more severe simply by an inability to move quickly. The traditional response mechanism - build a war room, figure out what is going on, assign responsibility - just does not match the agility of a would-be hacker.
Instead, broadcasters need a Tiger Team of video engineering, specialist application developers and security professionals who are running regular training scenarios throughout the year and stand ready to respond to attacks immediately.
However this team is placed within the organisation, it should have a reporting line into the CEO/COO. This is important as it allows executive decisions to be made quickly against accurate available information. Lack of this is what former TalkTalk CEO, Dido Harding, cited as being the most significant issue in dealing with the 2015 security breach at the telecommunications company.
Success in cybersecurity requires a fundamental shift in mindset on what the modern-day threat profile looks like and investing appropriately to address what are widespread vulnerabilities in the broadcast space. Addressing these five points is a necessary first step in building business resiliency.
Nik Forman
Marketing Director, Friend MTS
Summary of report by Roger Thornton, IABM
Overall, the prioritization of business and consumer trends over pirate-driven ones was consistent across different types of media businesses. However, while pirate-driven trends were ranked as less important, most of them were classified as high priority, with cross-border illegal access to content and security breaches leading to content leakages identified as more important due to the Covid pandemic.
With most businesses predicting their original content investment will increase significantly over the next few years, it seems counter-intuitive that content protection solutions are not a primary business priority given the increase in piracy activity in all its various forms. One of the reasons behind this is the complexity of the advanced content protection landscape, which constantly needs to counter pirate-driven innovation. This means high budgets for content protection and a struggle to keep up with fast-paced developments effectively. In summary, the effects of complexity on technology understanding and pricing are important factors determining media businesses’ deprioritization of content protection technology.
Response to data and analysis circulated in IABM report sponsored by Axinom
The recent IABM report on content security trends in conjunction with our good friends at Axinom made for some interesting reading. As Roger Thornton mentions in his summary article, perhaps the most surprising takeaway is the discrepancy between a stated intention to invest in content, and a far lower priority in investment in content security technology to safeguard against the theft of that content, especially given the financial, operational and potentially creative resources that will be required to produce or acquire it. As Roger summarises, this seems counterintuitive, but budgets are finite and it could be argued that prioritizing content over business processes is where dutiful media providers should concentrate their majority resource.
However, with 80% of respondents indicating the high importance of DRM, and 55% already implementing watermarking technology, it’s clear that using technology to prevent unlawful acquisition and redistribution of content is seen as a critical business driver.
The excellent analysis provided towards the end of the report informs a list of positive and negative drivers of investment in content protection solutions, and as a provider of such products and services, these are the elements that drive the conversations that we have with our customers around the world.
First to the positive drivers:
Content security is no different to other media technologies – the most attractive solutions are those that offer both technical and commercial flexibility. Additionally, the ability to provide digital flexibility without impacting quality of product is key, and with the evolution of technologies such as client-composited watermarking, which is deployed at point of delivery, this flexibility has come a long way and continues to evolve. Crucially, this flexible architecture not only benefits the user operationally, but also enables providers such as Friend MTS to more easily keep pace with ever-evolving piracy technology.
It's no surprise that the other positive drivers all reflect the near-lockstep relationship between content value and the requirement to protect it, with sports and scripted content at the top of this pyramid, and lower-cost, niche programming appearing as a negative driver of investment. The world of video is ever-expanding, however, and the proliferation of unique content across non-traditional environments (for example content creators, corporate and other enterprise users) is already driving an increased requirement for video content security services in these non-traditional sectors.
The negative drivers of investment in content security are, if anything, of greater interest to providers such as Friend MTS; these represent areas of concern to users, and are the issues around which we need to continue to engage with our audiences, to provide greater clarity, if required, and guidance on the most efficient and cost-effective content security solution for every customer’s specific content types and distribution models.
The primary issue to address is at the top of the table, thus stated in the report analysis: “The effects of complexity on technology understanding and pricing are important factors determining media businesses' deprioritization of content protection technology.”
Again, it’s not surprising that a perceived lack of transparency of technology, convoluted pricing structures and a potentially bewildering array of solutions from which to choose are off-putting to buyers. Simple questions such as “Which is the best DRM solution for my organisation?” or “How much should I budget for a watermarking solution?” often don’t have easy answers, and although this is never for reason of deliberate complexity, or to create an aura of mystique, it’s clear that these are areas that can, should and are being addressed by technology providers to help customers arrive at the solution that best fits their requirements as painlessly as possible. As an industry we’re improving, but obviously work remains to be done.
The good news is that, as pointed out by respondents, there continues to be a noticeable move away from previous levels of complexity around security solutions, driven primarily by new technology and deployment models. In the field of watermarking, for example, extremely lightweight, client-composited watermarking can now be deployed in as little as two weeks, with minimal impact on existing infrastructure. This type of watermarking is now amongst the most widely deployed in the world, bringing the obvious commensurate business benefits, and is a direct result of the drive to provide simpler solutions and less complex deployment. Of course, there are different types of watermarking, and some may be more suitable for certain content types and distribution models; likewise with DRM, CAS management, and other content security services such as monitoring and enforcement. The key is understanding how this potentially complex environment fits together, and which components will do the right job for each individual customer’s content model and distribution environments – and communicating this to them clearly and effectively. The mission-critical foundation that underpins this entire process is independent, agnostic advice based on expertise, operational environment analysis and business intelligence. In short, when looking to implement a content security environment, make sure you consult a proven expert who won’t just sell you what’s on their particular van.
Finally, the report touches on platforms and providers, and the role that they can play in anti-piracy efforts; this importance cannot be overstated, and the efforts that large-scale platforms such as YouTube already undertake are to be applauded. Each dedicates significant resources to anti-piracy measures, and the ability to engage and collaborate with them is vital to the monitoring and enforcement services operated by companies such as Friend MTS. As before, work remains to be done, but increased collaboration between platforms, ISPs and other service providers continues to reap rewards. As an example, a recent program of close data collaboration between a major ISP and Friend MTS led to a 90% reduction in illegal streaming of certain content on their platform, and it is this type of collaboration and business analysis that continues to be the way forward in the fight against content piracy.
The IABM and Axinom report has some obvious takeaways for content security providers, and I’d like to thank both for their work in producing it. The research defines some clear goals – most notably greater flexibility and a drive to reduce complexity – and although we know that we’ve some way to go, I am happy to say that we’ve already come a long way towards achieving them.