Venugopal Lyengar
COO Digital, Planetcast International
A number of headlines in the press over the last year have spotlighted the dangers of playing fast and loose with video playout Disaster Recovery (DR) protection. But providing the protection broadcasters and other media services need is not a simple – or inexpensive – task.
Video content today is delivered through playout platforms across DTV, cable, satellite and, increasingly, over-the-top (OTT) services to billions of viewers. The requirements and, increasingly, outright demands of audiences using an array of devices and platforms mean that modern playout systems' sophistication is evolving rapidly. Today, playout is a multi-staged process linking a number of systems, including media asset management (MAM) for storing and preparing content, playout core and distribution platforms.
These connected systems handle various stages, including compression, format conversion and layering of dynamic overlay content, to deliver engaging viewer experiences with the expectation of unparalleled reliability. They are the bedrock on which modern broadcast and linear over-the-top (OTT) channels have built their ability to reach and maintain their audiences.
Disruption is not always avoidable
Despite high levels of resiliency, it is a fact of life that playout systems fail. Disasters such as fire, flood, or the accidental cutting of power or critical cabling can and do cause outages.
For instance, in October 2021, multiple UK channels suffered major disruptions in playout services due to a fire-related incident at a central data centre. The outages ranged from relatively minor issues – such as loss of voiceover and graphics on certain segments – to complete loss of programming, advertising and related digital content. The outages' duration depended on the various services' DR capabilities. This downtime ranged from a few minutes to around four hours in the worst cases.
However, even 10 days after the outage, several channels were unable to restore access services such as subtitling – and still experienced issues with video and audio quality. And outages are not just restricted to DTV. For example, in December 2021, one of the UK's largest pay-TV operators experienced a loss of all of its TV channels, impacting viewers for hours.
The problem is that even with the extreme levels of resiliency that broadcasters use for playout, disasters – whether natural, accidental or technical in origin – can simply overwhelm a playout facility. The reality is that even near-perfect 99.99% ("four nines") playout reliability means a statistical average of about an hour of downtime a year. Even where there is a disaster recovery plan in place, often DR services aren't tested regularly, so when needed, they sometimes don't work as expected – if at all.
In addition, the sheer cost of creating a fully redundant playout infrastructure is prohibitive for many broadcasters, many of which operate in a highly cost-competitive environment. However, when the worst-case scenario occurs, the cost in direct revenue loss through advertising, severe reputational and brand damage, plus failure to meet public service obligations – required by government regulation in some cases – can be devastating.
The unfortunate truth is that no matter how skilled TV staff are, occasional TV broadcast outages are inevitable without truly independent, ready-to-activate disaster recovery in place. It is not a case of if but when…
The benefit of cloud-based DR
So how can media companies protect themselves in a way that makes business sense? On the surface, the solution is simple: deploy DR for playout and test it frequently across a number of likely failure scenarios to ensure it works.
Traditionally, a full active/active on-premise DR set-up that has replicated the playout infrastructure – running in another geographic location ideally – has been one way to ensure resiliency. If you lose one site, then all of the workload moves to the other site.
As discussed, the drawback has always been cost – especially for smaller and Tier 2 and 3 broadcasters. But drill down to the details and it becomes clear that finding a DR solution is more critical than ever. In the past, a cost-benefit assessment could rationalise that losing a projected one hour a year of DTV to local TV audiences could be deemed acceptable. However, the increasing complexity of playout plus the growth of linked services such as OTT and international distribution agreements means that the impact and cost of such events has grown exponentially.
At the same time, over the last decade, advances in technology and virtualisation mean that, today, it is possible to deliver playout directly from the cloud. Media companies are increasingly moving parts of their playout to the cloud – non-core channels for national broadcasters, for instance – to give them more flexibility and rapid scalability for extended coverage of sports and entertainment events.
It is also feasible to create a complete playout solution in the cloud – with full content replication ready to spring into action in the event of a primary playout failure. In other words, thanks to advanced cloud playout technologies, it is now possible to offer a broadcast-grade DR solution with minimal recurring costs when the service is not activated. In simple terms, a complete playout DR solution is deployed and managed for a negligible fee until it is activated.
How does Cloud-based playout disaster recovery work?
A cloud playout DR solution includes three core elements as it is operated by Planetcast International. The first is a cloud-based MAM system that is always on and resides within the cloud that receives programme schedules and content. This element is linked to the second, which is a full-featured cloud-based playout platform. When activated, this playout sends content to the third element, a secure internet stream delivery service, as well as a broadcaster's distribution provider – or redirects content to an alternate distribution provider if needed. The result is a complete and highly resilient service.
Perhaps, most crucially, if a provider offers a "pay-as-you-use" cloud DR solution, it has virtually no running cost when not in its playout mode. The full cost of cloud playout should only kick in when activated. The cloud DR system should be integrated as part of an 'out-of-line' deployment that requires no changes to existing playout systems or workflow. This approach is crucial to streamlining deployment and means that the system is truly independent of the existing playout chain – making the backup feed more resilient to related failures.
What’s more, this cloud-based approach enables media companies to scale up DR as they grow and add new channels and services that also need to be protected. Lastly, the solution should be spun up monthly to do an active playout resiliency test to ensure it is able to deliver services as expected.
Taking action using a proven provider
It is essential to recognise that this approach to playout and disaster recovery works best when it uses proven and established cloud-based platforms, and each deployment is delivered based on each client's unique requirements. The disaster provider has to have the expertise, experience and depth of technology and staff to ensure it can do that – and keep doing it consistently and well, even as the client's systems and business models evolve.
It is vital to remember that a cloud DR programme needs to be separate, secure and independent, whether it is to protect an onsite playout environment or supplement an existing managed playout service. That way, it will keep broadcasters on-air if the worst-case scenario should occur.
It is also worth noting that the shift to IP infrastructure is fundamental to enabling effective disaster recovery today. If your infrastructure is SDI-based, you can't virtualise or design a cost-effective multi-facility architecture, let alone harness the cloud-based workflows required to achieve a best-in-class disaster recovery position.
Finally, media brands should view the adoption of cloud-based DR as part of a broader, ongoing journey of the whole business toward the cloud. For use cases such as pop-up and language-variant channels, cloud-based playout often makes more financial sense than heavy investment in on-premise alternatives. The flexibility and scalability of cloud-based playout not only helps enable cost-effective DR and new services but, as part of a broader hybrid architecture, it improves overall business agility.
To download the complete solution paper from Planetcast International on this topic, entitled 'Resetting the economics of media disaster recovery', click here.
