The Path to Business Resilience and The Cloud. From Sonic Boom & Crispy Chips to Kubernetes & Culture Change.

John O’Loan is CEO at iOMedia Group LNS. After launching Sky News, he assisted in the launch and operations of more than 40 other news, sport and documentary channels around the world. He holds an Executive Masters Degree in Culture Change at HEC Paris and The University of Oxford. In this article, he discusses how companies can best protect themselves from cyber threats.

On September 25, 2021 an incident at a broadcast service centre run by Red Bee Media in West London caused significant disruption to several broadcasters’ operations. Just before 6.30pm a fire alarm system was triggered. It was designed to quickly reduce the oxygen level in a technical area through the rapid release of nitrogen gas stored under high pressure. No human would survive, but the technical infrastructure should have remained intact and undamaged. Unfortunately, on this occasion, the gas was released with such speed and force that it created a sonic boom, damaging many computer server components beyond repair.

What was designed in the mainframe age to create efficient and timely resiliency had instead created massive failure.

Of the Reb Bee clients at that facility, Channel 4 was the worst affected. It was not fully resolved until mid-November. In the subsequent investigation by the regulator Ofcom, which took another 7 moths to compile and release, Channel 4 said its own disaster recovery facility had not been ready to provide subtitling and audio description services.

The regulator further found that Channel 4 had breached another of its licence requirements, by failing to effectively communicate with affected audiences after the incident. Ofcom has now called on broadcasters to improve their disaster recovery plans, warning that it will be reviewing its ‘TV Technical Codes’ later this year.

No Fried Chips.

At 8.30am on April 4, 2022 staff at ITN production headquarters in London received an email telling them there’d been "significant power outage" at their building on Gray's Inn Road and ‘some things’ wouldn’t be accessible at their desks, or on their laptops at home.

45 minutes later the regularly scheduled morning live talk show produced in that building was replaced by taped programmes, and when the power problem had still not been fixed, lunch time and then evening news programmes had to be relocated. In the case of Channel 4 News, that was 200 miles away to the North of England.

The primary generator for that part of London had failed, as had the backup generator.

The bigger problem for ITN, as it had been in the past when news programmes had to decamp from Grays Inn Road to be produced across town at their Westminster bureau, was not so much the power, but their air conditioning.

Without adequate power, they couldn’t run enough of the air conditioning and without cool air, the computer chips would fry, and their mother boards would melt in the rack rooms in the basement. The racks would then become useless, and as with the Red Bee situation in West London 6 months earlier, it would have taken weeks if not months to clear out the old equipment and get in the new.

The lessons from both incidents, at Red Bee and ITN are clear – and quite simply point towards The Cloud. Never again would we build the same power hungry, demanding, manpower intensive systems using up premium floor space, designed for an earlier era.

Resilience and Security.

On October 17, 2021 TV broadcasts for Sinclair-owned channels went down across the United States. Originally described as “technical issues” multiple sources later confirmed it was a ransomware attack. The Sinclair Broadcast Group is one of the largest media groups in the US, controlling 294 television stations in 89 markets.

The incident occurred in the early hours, taking down the Sinclair internal corporate network, email servers, phone services, and the broadcasting systems of their local TV stations. The incident came just after Sinclair performed a company-wide password reset for IT shared by its stations, following what was described as a “potentially serious network security issue.”

As a result of the attack, many local channels weren’t able to broadcast lucrative morning shows, news segments, and scheduled NFL games. However, the attack did not reach the part of the Sinclair group broadcast system master control, which allowed the company to replace the scheduled local programming on the affected channels, with a national feed. This allowed at least some channels to remain on the air, but loosing potentially millions of local advertising dollars, which is Sinclair’s life blood.

Over the past 3 and a half years, several other TV operators have been hit by cyberattacks, including France's M6 in October 2019; The Weather Channel in April 2019; the Cox Media Group also in the States, just prior to Scripps, in June 2021; Portugal's SIC in January 2021; Iran's IRIB in February 2021. There are likely to be others who have not reported being held for ransom.

Resilience and Sustainability.

Such examples given here are likely to increase. It’s a growing problem, upon which billions of dollars are balanced. It’s also clear that demand for video is also increasing.

Clearly cyber-attacks are a big issue for media organisations for resilience and sustainability, but so is aging, rack based technology, requiring expensive floor space, machine age manpower levels and massive amounts of electric power.

International research organisation Statista estimates that we’re heading for 500 Billion hours of video to be live streamed globally over the next 12 to 18 months. The Carbon Trust’s calculates the emissions caused by the current level of global production and B2C streaming will result in the loss of an area of sea ice the size of Scotland - annually.

We won’t reduce demand driven consumption, so the only way to provide the content, security and the power needed, within increasing requirements to reduce the energy used, is to change our technology, which firstly requires a change of culture.

Culture Change and Security.

Before any kind of systemic change needed in these industries can take place, to adopt new levels of security, sustainability and reliability, there needs to be general acceptance that change is in fact necessary. Without that acceptance, change will fizzle out.

The examples above have certainly proved the need for change, and the Covid pandemic provided the confidence to adopt that change. It proved the value of using cloud technology for production and distribution/ transmission.

Subsequent developments in system cyber security, such as the proper deployment of Kubernetes has proved the cloud’s ability to resist outside interference.

Limiting the lateral spread of any attack is arguably the best shield against ransomware attacks. This practice, known as zero trust, limits admins or users’ access privilege to a range of APIs while restricting access to more sensitive and mission-critical data stores that might include direct object storage components.

One such access limiting system is Kubernetes. Also known at K8, it provides several concepts and mechanisms that can help secure clusters, in the cloud.

To The Cloud – and Beyond.

At the iOMedia Group Limited we can see the culture change required – and the safe technology to allow it to happen.

Our cloud native live news and sports system - LNS Cloud 9 – is an entirely cloud enclosed workflow system, from planning and post ingest to editing, production, sequencing, graphics control and multi-platform playout, all working within cloud native technologies which are engineered specifically for the cloud - not just retro-fitted from an outdated idea.

Cloud enclosed solutions require less bandwidth; run off any web browser; consume much less power; resulting in more sustainable workflows. Saving on emissions will save business costs, as found in an independent study of cloud native video editing, which can alone reduce total cost of ownership by 35%, on top of the gains in staff efficiency and multi-platform distribution.

For us, all of these advantages are important. But the reason we did it was to produce a system that made story telling more efficient and communication easier to build and understand; to make story telling more reliable and resilient - along with the technology.