In this interview, Alex Grossman (Chief Marketing Officer, Symply a Gobal Distribution Brand) talks to IABM TV about Symply Workspace, an expandable desk-side multi-user storage solution for collaborative editing.
Information Technology and Connection in the Virtual World
Can IT providers provide the massive bandwidth, rapid application customization and storage infrastructure needed for today’s virtual world?
Over the years, technology has been a major force in evolving our world. As each major technological advancement happens (e.g., cars, elevators, movies, TV, the Internet, the smart phone, etc.), we see society change to adopt technology. Whether this is increasing mobility, building up instead of out, or increased transmission of information, humans quickly adopt new technologies and use them to their advantage.
But, is the opposite true? Are our current needs driving the innovation we require to support our increasingly virtual world?
What Societal Shifts are Pushing the Virtual World?
With the introduction of social distancing, our society has been forced to move a tremendous amount of educational, professional and social activities to a complete or mostly online model. Classes, work, performances, parties, meetings and meet-ups—these activities are more often than not done from the comfort of home. And, while these activities keep us connected through the Internet using tools such as Zoom and Slack, they are also consuming vast amounts of streaming resources. BBC News reported that 2020 streaming was up 71% over 2019 with 12 Million people subscribing to new streaming services during lockdown.
While we all hope that this will be a temporary situation, we know it may last longer than we would like, with experts predicting impacts well through 2021 and beyond. And for some, while they don’t like the cause, they do like the effect and many are actually thriving in the virtual world. So, there is reason to believe that many will continue to embrace the lifestyle changes that 2020 has imposed.
How Must Businesses Adapt to Support the Shift to Virtual?
To succeed and keep the world economy from collapsing, businesses must adapt to the ever-changing situation we live in—and to a large extent, that means bringing everything online and ensuring business continuity and disaster recovery planning is up to date. For most organizations, information and content is the lifeblood of our business—whether that be information about customers or content that customers need or want. Around the world, vast numbers of employees have gone to a remote work model, and business infrastructures must accommodate collaboration throughout the organization, making sure data is accessible to those who need it. To compound that, those valuable employees now often have children at home to manage during the work day who are competing for bandwidth both for school and entertainment. So, how must technology evolve to keep up with society?
3 Ways Technology Must Evolve to Keep Up in 2020
Technology must continue evolving to support society, and, from my perspective, the three areas where technology providers must step up are bandwidth, rapid application customization and storage infrastructure.
Bandwidth
With double-digit usage increases, we often see Zoom meetings glitch or the spinning wheel of death take over when trying to stream a video or view a site. While providers claim that they can handle the extra load on bandwidth, would that be true if every household had the technology they actually need to keep connected? About 10% of American households still lack an Internet connection, and according to UNESCO, just over half of households worldwide have an internet connection. Realistically, if you have to tell your family to stop their work or school so you can ensure your webinar goes smoothly, is your Internet service provider really “keeping up?”
Rapid Application Customization
Applications developed primarily for use in the corporate world have now been adopted by educational institutions. I spent a couple of evenings recently helping a friend learn the ins and outs of Zoom so he can teach music to college students. It became apparent to both of us quickly that the tool was not designed for classroom use. Imagine if you are using it to teach a class of 5-year olds, and have simultaneous demands to instruct both in-person and online students? We must have rapid application customization to modify existing tools so that they can help us educate our youth when social distance is mandated or preferred.
Storage Infrastructure
Data, data, and more data! Where is all that data we are generating going to live, and then how will we find it when we need it? And, how will we protect it in case of a disaster (natural or manmade)?
Luckily, the storage that is needed to keep up with data growth in this decade was pioneered 15 years ago with two companies at the forefront. One of those companies was Amazon, with the contribution of S3 cloud storage. The other was Caringo, which released its first content-addressable storage (CAS) solution in 2006. The Caringo Swarm Object Storage Solution is now market-hardened at release 11.3, and it is used by some of the largest government, high-performance computing (HPC) and media & entertainment companies in the world for S3-compatible on-prem, off-prem and hybrid storage solutions. You can read those success stories or learn more by watching our webinars. And, if you have questions, feel free to email us at info@caringo.com.
When and Where Performance Matters with Object Storage
Storage performance matters most when the content you are storing is at the very heart of your business model.
There are times when you want to store data and performance is a non-issue. You need to keep the data (e.g., to meet regulatory requirements or “just in case”), but you don’t need to retrieve it on a regular basis or rapidly. And then, there are times when you need to store data and then locate it and retrieve it quickly. And, of course, you want to make absolutely certain that your data is securely preserved.
Where Does Storage Performance Matter?
Storage performance matters most when the content you are storing is at the very heart of your business model (i.e., it directly impacts your business model and the product or service you are providing). And, with everyone working remotely and everything going online, it has certainly become a concern for businesses of all sizes. Some people think first of expensive SAN and NAS for keeping data available. However, recently, many are moving data to public clouds, which are actually built on object storage technology.
Is the Cloud the Best Storage for All Your Archives?
While the public cloud may be a good choice for some, it is not necessarily the end all and be all of cost-effective storage. Depending on the use case, your business or organization might be better served by either a hybrid cloud (which combines on-premise cloud storage with a cloud service such as Amazon S3, Microsoft Azure or Wasabi Hot Storage) or an on-prem cloud. Examples of use cases where a hybrid cloud model is beneficial include:
- Your data center footprint space is limited
- You need both onsite and offsite copies of your data (e.g., disaster recovery (DR))
Recently, Caringo launched Swarm Cloud DR, a managed, offsite disaster recovery solution powered by Wasabi. As our CEO Tony Barbagallo said in his blog The Future is Hybrid (Cloud), “adopting a hybrid-cloud approach to data storage allows you to enjoy the speed in your local data center and the flexibility of the cloud for additional compute services and offsite data preservation.” Now, who wouldn’t want that?
How Does Object Storage Deliver Performance?
Case in point, when STFC decided to integrate object storage into its JASMIN super-data-cluster, they required high aggregate throughput as well as durability and accessibility of data over a common protocol (e.g., S3 wrapped in a multi-tenancy framework). A number of leading object storage solutions were tested in STFC’s environment. Caringo Swarm met the requirements by delivering:
- Read performance equivalent to parallel file systems for S3 throughput
- NFS and S3 access to the same objects
- Easy management of thousands of tenants and billions of objects
- Simple internal and external file access and file sharing
How did Swarm meet or beat every requirement specified for the JASMIN project’s implementation of object storage? Since 2005, we’ve devoted ourselves to evolving object storage technologies so we have a complete ecosystem of S3-compatible products. With some of the longest running and largest object storage installations in the world, we’ve proven that our storage clusters actually get faster as they get bigger! This phenomena is discussed in depth in our whitepaper Artificial Intelligence & Machine Learning: The Smarts of Swarm.
How is Performance Testing Performed for Object Storage?
Performance Testing Object Storage systems presents challenges that are different from those in testing more traditional storage such as NAS and SAN, and our architects and engineers are experts in performance optimization. In our September Tech Tuesday webinar, Principal Solutions Architect Ryan Meek and Sales Engineer Jose Marcos will discuss their experiences, and also discuss:
- How object storage system architecture affects performance
- Relevant performance dimensions for object storage software testing
- Best practices for measuring and evaluating performance
- Actual test results gathered on Swarm in real-world scenarios
- The role of concurrent users, transactions, latency and throughput in testing
- Client configuration for better S3 performance
There will be live Q&A throughout the event, so don’t miss the opportunity to get your questions answered by Ryan and Jose!
Virtual Advertising Cutting Through The Regulation
This paper proposes a way forward for entities seeking to implement or take advantage of virtual advertising solutions. It focuses on use of this technology in a sporting context but the principles are as relevant to use of the technology within any live event broadcast.
Although there is a complex legal framework, with differing positions around the world, we believe it is useful to develop some broad principles in this emerging regulatory area to support the development of the Virtual Advertising sector.
Whether you are a sports rights holder, broadcaster, producer, advertiser or technology provider, we hope this paper will clarify the regulatory issues and provide you with a pragmatic way forward which we propose can act as a useful benchmark for the industry and regulators around the world to refer to and adopt.
Standards and Regulations Update
Despite the dominant news in 2020 all about the Coronavirus pandemic, the preparations for the UK’s exit from the European Union, and a Presidential election in the USA, there have been important changes to note in environmental regulations too. By way of an example, ‘Chemical Watch’ contacted its membership to help understand the impact of the pandemic and notwithstanding many businesses of all types suffering from the financial shock to the system, 40% said that their regulatory workload had increased, whereas only around 20% had seen a decrease. Chemical Watch sees very little let up in the regulatory agenda and suggests that the challenges created by this are made more complex by the likely impending global recession, combined with so many teams now working entirely remotely, or already being affected by job losses.
Proposed amendments to the EU’s RoHS Directive list of restricted substances and exemptions have continued to be evaluated during 2020 by the Oeko-Institut e.V. (Institute for Applied Ecology) and the Fraunhofer-Institut for Reliability and Microintegration with details of the various consultations and reports can be found on their project website here. Meanwhile, as noted by Chemical Watch in their report ‘Chemicals in Electronics – Regulatory Developments From Around The World’ published in May 2020, laws and regulations having a similar intent to the RoHS Directive have been proposed, are being introduced, or are already in place in more than 45 countries other than those associated with EU. It notes that in the USA, a number of substances used in electronics are either already undergoing risk evaluation by the Environmental Protection Agency under the Toxic Substances Control Act (TSCA) or have been designated as a high priority for future risk evaluation.
Despite the Trump administration rolling back on the US reporting regulations on conflict minerals such that US Securities and Exchange Commission (SEC) decided not to enforce the law, it is worth highlighting that the EU’s version of this is still due to come into effect on January 1st 2021 and details can be found here. The UK, of course now destined to go its own way on this matter, only ‘strongly encourages’ companies to follow the Organisation for Economic Co-operation and Development (OECD) due diligence guidance documentation.
The EU’s Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH) regulation is one of many problems that Brexit will cause. Since this piece of legislation acts directly within EU Member States and does not require transposition in national laws, the exit of the UK from the EU means that as of January 1st 2021, this no longer has any direct impact on UK companies or products imported into the UK. Instead the Government has now confirmed it will put in place a separate UK REACH regime irrespective of any negotiated deal with the EU or a “no deal scenario.” For most companies this would appear to mean that a dual registration for both the new UK and existing EU REACH databases will be required. Meanwhile, the EU REACH system is, itself, due to get more complex with the new ‘Substances of Concern In articles as such or in complex objects (Products)’ (SCIP) database reporting requirements. While companies supplying products into the EU already have a responsibility to let their customer know if they contain substances of very high concern (SVHCs) on the REACH Candidate List in a concentration above 0.1% by weight, they will now have to submit this information to ECHA for inclusion in the SCIP database, as from 5 January 2021.
There is some pressure being applied on ECHA to delay the SCIP implementation deadline, as the task really cannot be high on anyone’s priority list in 2020. A trade body representing SMEs in Europe wrote urgently to the European Commission, even before the main Covid-19 lockdowns were imposed, expressing great concern about the impact on its members, saying “…we doubt that the database will be stable from 5th January 2021 on and we expect continuous adaptations. Such a situation is highly problematic for SMEs, which do not have the resources to regularly implement such adaptations.”
Keeping up Standards
In the Covid-19 era, the regular standards meetings that we attend have certainly been disrupted, but like many other aspects of our industry, virtual technology has stepped in as a substitute. The system is working, though not being on the meeting’s timezone can require attendance at some difficult hours!
Many of the sub-groups have already been using virtual meetings for several years and this has proved an efficient way to develop documents.
Who Needs Standards?
I’ll briefly revisit this perennial question. Sure, engineers developing latest / greatest code would like all the freedom they can get. But Standards are more of a strategic tool at Corporate level. The IABM has established that customers want to build their systems from best-in-class products, without being tied to one vendor. Standards achieve this, and they can help to make customers take new technologies seriously – e.g. IP with ST 2110, streamlined mastering with IMF.
For the purposes of this article, I will also cover some relevant trade organizations and consortiums that are producing important standards-like documents.
Latest Standards News
SMPTE’s “public CD process”
For a few months, SMPTE has been trialing a process where a document is released to the public while it is still at the Committee Draft (CD) stage. This exposes the document to the public before it has been balloted. The intention is twofold. The public becomes aware of the work much earlier than it would with the full publication process and it permits implementations to “test” the provisions and contribute to improvements before publication. Documents currently at public CD status are two multichannel audio documents and two interoperable mastering format (IMF) documents. Details here. More public CDs are on their way.
STOP PRESS! SMPTE currently also has a very similar Technical Specification process – here. This will be discontinued and transitioned to use the public CD process.
Video Compression
The video compression landscape was once dominated by MPEG (partnered with ITU-T). However, since MPEG-2, the licensing and royalty arrangements have become ever-more complicated and that is thought to still be the case with the latest offering, VVC, if you want to do anything beyond the baseline.
In a surprise move in June, ”father of MPEG”, Leonardo Chiariglione, resigned from MPEG declaring MPEG “dead”!
He has since initiated the Moving Picture coding by Artificial Intelligence (MPAI) community.
Meanwhile, the Alliance for Open Media (AOM) that was formed in 2015 to develop a royalty-free alternative, continues to grow. Its AV1 codec competes with HEVC, particularly in the online market.
SMPTE’s latest compression standard, VC-6, has just been published as ST 2117. It standardizes technology from V-NOVA.
Media over Managed IP Networks
The core parts of the ST 2110 transport suite are mature now, though some are getting minor revision as part of the SMPTE one-year-review process. Details on published parts. New parts in development are:
- Fast Metadata (FMX)
- Data Item Format for ST 2110 Technical Metadata
- Special Considerations for Standard Definition Video using SMPTE ST 2110-20
Alongside the ST 2110 transport suite is the growing suite of NMOS specifications from AMWA that are targeted towards control aspects:
- Discovery & Registration – IS-04
- Device Connection Management – IS-05
- Network Control – IS-06
- Event & Tally – IS-07
- Audio Channel Mapping – IS-08
- System Parameters – IS-09
- Authorization – IS-10
The latest status information for these documents is available here together with other documents on Natural Grouping; Securing Communications; Authorization; Certificate Provisioning; Parameter Registers. There is a very good introduction to NMOS here.
Sitting atop SMPTE 2110 and NMOS is the Joint Task Force for Networked Media (JT-NM). This task force laid some early groundwork for these technologies and in 2018 published “System Environment and Device Behaviors For SMPTE ST 2110 Media Nodes in Engineered Networks – Networks, Registration and Connection Management” – document TR 1001-1, freely available here.
It aimed to “plug the gaps” that still existed in the networked media environment for implementing fully interoperable systems. It includes an EBU Media Node Pyramid. Top-to-bottom slices are: Media Transport; Time and Sync; Discovery and Connection; Configuring and Monitoring; Security. The EBU has recently revised its source document with an updated pyramid that fortunately has less red! Webinar here.
JT-NM is also running a “JT-NM tested” program for validating products against ST 2110, NMOS and TR 1001-1. This year, “self-tested” catalogs were added – see all catalogs here.
Other Media Standards Work
A group within the Video Services Forum has been developing ideas for ST 2110 transport over WAN. This AIMS / VSF webinar from last month describes the issues. There is similar work in AES, where the group is writing a report – working title AES67 beyond the LAN – How to use AES67 on WAN and Cloud applications.
Microservices work has reactivated in SMPTE with a standards project “IMF Registration Service API”. A second microservices project is expected shortly on Device Control and Logging.
A joint task force between the Entertainment Technology Center and SMPTE has started. It will gather requirements and study use cases around artificial intelligence (AI) related to media production and consumption. Project here.
IABM Standards Resources
A good way to get up-to-speed on media standards is our June 2020 webinar.
Standards Meeting Reports
We produce reports after each SMPTE standards meeting round here and after each AES standards meeting round here
Deeper Dive – the IABM Standards Monitoring Group
We have selected SMPTE and AES project groups whose work seems most important to IABM members and we participate in their teleconferences – typically held weekly or fortnightly. This provides an up-to-date picture of the state of their documents and development work.
We make that information available to technologists in our member organizations with our Standards Monitoring Group (SMG). The SMG is a forum where members can comment on the provisions contained in draft standards documents and the IABM can submit those comments as part of the consensus-building process in the drafting group. It is usual and encouraged for the participants in drafting groups to consult with their colleagues to review and improve the provisions of the document. Information on joining the SMG is on this website page.
SMPTE projects currently monitored by the SMG:
- Media over Managed IP Networks – ST 2110 suite
- Network-based Synchronization – ST 2059 suite
- Microservices in Media
- Extensible Time Label (new timecode plus features such as source identification)
- Required application protocol standards for IP-based media production
- AI and Media
AES projects currently monitored by the SMG:
- Streaming audio metadata over IP
- Streaming audio-over-IP interoperability – AES67 (continuing revision and associated work)
- Open Control Architecture – AES70 (includes revision and adaptations for connection management of AES67 and other formats)
- Open Directory Architecture – Study group
- Projects on EMC mitigation and audio interconnections
Collaboration is the new normal
A decade ago, when Peter White became CEO of IABM, its membership was about 250 and the staff numbered five. At its peak in 2020 before Covid-19 hit, membership reached 650+ and the staffing had quadrupled. George Jarrett reports.
IABM now bristles with valuable data and knowledge, and smart initiatives and partnerships, but when White emerged from another industry that had adopted digitisation, he was surprised by the sloth with which the broadcast business jumped from tape to file, and joined the digital age.
“We just played catch up, but very quickly in doing a decade’s development work in five years. That’s what’s happened with Covid-19 too – two years’ work in just three months,” said White. “It has put us at the leading edge now: we have the technology, the expertise and the knowledge, and it has all been fast-tracked.”
How did Covid-19 impact IABM and its relationships with its incredibly varied membership?
“Nothing in my lifetime has caused as much upset. This has changed the world, and it has changed how we are going to view the industry going forward. It has accelerated the processes of remote production, of cloud usage and of IP adoption,” said White. “Shows won’t be the same, and neither will organisational structure. The virtual office will be more prevalent.
“We were particularly well placed for the pandemic because IABM has been widely distributed and by necessity an online, digitally connected business for a long time. We very quickly created a virtual platform that was free for members to use,” he added. “We produced tons of data, and did dozens of interviews through our own TV channel.”
IABM members were engaged with a wide-ranging programme of webinars and webcasts, and a series of courses that handle the issues around managing and changing of mindsets with a new normal in mind. IABM also teamed up with SMPTE, the DPP, SVG and HPA for joint web events.
“We have played our part in bringing the industry together. We worked with all those bodies so that our members and the industry in general do not have to worry what camp they are in. It is now all about working together in a collaborative way,” said White. “This is the way of the future.”
Can IABM members make the commitment of coming back greener, while sustaining R&D budgets?
“That is a very credible wish. I heard great stories about organisations that just continued with their R&D development throughout lock down, working with engineers distributed all over the place to produce new products. It is not just doom and gloom. During lock down there was cleaner air, and an attitude that was more responsible. There has been a shift in the mindset to be more aware of our planet,” said White. “The longer burn of environmental pollution needs to be dealt with, and we can start with not getting in our cars and travelling to offices. We can be as virtual as is practicable, and with shows we need to think of ways of making the experience good, while massively reducing our carbon footprint.
“All the owners of IBC and all the members of all the industry bodies have been similarly impacted by Covid-19, so it makes sense from a general logical aspect that we collaborate and drive things forward together,” he added. “Doing those web events with the other organisations proves they are willing to do this.”
Many IABM members were hit by the wipe out of production during lockdown, others in niche product areas may have struggled too. But live streaming companies, codec vendors, and platform and cloud enterprises have had a field day, whilst other companies just concentrated on R&D innovation.
How does Brexit mix in here?
“Brexit has not been a particularly constructive thing for our industry from a UK perspective, and it can only be exacerbated by a no deal Brexit. Covid-19 has magnified the other big issues that were there.”
White is worried by another issue that Brexit will magnify. “There has been little support for the freelance part of the creativity chain in the UK, and that has impacted tremendously on the level of unemployment. This will take time to recover,” he said.
Has the IABM Global Engaged Partner Program, giving it a dialogue with end users, been a help during the pandemic?
“Massively. Engaging when they were working from home has been easier, so we have been able to stage some great end user panel sessions. That has enabled them to understand what IABM does and how valuable this dialogue with the supply side of the industry can be,” said White. “It is important from a collaborative development point of view. They have a different need, a thirst for knowledge, and we can make that available to them.”
How will the IABM create a new normal for itself?
Its Business Intelligence Unit has fed out a mass of reports covering technical change. And IABM sits on huge banks of both data and content.
“Our Knowledge Hub has grown and grown, particularly with the amount of activity during the pandemic lock down. And continuing the amount of virtual work we were doing has just added to the Knowledge Hub,” said White. “Our focus now is making that knowledge much more discoverable, with a better user experience.
“Making the journey through our website easier is one significant change. We won’t be back to being such a physical presence everywhere. We will choose our moments and will take a more hybrid approach and be more discerning about what we deliver and what we invest in,” he added. “We have had to focus our resources like the rest of the industry has.”
IABM is also looking at getting back to supporting some of the big shows.
“We are aware that even that will be different going forward. The game changers, the disruptors, the new service giants are going to shape the way the shows look in the future. It will not just be about a supermarket of products, but about delivering innovation and building around it,” said White.
“We keep on preaching ‘adapt to change’ and that is what we have done ourselves. It has involved changing our structure and changing our approach. I mean to transition the IABM by keeping it moving and keeping it fresh,” he added. “To give it a younger image, we will make sure that our younger team members, who are brilliantly talented, are always at the forefront of delivery. That is what our industry is about.”
With the broadcast industry now morphed into a wider media business, and with many enterprises using the same core technologies as other industries, does media sit at a top table now?
“Yes. We are at the top table now in terms of developing technologies in partnerships with Telecoms, Automotive, Medical, IOT, Cyber Security, IT and IP companies, and Cloud services,” said White. “In the future it will be difficult to look for boundaries. We will still need optics to capture the image, but the rest of it is becoming so virtualised, so software driven; all the barriers have now gone.
“We still have those core production fundamentals around creating content, and that is never going to change. The way it is being finished, managed and delivered is so much more exciting. New people keep bringing in new things that we could not have dreamt of even a decade ago,” he added. “I was impressed by the HPA’s complete remote production of a feature film, which it showed at its retreat. The workflows and the focus on collaboration and efficiencies was kind of like a proof of concept for what happened during the pandemic lock down; I think HPA’s work before and during lockdown has been sensational.”
This article originally appeared on IBC365 – reproduced here with IBC’s kind permission.
In Conversation with HESP Alliance
In this IABM TV interview, Bart Van Oosterhout (Program Director of HESP at THEO Technologies) discusses the High Efficiency Streaming Protocol (HESP) Alliance including:
- What HESP is
- What the HESP Alliance will focus on to bring HESP to the market
- Why media companies and video vendors should become HESP Alliance members
- Which customers will benefit the most from HESP
- When we can expect the first HESP solutions
The Rise of the Corporate Studio, Enterprise Video, and Dynamic Media Orchestration
Success with video doesn’t come from choosing a mystical single best-of-breed MAM, Edit Suite, Live Switcher, DAM – or any other system.
Instead, it comes with being able to easily orchestrate human, semi-automated and automated workflows among these systems, and others that will inevitably emerge.
What you will learn with this White Paper:
For most enterprises, internal staffers feel that video is coming at them like a freight train. This is certainly true of consumer brands whether retail, eCommerce, or travel and tourism where there is a clear place for consumer-facing messaging — but it’s also true amongst nearly every other industry sector.
Need for speed: protocols for low-latency broadcasting
Low broadcast latency has become a mandatory requirement in any tenders and competitions for building of head-end stations and CDNs. Previously, such criteria were applied only to sports broadcasting, but now operators require low latency from broadcast equipment suppliers for every sphere: broadcasting news, concerts, performances, interviews, talk shows, debates, e-sports and gambling.
Latency in general terms is the time difference between the time when a particular video frame is captured by a device (camera, playout, encoder, etc.) and the time when this frame is played on the display of an end user.
Low latency should not reduce quality of signal transmission, which means that minimal buffering is needed when encoding and multiplexing while maintaining a smooth and clear picture on the screen of any device. Another prerequisite is guaranteed delivery: all lost packets should be recovered and transmission on open networks should not cause any problems.
More and more services are migrating to the cloud to save on rented premises, electricity and the cost of hardware. This increases the requirements for low latency with high RTT (Round Trip Time). This is especially true when transmitting high bitrates during the broadcasting of HD and UHD videos – for example, if the cloud server is located in the USA and the content consumer is in Europe.
In this review, we will analyze current market offers in terms of low-latency broadcasting.
UDP
Probably the first technology that was widely used in modern television broadcasting and associated with the term “low latency” was multicast broadcasting with MPEG Transport Stream content over UDP. Typically, such format was chosen in closed unloaded networks, where the probability of lost packets was minimized. For example, broadcasting from an encoder to a modulator at a head-end station (often within the same server rack), or IPTV broadcasting over a dedicated copper or fiber optic line with amplifiers and repeaters. This technology is used universally and demonstrates excellent latency. In our market, domestic companies achieved latency associated with encoding, data transfer and decoding using an Ethernet network of not more than 80 ms at 25 frames per second. With a higher frame rate this characteristic is even less.
The first picture shows a signal from a SDI capture card. The second picture illustrates a signal that has passed through the encoding, multiplexing, broadcasting, receiving and decoding stages. As you can see, the second signal arrives by one unit later (in this case, 1 frame, which is 40 ms, because there are 25 frames per second). A similar solution was used at the Confederations Cup 2017 and the FIFA World Cup 2018 with only a modulator, a distributed DVB-C network, and a TV as an end device added to the entire architectural chain. The total latency was 220–240 ms.
What if the signal passes through an external network? There are various problems to overcome: interference, shaping, traffic congestion channels, hardware errors, damaged cables, and software-level problems. In this case, not only low latency is required, but also retransmission of lost packets. In the case of UDP, Forward Error Correction technology with redundancy (with additional test traffic or overhead) does a good job. At the same time, the requirements as to network throughput rate inevitably increase and, consequently, so do latency and the level of redundancy depending on the expected percentage of lost packets. The percentage of packets recovered due to FEC is always limited and may vary significantly during transmission over open networks. Thus, in order to transfer large amounts of data reliably over long distances, it is necessary to add vast amounts of excess traffic to it.
TCP
Let’s consider technologies that are based on the TCP protocol (reliable delivery). If the checksum of the received packet does not match the expected value (set in the TCP packet header), then this packet is resent. And if the Selective Acknowledgment (SACK) specification is not supported on the client and server sides, then the entire chain of TCP packets is resent – from the lost packet to the last one received at a lower rate.
Previously, the TCP protocol was avoided when it came to low latency for live broadcasts, as latency increased due to error checking, packet resending, three-way handshake, “slow start” and prevention of channel overflow (TCP Slow Start and congestion avoidance phase). At the same time, the latency before the start of transmission, even with a wide channel, may reach five times the round-trip time (RTT), and an increase in throughput has very little effect on the latency.
Also, applications that broadcast using TCP do not have any control on the protocol itself (its timeouts, window sizes for re-broadcasting), since TCP transmission is implemented as a single continuous stream and before an error occurs the application may “freeze” for an indefinite period of time. And the higher level protocols do not have the ability to configure TCP to minimize broadcasting problems.
At the same time, there are protocols that work efficiently over UDP even in open networks and over long distances.
Let’s consider and compare various protocol implementations. Of the TCP-based protocols and data transfer formats, we note RTMP, HLS and CMAF, and of UDP-based protocols and data transfer formats, we note WebRTC and SRT.
RTMP
RTMP was a Macromedia proprietary protocol (now owned by Adobe) and was very popular when Flash-based applications were popular. It has several varieties supporting TLS/SSL encryption and even UDP-based variation, i.e. RTFMP (Real Time Media Flow Protocol, which is used for point-to-point connections). RTMP splits the stream into fragments which size can dynamically change. Inside the channel, packets that relate to audio and video may be interleaved and multiplexed.
RTMP forms several virtual channels on which audio, video, metadata, etc. are transmitted. Most CDNs no longer support RTMP as a protocol for distributing traffic to end clients. However, Nginx has its own RTMP module supporting the plain RTMP protocol, which runs on top of TCP and uses the default 1935 port. Nginx may act as a RTMP server and distribute the content that it receives from RTMP streamers. Also, RTMP is still a popular protocol for delivering traffic to CDNs, but in future traffic will be streamed using other protocols.
Today Flash technology is outdated and practically unsupported: browsers either reduce its support or completely block it. RTMP does not support HTML5 and does not work in browsers (playback is via Adobe Flash plugin). To bypass firewalls, they use RTMPT (encapsulating into HTTP requests and using standard 80/443 instead of 1935), but this significantly affects latency and redundancy (according to various estimates, RTT and overall latency increase by 30%). RTMP is still popular, for example, for broadcasting on YouTube or on social media (RTMPS for Facebook).
The key disadvantages of RTMP are the lack of HEVC/VP9/AV1 support and the limitation allowing two audio tracks only. Also, RTMP does not contain time stamps in packet headers. RTMP contains only labels calculated on the basis of the frame rate, so a decoder does not know exactly when to decode this stream. This necessitates that a receiving component evenly generates samples for decoding, so the buffer has to be increased by the size of the packet jitter.
Another RTMP problem is the resending of the lost TCP packets, which is described above. Acknowledgments of receipt (ACKs) do not go directly to the sender, in order to keep back traffic low. Only after receipt of the packet chain is a positive (ACKs) or negative (NACKs) acknowledgment sent to the broadcasting party.
According to various estimates, the latency in broadcasting using RTMP is at least two seconds with a full encoding path (RTMP encoder → RTMP server → RTMP client).
CMAF
Common Media Application Format (CMAF) is a protocol developed by an MPEG (moving picture experts group) commissioned by Apple and Microsoft for adaptive broadcasting (with an adaptive bit rate that changes based on changes in bandwidth rate of the entire network) over HTTP. Typically, Apple’s HTTP Live Streaming (HLS) used MPEG Transport Stream, while Microsoft’s MPEG DASH used fragmented MP4. In July 2017, the CMAF specification was released. In CMAF, fragmented MP4 segments (ISOBMFF) are transmitted via HTTP with two different playlists for the same content intended for a specific player: Apple (HLS) or Microsoft (MPEG DASH).
By default, CMAF (like HLS and MPEG DASH) is not designed for low-latency broadcasting. But attention to and interest in low latency is constantly growing, so some manufacturers offer an extension of the standard, for example Low Latency CMAF. This extension assumes that both the broadcasting and the receiving parties support two methods:
1. Chunk encoding: dividing segments into sub-segments (small fragments with moof + mdat mp4 boxes, which ultimately make up a whole segment suitable for playback) and sending them before the entire segment is put together;
2. Chunked Transfer Encoding: using HTTP 1.1 to send sub-segments to CDN (origin): only 1 HTTP POST request for the entire segment is sent per 4 seconds (25 frames per second) and henceforth 100 small fragments (one frame in each) may be sent within the same session. The player may also try to download incomplete segments, and the CDN, in turn, provides the finished part using Chunked transfer encoding and then maintains the connection until new fragments are added to the segment being downloaded. The transfer of the segment to the player will be completed as soon as the entire segment is formed (started) on the CDN side.
To switch between profiles, buffering is required (minimum 2 seconds). Given this as well as potential delivery problems, the developers of the standard claim a potential latency of less than three seconds. Meanwhile, such killer features as scaling via CDN with thousands of simultaneous clients, encryption (together with Common Encryption support), HEVC and WebVTT (subtitles) support, guaranteed delivery and compatibility with different players (Apple/Microsoft) are maintained. Of the disadvantages, one may note obligatory LL CMAF support on the player’s side (support for fragmented segments and advanced operation with internal buffers). However, in case of incompatibility, the player may still work with content within the CMAF specification with a standard latency typical for HLS or DASH.
Low Latency HLS
In June 2019, Apple published a specification for Low Latency HLS.
It consists of the following components:
1. Generation of partial segments (fragmented MP4 or TS) with a minimum duration of up to 200 ms, which are available even before the completion of whole segment (chunk) consisting of such parts (x part). Outdated partial segments are regularly removed from a playlist.
2. The server side may use HTTP/2 Push mode to send an updated playlist along with a new segment (or fragment). However, in the last revision of the specification of January 2020 this recommendation was excluded.
3. Server’s responsibility is to hold the request (block) until a version of the Playlist that contains new segment is available. Blocking Playlist reload eliminates polling.
4. Instead of the full playlist, the difference in the playlists (also known as a delta) is sent (the default playlist is saved and then only the incremental difference/delta – x skip – is sent when it appears, instead of sending the full playlist).
5. The server announces the upcoming availability of new partial segments (preload hint).
6. Information about playlists is loaded in parallel in adjacent profiles (rendition report) for faster switching.
The expected latency with the full support of this specification by CDN and the player is less than three seconds. HLS is very widely used for broadcasting in open networks due to its excellent scalability, encryption & adaptive bit rate support cross-platform functions and is backward compatible, which is useful if the player does not support LL HLS.
WebRTC
Web Real Time Communications (WebRTC) is an open source protocol developed by Google in 2011. It is used in Google Hangout, Slack, BigClueButton and YouTube Live. WebRTC is a set of standards, protocols and JavaScript programming interfaces that implements end-to-end encrypting due to DTLS-SRTP within a peer-to-peer connection. Moreover, the technology does not use third-party plugins or software, passing through firewalls without loss of quality and latency (for example, during video conferencing in browsers). When broadcasting a video, the WebRTC implementation over UDP is typically used.
The protocol works as follows: a host sends a connection request to a peer to be connected to. Until the connection between the peers is established, they communicate with each other through a third party – a signal server. Then, each of the peers approaches the STUN server with the query “Who am I?” (how to get to me from the outside?). At the same time, there are public Google STUN servers (for example, stun.l.google.com:19302). The STUN server provides a list of IPs and ports through which the current host can be reached. ICE candidates are formed from this list. The second side does the same. ICE candidates are exchanged via the signal server, and it is at this stage that a peer-to-peer connection is established, i.e. a peer-to-peer network is formed.
If a direct connection cannot be established, then a so-called TURN server acts as a relay/proxy server, which is also included in the list of ICE candidates.
SCTP (application data) and SRTP (audio and video data) protocols are responsible for multiplexing, sending, congestion control and reliable delivery. For the “handshake” exchange and further traffic encryption, DTLS is used.
Opus and VP8 are used as codecs. Maximum supported resolution: 720p, 30 frames per second with a bit rate of up to 2 Mbps.
A disadvantage of WebRTC technology in terms of security is the definition of a real IP even behind NAT and when using a Tor network or a proxy server. WebRTC is not intended for a large number of simultaneous peers for viewing (it is difficult to scale) due to the connections architecture, and also it is rarely supported by CDNs at the moment. Finally, WebRTC is inferior to its colleagues in terms of the coding quality and maximum amount of transmitted data.
WebRTC is not available in Safari and partially unavailable in Bowser and Edge. The latency claimed by Google is less than a second. Meanwhile, this protocol may be used not only for video conferencing, but, for example, for file transfer.
SRT
Secure Reliable Transport (SRT) is a protocol developed by Haivision in 2012. The protocol operates on the basis of UDT (UDP-based Data Transfer Protocol) and ARQ packet recovery technology. It supports AES-128 and AES-256 encryption. In addition to listener (server) mode, it supports caller (client) and rendezvous (when both sides initiate a connection) modes, which allows connections to be established through firewalls and NAT. The “handshake” process in SRT is performed within existing security policies, therefore external connections are allowed without opening permanent external ports in the firewall.
SRT contains timestamps inside each packet, which allows playing at a rate equal to the stream encoding rate without the need for large buffering, while aligning the jitter (constantly changing packet arrival rate) and the incoming bitrate. Unlike TCP, where the loss of one packet may cause resending of the entire chain of packets, starting with the lost one, SRT identifies a particular packet by its number and resends only this packet. This has a positive effect on latency and redundancy. The packet is resent with higher priority than standard broadcasting. Unlike the standard UDT, SRT has completely redesigned the architecture for resending packets to respond immediately as soon as the packet is lost. This technology is a variation of selective repeat/reject ARQ. It is worth noting that a specific lost packet may be resent only a fixed number of times. A sender skips a packet when the time on the packet is more than 125% of the total latency. SRT supports FEC, and users themselves decide which of these two technologies to use (or use both) to balance between the lowest latency an the highest reliability of delivery.
Data transmission in SRT may be bi-directional: both points may send data at the same time and may also act both as a listener (listener) and as the party initiating the connection (caller). Rendezvous mode may be used when both sides need to establish connection. The protocol has an internal multiplexing mechanism, which allows multiplexing several streams of one session into one connection using one UDP port. SRT is also suitable for fast file transfer, which was first introduced in UDT.
SRT has a network congestion control mechanism. Every 10 ms a sender receives the latest data on RTT (round-trip time) and its changes, available buffer size, packet receiving rate and approximate size of the current link. There are restrictions on the minimum delta between two packets sent in succession. If they cannot be delivered in time, they are removed from the queue.
The developers claim that the minimum latency that may be achieved with SRT is 120 ms with a minimum buffer for transmission over short distances in closed networks. The total latency recommended for stable broadcasting is 3–4 RTT. In addition, SRT handles delivery at long distances (several thousand kilometers) and high bitrates (10 Mbps and higher) better than its competitor RTMP.
In the example above, the laboratory-measured latency of SRT broadcasting is 3 frames at 25 frames per second. That is, 40 ms * 3 = 120 ms. From this we may conclude that ultra low latency at the level of 0.1 seconds, which may be achieved in UDP broadcasting, is also attainable during SRT broadcasting. SRT scalability is not at the same level as with HLS or DASH/CMAF, but SRT is strongly supported by CDNs and forwarders (restreamers), and also supports broadcasting directly to end clients through a media server in a listener mode.
In 2017, Haivision revealed the source code for SRT libraries and created the SRT Alliance, which comprises over 350 members.
Summary
As a summary, the following comparative table for protocols is provided:
Today, everything open-source and well-documented is quickly gaining popularity. It may be assumed that such formats as WebRTC and SRT have a long-term future in their respective spheres of application. In terms of minimum latency, these protocols already exceed adaptive broadcasting over HTTP, while at the same time maintaining reliable delivery, having low redundancy and supporting encryption (AES in SRT and DTLS/SRTP in WebRTC). Also, recently SRT’s “little brother” (according to the age of the protocol, but not in terms of functionality and capabilities), the RIST protocol, is gaining popularity, but this is a topic for a separate review. Meanwhile, RTMP is actively being squeezed out of the market by new competitors, and due to the lack of native support in browsers, it is unlikely to be widely used in the near future.