Signiant – Bridging the gaps between files and objects for media
Ian Hamilton, Chief Technology Officer, Signiant
File and object storage are both common technologies used for persistently storing digital media. While files and objects have many similarities, there are some notable differences.
A key difference is that while the contents of a file can be changed at any time, the content of an object can only be set when it’s originally created. An object can be replaced with an entirely new object with the same name (or “key”), but parts of an object cannot be changed independently. Simplifications like this facilitate improved scaling, reliability and data durability when all that’s needed is a mechanism to reliably store and retrieve fixed blobs of data.
Files and objects are also identified differently. A file is part of a file system with a hierarchical folder structure. Files can be placed in folders at any point in this hierarchy. Files can be renamed and moved efficiently within the file system folder structure without copying data. Objects, on the other hand, are referenced by a key, which remains constant over the object’s lifetime. Object keys can be created to mimic file paths, but changing the key associated with a blob of data can only be accomplished through adding a new object and deleting the old object.
These naming differences tie into differences in how files and objects are accessed. File systems are first “mounted” on a computer system. Applications then use local Operating System API (with operations like: open, read, write, close) to access files on mounted file systems. Object storage is accessed like a web server over a network (with operations like: put, get). There are standards for both methods of interacting, including protocols for accessing file systems over Internet Protocol (IP) networks, but accessing objects over IP networks is native to object storage.
Of course, most people interact with files and objects through applications. Applications can abstract many of these differences, but applications can also be rigid with the types of storage they support. Many web applications bundle storage and only allow access to the storage through the application. This provides minimal choice over how and where data is stored … an important consideration when working with large media files.
Most desktop media tools, like editors, can’t natively interact with object storage. Media stored as objects must first be copied into working file storage. If only part of an asset is required, the entire object must be copied into working file storage before trimming the asset. This wastes time, bandwidth and working storage space.
Signiant Media Engine facilitates user-friendly search and preview of media assets stored using objects or files and allows assets to be clipped before they are moved, all while retaining the original professional media format. This type of functionality helps bridge the gap between object storage and desktop applications like editing software.
While accessing object storage over IP networks is integral to the design, long distance networks still present challenges. The HTTP-based object storage access protocol performs much better over long distances than protocols for accessing files over a network like NFS, but we can do better.
Signiant’s patented intelligent transport technology accelerates access to storage by optimizing the choice of transport protocol (Signiant-UDP or TCP) and other esoteric transfer parameters like concurrent parts and part size based on the data set and operating environment. This optimization is done using a machine learning model trained on anonymized information from millions of transfers performed by Signiant under widely varying conditions.
When media is stored as files, built-in tools like the Mac Finder or Windows File Explorer can be used to browse files, view thumbnails, search and sometimes perform basic media operations like playing and trimming video. Folder structures can also be used to organize and group files. However, the file system containing these files must be mounted on the user’s computer. Tools for browsing object storage may present a file-system-like hierarchical view, assuming keys have been formatted to mimic file paths, but richer interaction with media stored as objects requires better tooling.
The Signiant Platform abstracts differences between public and private cloud and file and object storage. Signiant does this without having to take over ownership of the storage or the media. We simply connect to the storage and present stored media in a secure, intuitive and performant manner. Other applications can continue to interact with storage as they always have and Signiant stays in sync with any changes.
The Signiant Platform also enables normalized powerful search across all these types of storage using metadata discovered in files and objects either embedded with the media or in common sidecar formats like Adobe XMP. With support for a broad range of professional media formats, Media Engine expands what’s traditionally possible when using both file and object storage.
