Caringo Blog – A Perfect Trio: Object Storage, Metadata and Elasticsearch

There is a common saying that “good things come in threes” derived from the Latin phrase “omne trium perfectum” and, over the years, we’ve seen an ever-increasing interest in the three topics of metadata, Elasticsearch and object storage. That comes as no surprise to us, as we continually search for ways to make sure that users reap the benefits of our technology and these three things are the very foundation on which our intelligent data management platform is built. So, let’s break down these three “good things” (not to sound like Martha Stewart or anything) and talk about why they are important, and the difference that use of them can make to an organization.

How is Object Storage Different from Block and File-based Storage?

Object storage takes each piece of data and uses it as an object, file storage stores whole data in a folder to help organize it, and block storage breaks down a file into blocks and stores them as separate pieces. However you care to phrase it, the topic of object vs. block vs. file and object vs. SAN/NAS has by far been the number one topic on our website for those who want to review the fundamentals of data storage.

What is Metadata and Why is it Needed?

In the simplest possible terms, metadata is data about data, and it is important because when used properly, it can greatly expand your ability to play, find, display, manage and analyze digital data and assets. With Swarm, each object has its own metadata and that metadata is stored with the object (buyer beware storage that stands up a separate metadata database, as it can be cumbersome and inefficient).

A subset of objects that come with their metadata allows analytics to be performed locally at the highest speed possible, and this is, of course, critical in a number of arenas. I recommend you take a deep dive on metadata for a more comprehensive look at this topic.

What is Elasticsearch and Should You Use it?

Elasticsearch is a distributed, RESTful open-source search engine built for the cloud that indexes object metadata in near real time and lets users perform ad hoc searches on the attributes and metadata of their stored objects. This serves to extend our Swarm API with methods that provide access to the content indexing functions to query Swarm objects in terms of their metadata (SCSP/S3).

This month on our Tech Tuesday webinar, I’ll be joined by Jamshid Afshar, Sr. Software Engineer, for an exploration of how object storage uses Elasticsearch. By watching this webinar, you will learn:

• How using Elasticsearch with object storage will benefit your organization
• Time-tested best practices for using Elasticsearch with object storage
• About upgrading from Elasticsearch 6 to Elasticsearch 7
• How to deploy Elasticsearch on Kubernetes

Register now to watch live or on demand.

Does Object Storage Make a Difference for Organizations?

Caringo Swarm Object Storage has been making a difference for our customers for 15 years. If you are wondering how object storage can make a difference for your organization, email us at info@caringo.com. We are happy to help you explore the possibilities.