Lions & Tigers & Millions of Files… Oh My!
By Samantha Sjodin, Content Marketer, FileCatalyst
Wed 11, 03 2020
When organizations are faced with the challenge of migrating a large volume of data, there are many considerations that have to be taken into account. It may not be as simple a problem to solve as you might think. Many questions need to be answered to figure out the best way forward.
- How much data is being transferred?
- What is the distance to the transfer location?
- Will it be over public internet?
- How much bandwidth will be needed?
- Are the source files on local storage, or network storage?
- What protocols are being used to read the files, and to send the files?
- How much time is allocated for the migration?
- Is there sufficient storage capacity at the destination, and if it’s on the cloud are there cost considerations?
- Are the files destined for object storage or file based storage?
- Are there firewall considerations? Is it for archival, or is frequent access required?
Once these questions are reviewed, there are many variables at play, and obviously many different permutations of these variables. Transfer tools typically have a limited number of skill-sets. Some are useful for ad-hoc transfers, transferring a few files up to a few thousand. There are certain technical tools that are proficient at transfer automation, as well as tools that are capable of integrating cloud storage, but are actually very slow at data transferring. Synching files is a benefit that some tools have the equivalency of as well. However, there are very few tools that are accomplished enough to master this complete set of skills.
Let’s pretend for a minute that storage capacity is not an issue, bandwidth is available, firewall is setup correctly, and cost structures are calculated and approved. A well oiled file transfer machine… well almost. You still need to consider the volume of data, the storage types and locations involved. All you need to do is start moving the data, right? Not quite.
One of the biggest considerations for choosing file transfer tools when it comes to migrating data, is the volume of files, not the total size of the data. Sending a couple of files that total 1 TB in size is usually no issue for most tools. It’s when you start to deal with millions of files that issues start to materialize.
The quantity of files is one of the biggest problems faced by IT teams trying to migrate data and it is a massive problem for the file transfer tools themselves. As an example, trying to transfer millions of files with a tool that is not designed to handle that volume can lead to severe performance issues, delays in transfers and can even result in the tool hanging indefinitely.
So why is this the case? It’s because most file transfer tools want to know things in advance and aim to present the end user with as much information as possible. In order to calculate how long a transfer will take, how many total files need to be transferred and the total size of all files, it requires all of the file attribute data in advance to perform the required calculations.
In the continuation of this blog we will take a deeper dive into what could be slowing down your file transfers and how to solve the issue.
Based on your recent activity and preferences we recommend the following pieces of content
In this IABM TV interview, Paul Briscoe (Chief Architect, TAG Video Systems) discusses the challenges faced before IP is a reality in the cloud. Q1. What challenges do...
IABM2 days ago
IABM Webinar - What Vendors need to know about Cyber SecurityOften having a cyber security plan gets forgotten - until it is too late. Too late doesn't necessarily mea...
IABM6 days ago
Rise In conversation with Cait O’Riordan - Chief Product & Information Officer - Financial TimesCait O’Riordan is the Chief Product & Information Officer at th...
IABM6 days ago
IABM Webinar - Monetising Content ArchivesThe coronavirus pandemic showed us all how reliant we are on good quality, interesting content to drive advertising revenues ...
IABM6 days ago
In this IABM TV interview, Wei An (Partner and General Manager, BaishanCloud North America Corporation) explains how COVID-19 has impacted the streaming market and how...
IABM7 days ago
IBM Aspera and Irdeto On Demand Webinar - From production to distribution – Optimize operations and secure your content with watermarking in the CloudContent Owners ar...
IABM8 days ago
In this panel, we hear how the Coronavirus has affected each of these businesses and what kind of permanent effect it will have on the industry. We hear what the new k...
IABM13 days ago
While the concept of ‘X’-as-a-Service is not new, it is the new on-trend topic for the Telecom, Media & Entertainment (M&E) industry. In fact, Software-as-a-Se...
MediaKind Application Paper13 days ago