Lions & Tigers & Millions of Files… Oh My!
By Samantha Sjodin, Content Marketer, FileCatalyst
Wed 11, 03 2020
When organizations are faced with the challenge of migrating a large volume of data, there are many considerations that have to be taken into account. It may not be as simple a problem to solve as you might think. Many questions need to be answered to figure out the best way forward.
- How much data is being transferred?
- What is the distance to the transfer location?
- Will it be over public internet?
- How much bandwidth will be needed?
- Are the source files on local storage, or network storage?
- What protocols are being used to read the files, and to send the files?
- How much time is allocated for the migration?
- Is there sufficient storage capacity at the destination, and if it’s on the cloud are there cost considerations?
- Are the files destined for object storage or file based storage?
- Are there firewall considerations? Is it for archival, or is frequent access required?
Once these questions are reviewed, there are many variables at play, and obviously many different permutations of these variables. Transfer tools typically have a limited number of skill-sets. Some are useful for ad-hoc transfers, transferring a few files up to a few thousand. There are certain technical tools that are proficient at transfer automation, as well as tools that are capable of integrating cloud storage, but are actually very slow at data transferring. Synching files is a benefit that some tools have the equivalency of as well. However, there are very few tools that are accomplished enough to master this complete set of skills.
Let’s pretend for a minute that storage capacity is not an issue, bandwidth is available, firewall is setup correctly, and cost structures are calculated and approved. A well oiled file transfer machine… well almost. You still need to consider the volume of data, the storage types and locations involved. All you need to do is start moving the data, right? Not quite.
One of the biggest considerations for choosing file transfer tools when it comes to migrating data, is the volume of files, not the total size of the data. Sending a couple of files that total 1 TB in size is usually no issue for most tools. It’s when you start to deal with millions of files that issues start to materialize.
The quantity of files is one of the biggest problems faced by IT teams trying to migrate data and it is a massive problem for the file transfer tools themselves. As an example, trying to transfer millions of files with a tool that is not designed to handle that volume can lead to severe performance issues, delays in transfers and can even result in the tool hanging indefinitely.
So why is this the case? It’s because most file transfer tools want to know things in advance and aim to present the end user with as much information as possible. In order to calculate how long a transfer will take, how many total files need to be transferred and the total size of all files, it requires all of the file attribute data in advance to perform the required calculations.
In the continuation of this blog we will take a deeper dive into what could be slowing down your file transfers and how to solve the issue.
Based on your recent activity and preferences we recommend the following pieces of content
In this IABM TV interview, Lorenzo Zanni (Head of Insight & Analysis, IABM) and Janne Yli-Äyhö (CTO, Technology & Development at YLE) discuss the latest tech...
IABM4 days ago
IABM Coronavirus Impact TrackerIn conversation with...Hear from our Head of Insight & Analysis about how your data is helping IABM track the impact of this pandemi...
IABM6 days ago
How Coronavirus is impacting the media industryThe view from the front lineIn this on demand webcast, IABM and the DPP collaborated to put together a programme of broa...
IABM11 days ago
This webinar, powered by IABM regional research, looks at major developments in the MENA Broadcast and Media market, focusing on the forces driving spending in the reg...
IABM25 days ago
The FileCatalyst team has been hard at work bringing new updates and features to FileCatalyst Workflow. Co-Founder & President, John Tkaczewski, hosted a webinar o...
IABM27 days ago
2019 was truly action-packed for the Media & Entertainment (M&E) industry! The wheels of consolidation gained major momentum with several megamergers taking pl...
IABM67 days ago
IABM CTO, Stan Moote, discusses 2020, new emerging technologies and the IABM Technology & Trends Roadmap©.
IABM81 days ago
Cloud collaboration and integrationOn-demand webinar with ImagenWe all know the benefits of the cloud, but in this on-demand webinar, we hear from leading MAM provider...
IABM82 days ago