When organizations are faced with the challenge of migrating a large volume of data, there are many considerations that have to be taken into account. It may not be as simple a problem to solve as you might think. Many questions need to be answered to figure out the best way forward.
- How much data is being transferred?
- What is the distance to the transfer location?
- Will it be over public internet?
- How much bandwidth will be needed?
- Are the source files on local storage, or network storage?
- What protocols are being used to read the files, and to send the files?
- How much time is allocated for the migration?
- Is there sufficient storage capacity at the destination, and if it’s on the cloud are there cost considerations?
- Are the files destined for object storage or file based storage?
- Are there firewall considerations? Is it for archival, or is frequent access required?
Once these questions are reviewed, there are many variables at play, and obviously many different permutations of these variables. Transfer tools typically have a limited number of skill-sets. Some are useful for ad-hoc transfers, transferring a few files up to a few thousand. There are certain technical tools that are proficient at transfer automation, as well as tools that are capable of integrating cloud storage, but are actually very slow at data transferring. Synching files is a benefit that some tools have the equivalency of as well. However, there are very few tools that are accomplished enough to master this complete set of skills.
Let’s pretend for a minute that storage capacity is not an issue, bandwidth is available, firewall is setup correctly, and cost structures are calculated and approved. A well oiled file transfer machine… well almost. You still need to consider the volume of data, the storage types and locations involved. All you need to do is start moving the data, right? Not quite.
One of the biggest considerations for choosing file transfer tools when it comes to migrating data, is the volume of files, not the total size of the data. Sending a couple of files that total 1 TB in size is usually no issue for most tools. It’s when you start to deal with millions of files that issues start to materialize.
The quantity of files is one of the biggest problems faced by IT teams trying to migrate data and it is a massive problem for the file transfer tools themselves. As an example, trying to transfer millions of files with a tool that is not designed to handle that volume can lead to severe performance issues, delays in transfers and can even result in the tool hanging indefinitely.
So why is this the case? It’s because most file transfer tools want to know things in advance and aim to present the end user with as much information as possible. In order to calculate how long a transfer will take, how many total files need to be transferred and the total size of all files, it requires all of the file attribute data in advance to perform the required calculations.
In the continuation of this blog we will take a deeper dive into what could be slowing down your file transfers and how to solve the issue.
You don’t want to miss the next post, be kept up-to-date by following us on Twitter @filecatalyst & sign up to our monthly newsletter HERE
