Speed is one of the most important parts of a data migration. Data movement needs to be fast to be worthwhile. And when it comes to speed, Movebot's one of the fastest cloud data migration tools around. So how fast is fast?
Movebot has pushed through over 200TB per day on a single migration. That's really fast.
There are a lot of factors that can limit that. Max speeds are important, but what you really want to know is how fast will MY data transfer be?
Depending on what systems you’re migrating to and from, you can expect Movebot to transfer between 2-5TB per day. We’ll go over some scenarios to help you know when you can expect to be at the lower end, or when you can expect to push 5TB or more per day.
When you’re driving, you stick to the speed limit so you don’t get pulled over and come to a complete stop. Those speed limits are put in place to keep you safe… But sometimes they’re unnecessary.
Movebot on its own is like the Autobahn. And like the Autobahn, having no speed caps isn’t an accident waiting to happen. There aren’t any limiters in Movebot, because we want you to come in, do your migration, and be done with it. Our goal is for Movebot to be the fastest cloud data migration tool around so your transfers are quick and easy with no hassle.
So if Movebot isn’t limiting speed, why aren’t all migrations going at maximum speed? The main factors that limit speed right now are provider limits, hardware limits, and bandwidth uplink.
One of the most common hang-ups for speed is provider limits. If Moveboot is the Autobahn, then provider limits are like taking an exit and being slowed down to a crawl through traffic.
When we speak about provider limits, there are normally five things at play that will ultimately limit your migration speed. These limits are outside the control of Movebot normally.
| Rate Limiting | Rate limiting limits the number of API requests you can make. |
| Upload Limits | Some platforms limit the amount of data you can upload per day. |
| Download Limits | Some platforms limit the amount of data you can download per day. |
| Eventual Consistency | Eventual consistency is when you perform an action, but it does not show up for a "while" after. |
| Platform Limits | Things like maximum objects in a cloud drive. For example, Google limits the number of files in a single team drive to 400,000. |
Rate limiting is an interesting case, because on the surface it might seem like it’s simply limiting users to deter them from leaving. But its primary use is for protection against bots and DDOS attacks. Rate limiting helps protect websites or applications from increases in load (sometimes with malicious intent) that could degrade and impact the platform as well as brute force attacks.
When it comes to API calls, rate limiting is sometimes used to stop an API from being overwhelmed. If too many calls are made at once, the API may disconnect completely and stall services that rely on the API. Some providers include rate limiting to throttle the connection so that it doesn’t disconnect completely, but slows down instead.
Other times, rate limiting is for cost and resource controls. It can help companies estimate costs better and have them less likely to spiral out of control. This reasoning doesn’t benefit the consumer, however, and instead protects the developer from unexpected cost spikes.
When it comes to rate limiting, SharePoint is one of the strictest platforms there is. This has long made SharePoint one of the slowest for data migrations, and that was simply the way it is. Until Movebot developed a new method to increase these speeds.
SharePoint Ludicrous Mode is a solution that uses the SharePoint Migration batch API to significantly increase migration speeds when using Movebot. With this turned on, you can expect to see five times the amount of files moved per hour or even more, drastically speeding up SharePoint migrations.
This feature is currently in beta and we recommend talking to Support before enabling.
While Movebot has some ways to work around some provider limits, hardware limits are a different story. Since Movebot simply adds more machines to your job if the transfer requires it, compute power isn’t an issue with the software.
However, if you have a 15-year-old server with data you want to move to the cloud, that machine needs to process everything on its end which can take time. It’s similar to moving with a truck that has a speed limiter on it. Even on the Autobahn, a truck that physically can’t go faster than 60 MPH is stuck at that speed, regardless of the speed limit.
To help alleviate this, we normally recommend using a Virtual machine to run the migration of on-prem servers.
| vCPUs | 4-8 |
| Memory | 16GB |
| HDD | 64GB |
But ultimately, if the server is smoking, the migration is going to crawl.
More important than server performance, is having a reliable internet connection, and not just for watching Netflix. If the connection is slow because you don’t want to lose your $10/month dial-up plan grandfathered in from 1994, your migration is going to crawl.
If you’re planning a transfer with a lot of data and have a server connected with anything under 100mb/s, it might be worth upgrading the plan if possible to speed up the job. If you aren’t able to do that, keep in mind that your connection speed is going to limit your transfer speed.
Based on our experience, transfer speed in Movebot tends to average about half of the theoretical uplink. The table below shows how different speeds affect how long it should take to move 10TB of data.
| Data Rate | Expected daily data transfer with Movebot | Expected time to move 10TB | Comments |
| 28.8 kb/s | 1.24 GB/day | 22 Years | You... have... got... to... be... kidding... |
| 50 Mbit | 270 GB/day | 1.25 Months | Slow and steady won't win this race |
| 100 Mbit | 540 GB/day | 18 Days | A speed boost would be nice |
| 250 Mbit | 1.35 TB/day | 1 Week | On the slow side, but may be manageable |
| 500 Mbit | 2.7 TB/day | 3-4 Days | Looking reasonable |
| 1 Gbit | 5.4 TB/day | 2 Days | Fastly furious |
| 10 Gbit | 54 TB/day | 4.5 Hours | It's Turbo Time! |
If you are unsure what your current bandwidth is, you can run a free speed test.
When it comes to speed, provider limits can greatly affect how fast your transfer is. But all of those are assuming that you are using traditional file storage. Block storage is an exception to the rule and can be significantly faster.
By “block storage”, we’re referring to S3 based storage. When Amazon developed S3 storage, they changed the game. Now, cloud platforms are following the trend and building block storage engines that conform with the S3 standard or come pretty close. Many have already made the transition.
The most noticeable feature with block storage is performance. In this case, speed. It’s fast. Blazingly fast (most of the time). Because it’s so fast, migrating into and out of block storage is blazingly fast too.
Block storage typically does not rate limit and performance can be scaled linearly. This allows the potential for incredible speeds. It is not unheard of for Movebot to transfer over 100-200TB of data per day on a S3 to S3 (or similar) migration.
If you want to get 5TB/day or more from your migration, there are a few settings you can optimize in Movebot that emphasize speed. One thing to take note of when setting up the migration is the geographic region; make sure to pick the one closest to where your data is stored for max speed.
As for optimization during a migration, as a next-gen cloud data migration tool, Movebot handles all of that for you. If it notices an opportunity to speed up your transfer, the systems will take advantage of the optimizations automatically. Once you've hopped on the Autobahn, you can let the driver handle the rest for you!
After you’ve connected the platforms and begun your data migration, you can see stats about the transfer right there in Movebot. The system collects stats like amount of data transferred, errors and failures, and much more. Summary stats give you an overall picture of the migration, as shown below.
For checking and debugging performance issues, the Performance Advisor is your best friend. Available under your migration project, it provides real-time stats as well as recommendations and an assessment on what is limiting your migration.
Along with the summary stats and Performance Advisor, Movebot has graphs with real-time stats on the following:
Rate limiting errors/warnings
Average upload and download time
Data transfer rates
Errors and error rates
vCPU usage for the moveworkers
This means you can pop your head in at any time to check your stats and see how much time you can expect the rest of the job to take–often less time than it says.
And, if you’re in the midst of a migration and aren’t sure where to look for the information you need, or transfer speeds aren’t as fast as expected, feel free to reach out to us. Our team are experts and have a few tricks up our sleeves.
At Movebot, when we say our data migrations are fast, we mean it. You can expect to move 2-5TB per day, but it could be much higher moving between certain platforms with an optimal setup.
Ready to move with a faster data migration tool? You can make an account and connect your storage for free, including running a pre-migration scan to see how much data you have and learn about potential concerns before the transfer. Start a 14-day Movebot trial and find out why it's a leader for giving actionable insights and agility for data.