Picture this: a busy office is switching storage providers and have terabytes of files they need moved over. Stopping operations is a no-go, because there’s too much work that can’t wait. So the migration is scheduled for over a weekend, and everyone involved prays everything goes perfectly and makes it over before the next business day.
Stressful, right? Especially for the people in charge of the migration. Fortunately, there’s a way to remove this stress entirely, a way for users to keep doing business as usual while all the files they create and modify magically appear at the destination, updated and ready to go.
So how is this possible? It’s all thanks to something called a Delta Migration.
Delta Migration defined
A simple way to define a Delta migration is that it’s a migration that only moves files that are different in the source than the destination. Delta simply means “change” or “difference” here, which is what you move during a delta migration.
Where the specifics change is how the migration tool defines “difference”. In a lot of tools, it’s simply checking for brand new files made while the migration was in process. These newly-made files are then moved across.
With other tools like Movebot, delta migrations are more advanced and can check for other differences to make sure the most up-to-date version of the file is at the destination.
Why do delta migrations matter?
Delta migrations are the best method to get data from one platform to another, especially when the source is still being used. Without delta migrations, to move new files over you’d have to move all the files again. Moved 10 Terabytes and got 200GB of new files you need to move? Without delta migration functionality, that’s 10 terabytes again.
Obviously, that’s insane, but if the tool doesn’t support delta migrations you’re stuck with redoing the migration or finding another way to move the data. For a few dozen GB it would be easy enough to find a way to get those files, but what about when hundreds of gigabytes or even terabytes of files were updated during the migration? Redoing a migration of that size isn’t feasible in time or cost, so delta migrations are the only real option.
Methods of Delta Migrations
There are a few different ways that data migration tools handle delta migrations. Delta migrations tend to either move across brand new files, or new files and modified files.
Moving new files only
One method is to check the source and destination and only pull across new files. It’s a straightforward and simple approach but has some drawbacks. If nobody is working in the source while it’s running, like a small migration done over a weekend, this method is perfectly fine. The delta migration will check for any new files and move them and everything will be up to date.
Where this is a problem is when users are actively working in the source. This is common in larger projects of several terabytes or more, when operations can’t simply stop because of a migration. So what happens to updated files in this scenario? If all the delta migration does is move over new files, any updated files won’t show up in the destination. This puts the responsibility on the user to keep track of what files they changed and where they’re located if they want the most up-to-date version in the destination.
Moving new and modified files
A solution to this is the second method of delta migrations that moves newly-modified files. Along with checking files that don’t exist in the destination, this kind of delta also checks each file’s modification time. If the file at the source was updated more recently than the one in the destination, the delta migration will move and overwrite the older one in the destination.
With this option, users can work on the source as long as they need to and the newest version will be saved. Another way to think of it is a one-way sync: every delta migration will move a newer version at the source to the destination but won’t move destination files back to the source.
Delta migration speed
There's a cost with moving data, and the more data you move, the higher that is. Having to continuously do mass migrations in place of delta migrations would be costly and not a productive project. Even with just 1TB of data, without delta migration functionality you'd have to pay to move that 1TB each time, which is just wasting money.
With delta migrations, as you are only moving files that have been modified, often this is in gigabytes or even megabytes in place of terabytes of data. Because you are only moving small amounts of data and only data that has been modified, your overhead and overall cost will be significantly lower.
Speed and cost
One variable that might not seem relevant at first glance for delta migrations is speed. So how does speed play into everything? Speed is one of the most important parts of a migration, and not just for the direct benefit of not having to wait around for the job to finish. But that’s a good perk too.
Migration speeds also have a huge impact on disruptions and cost. For instance, one migration best practice is not to restructure folder structures while the migration is running because doing this would make the migration tool see the file as new and copy over a duplicate during the delta migration. But if a slow migration tool (not Movebot) estimates it will take a year to finish the job, it can be a big ask to have the entire organization avoid restructuring for that entire time.
Since a delta migration only transfers modified files, the amount of files being transferred is tiny in comparison to the initial migration. However, speed is a major factor here too. Slower speeds give more time for new files to be added, which increases the volume, which then must all be moved in the delta migration. For slower migrations with large data volumes this can add up quickly, which can cost thousands of extra dollars from “business as usual” simply by being slow.
Even when the tool supports delta migrations, if it’s slow or only moves new files, you could still be out a lot of unnecessary time and money. Fortunately, Movebot does things a bit differently.
Delta migrations with Movebot
Having powerful delta migration functionality can save a lot of trouble for migrations, so Movebot has several options to make sure the newest version of the data arrives at the destination and gets there fast.
First, Movebot’s delta migrations move both new and modified files, so feel free to update files while a migration is running. Delta migrations can also be run on a folder level instead of needing to be done for an entire drive or organization. This means faster, more efficient delta migrations because Movebot doesn’t waste time and resources rescanning and rechecking folders that definitely haven’t been updated recently.
The primary reason why service providers and companies use delta migrations with Movebot is to ensure that the data has made it across and is up to date prior to teams picking up the new platform. So for larger migrations--like moving 20 terabytes of data--this isn't going to appear overnight. We recommend, depending on how much data you have, that you do the mass migration a week or two prior to your go-live date.
Once the first wave has completed, we recommend enabling regular delta migrations (or a sync) to ensure that any modified files have been migrated across. It would typically look something like this:
- Wave 1. Initial mass migration, this would move all of your data, content and users across to the new platform. This is a large-scale migration that will move everything in the cloud and can easily be in the terabytes.
- Wave 2-n. At 3-4 day intervals run a delta migration (sync) as required until the weekend before going live with the new platform.
- Final cutover. Over the weekend prior to going live of the new platform, run a final delta migration so that all of the data is up to date. After this, users can start working in the new cloud going forward.
Always Use Delta
Always Use Delta is a setting in Movebot that uses the same delta migration concept for the entire project. So why would you want this? One common use case is for a one-way sync from one storage platform to another. Instead of having to identify files that are different in the source, Movebot can scan the source and find them for you. And this can be done down to the folder level if you only need specific directories to sync.
We often recommend turning this on even for new migrations, especially if the destination has already started being used. This will help make sure you don’t end up with duplicate files. And it works for full migrations too, because if the destination has no files Movebot will still see the “difference” so everything will still get moved across.
Fast delta migrations
Movebot’s speed is a huge benefit when it comes to delta migrations. Movebot typically moves terabytes per day between storage platforms, including for delta migrations. That means less time to add or update files, which means fewer files that need to be moved. And fewer files mean less time and lower volume, which also lowers costs, and also means fewer times deltas need to be run.
Make data moving easy with delta migrations using Movebot
Delta migrations are a neat feature that move only files that are different at the source than the destination, either because they’re new or have been recently updated. Delta migrations save you time and ensure that productivity is not impacted as the new platform can be adopted quickly and seamlessly by teams as if they were using the same platform. And Movebot has some of the best delta migration capabilities there is, with better speed, functionality, and options than other data migration tools.
Check out Movebot for yourself by registering for a free trial with 50GB free to move files or emails wherever you like. You can explore the tool, connect data sources, and even try moving data between platforms. There's no credit card required and you don't have to jump on a product demo call if you don't want to. Simply sign up and go. Get started with your trial today.
