Movebot Blog | Data Migration Tool

AWS S3 Migrations | Movebot - Data Migration Tool

Written by Dallen Clark | 8/2/24 2:27 AM

With massive scalability, a flat hierarchy, and customizable metadata, AWS S3's object storage system is an excellent option for backups, content hosting, big data, and more. However, the different data structure compared to traditional file storage might make it seem like a challenge to move large amounts of data in and out, such as in a data migration project. But AWS S3 migrations are no problem with Movebot. Move large amounts of data with a fast, simple, and reliable data migration tool that just works. 

Moving between Amazon S3 and other platforms with  Movebot

Movebot supports moving between over 30 cloud storage platforms, including block storage like Amazon S3. And because Movebot is platform agnostic, you can freely move data between any of them--including moving emails from Imap, Google Workspace Gmail, and Exchange into S3 for backup and archival. Whether you're migrating from another block storage system or a cloud file system, it won't be a problem for Movebot. 

 

Moving between block storage systems

Along with the major players in cloud storage, Movebot supports several different block storage platforms. It's simple to move between S3 and Lyve Cloud, Wasabi, Backblaze, Azure, and more. Not only can Movebot do these migrations, but it can do them fast

When block storage is both the source and destination, data migrations are incredibly fast. We've seen speeds of over 100TB in a day, meaning no matter how much data you have you aren't going to be left waiting around for the project to finish. 

 

Moving to and from cloud file storage

S3 migrations with cloud storage are simple as well. Simply specify the S3 bucket and which directory you want to move to/from when setting up your transfer and Movebot will handle the rest. 

One benefit of using Movebot is that you don't have to do full organization migrations. Instead, you can choose what data to move down to the folder level. Because Movebot can do this, you can set up regular large-scale transfers.

For instance, you might have a Google Shared Drive with contents you want to be moved into S3 on a quarterly basis for a backup. Instead of having to move everything in the Google Workspace account--which you wouldn't want to do in this instance--you can instead move just that drive into an S3 bucket. You can then use bucket policies to automatically sort data through different parts of the lifecycle stage without having to worry about doing these tasks manually.

 

What about AWS migration tools? 

AWS migration tools have some significant limitations compared to dedicated data migration tools, but they can do the job for specific migration scenarios. 

For example, support for Amazon WorkDocs is ending in April 2025. Because Amazon is shutting down the service, they're helping customers get out of WorkDocs and into S3 with their own data migration tool. This involves setting up specific bucket policies and then moving all of the data by initiating the migration from within the AWS WorkDocs console. Amazon doesn't charge for using this tool. 

However, even in this specific scenario, there are some caveats. The biggest is that the data can only be moved into an S3 bucket. If you want to move to another storage platform, Amazon recommends downloading and transferring to another provider. This extra step could also add data egress costs, along with the extra time it will take using S3 as a temporary store. Instead, using a data migration tool like Movebot would immediately save time and money by skipping this step. 

Another issue is that you have to move all the data as is into the S3 bucket. You can't choose specific folders or files to move or leave behind, it's all or nothing. So while in specific cases using the AWS migration tools makes sense, oftentimes it will save you time, money, and headaches to use a dedicated data migration tool instead. 

 

Setting up an AWS S3 migration with Movebot

Movebot uses a simple three-step process for migrations, no matter what your source and destination are. By following the Plan, Advance, Cutover plan, your S3 migration will be set up and ready to go quickly. 

 

Connecting Amazon S3 to Movebot

Before setting up any migrations, you'll need to connect your Amazon S3 account and decide if you want to specify a specific bucket. This only takes a few minutes and is only required when making the initial connection, which can then be reused over multiple projects. 

To connect S3 to Movebot, choose to make a new storage connection in Movebot. Choose Amazon S3 and give it a name. You'll need your S3 Access Key and S3 Access Key Secret to connect, and you can also specify a particular S3 bucket. If you aren't sure about where to find these access keys, see this guide from Amazon on Managing access keys for IAM users

Once you paste the keys in Movebot, test that the connection is working and you can move on. 

 

Plan

Because Movebot lets you decide what you actually want to move, part of the planning phase is deciding what files you want to include. Depending on how the files are organized, this might be as simple and straightforward as choosing a specific drive or excluding a few high-level folders. But what if the data is less organized or you aren't quite sure how to find it all? 

This is where Movebot's filtering and Rules come into play. If you know where the objects are that you want to ignore, you can select those when setting up your migration process. If not, you can use content exclusion rules such as "File extension matches" to find files that meet your criteria. Movebot has several different filters you can combine to isolate files that you want, and then you can choose whether to include or exclude those files. 

Movebot can solve many common migration issues for you automatically as well. File name incompatibilities and doctype conversion are two of these. Movebot uses practical defaults to solve common migration concerns like these automatically, although you still have the choice to disable or change these default settings in the expert options. 

 

Advance

Since Movebot doesn't make you move everything at once, you have a lot more control over the migration process. We recommend always starting with some test data or a very small portion of your whole project and then making sure the data with S3 and the other storage platform functions as expected once moved over.

One thing to note is that Movebot doesn't actually move data like the name implies; instead, it's more like a giant copy-and-paste option. Movebot makes a copy of the data from the source and moves it to the destination. This means if you want to stop using the source, you'll need to later delete the data in the S3 bucket or in the other source platform manually. The reason we don't move data is because there are a lot more risks, including the potential for data loss. That risk is practically non-existent when making a copy because the source data remains untouched. 

After running some tests, you can then specify individual transfers to move to and from S3. A common strategy to break up large migrations is by user or drive, by moving the source into an equivalent location in the destination. You can run multiple simultaneous transfers, and since Movebot charges purely on the amount of data moved, there are no downsides to breaking up your S3 migration into manageable pieces that suit your migration plan.

Movebot will also let you schedule transfers so they can happen at an ideal time, although there's no need for users to stop working in the source even while the migration is running. Because the files are copied instead of moved, the source files aren't ever tampered with, and any changes users make while the migration is running can be pulled across in the Cutover phase with delta migrations. 

 

Cutover

When the bulk of the data is moved over, you can start cutover for your S3 migration. Once a transfer is finished, Movebot will generate a post-transfer report that shows the amount of data transferred, number of files copied, and more. It will also list of any objects that failed to transfer (if there are any) and the reason why. From the report, you can rerun failures or start a delta migration. Or you can do either of these later from the Transfers section of your project.

These delta migrations are a little bit different in Movebot. Delta migrations check for differences in the source and destination and only move new and updated files across. This means that you can have a massive migration in the petabytes running for weeks and still have users add terabytes worth of data to the source, then move only the updates instead. Organizations don't have to stop operations or pay for a second full migration, only for the amount of data that is transferred. 

 

Make your AWS S3 Migrations easy with Movebot

Movebot is all about speed and simplicity, and your Amazon S3 Migrations will be fast and straightforward whether it's the source or the destination. Move terabytes (sometimes dozens) between S3 and other storage with Movebot's easy-to-use web interface, with no infrastructure to set up, manage or maintain. Just set your connections, choose what to move, and hit go. 

Want to make sure Movebot is right for your project? Give it a try yourself, with 250 GB free at sign-up. There's no credit card, sales call, or demo required either if you don't want it; you can simply sign up and start moving data in minutes. Register for your account now to get started.