Author: David Masri Founder & CEO

This is a question I get asked a lot, and my answer is always the same: "By far, the biggest mistake you can make when migrating data to Salesforce, is thinking of your data migration as a one-time task."

Now you may be thinking: "Aren't data migrations one-time by definition?" this is wrong, or at least it's not the whole truth. Let me explain.

From a strictly definitional perspective, a data migration is when we need to move data from one system (legacy or source) to another (target), where the target system will wholly replace the functionality of the source system. So, after the data is moved over, and the new system goes live, there is no need to work in the legacy system any more (at least for the functionality that was migrated). So, this is a "one-time" data movement. But this is the wrong way to think about it. "One-time" should be thought of as "one-time to production." What about our migrations for the QA and UAT rounds? Those count!

The larger issue here is the consequences of this kind of thinking, thinking of data migrations as one-time-run-and-done implies that somehow, it's OK to hold data migration code to a lower standard than any other piece of code. Additionally, it leads to:

1. Thinking that you can start planning your data migration a week before go live, it's the "just ask our client for CSVs of the data and we'll load them in" approach. (No, Start Early.)
2. Thinking that a long, complex manual process is OK. (It's not.)

I have seen this story repeated so many times I've lost count, it goes like this:

The person performing the data migration started off in a smaller shop, working on smaller projects with small data migrations. They work with small datasets that can be manipulated easily in Excel and need minimal transformations. So, they can manage the migration with a short half-page checklist (sometimes memorized), and they can run through that checklist in an hour or two with minimal risk of error. Essentially, they are leveraging a bit of planning to make up for automation. Then, one day, they are assigned to a monster project with a ton of data and lots of complex data transformations. As time goes by, Excel craps out, and the checklist grows and grows until the point where it takes a three and a half days to run through the checklist, and the entire process turns into a nightmare.

Performing data migrations off long check lists of procedures never turn out well. During QA and UAT defects are found and rather than fix the process (code) they fix the loaded data, then when the production migration is done, a bunch of the same defects found during QA & UAT resurface and the client is fuming. Then a series of emergency data fixes are performed directly against production, sometimes continuing for months after go-live. It's a complete nightmare.

There is simply no way to guarantee that a long manual process will be done correctly and consistently. Your data migrations need to be as automated as possible.

If a migration is not automated, then all the rounds of QA/UAT are, essentially, nullified. Think about it. Suppose we spend two days running carefully through a four-and-a-half-page checklist of actions to perform a data migration. Then, our client performs UAT and finds a few defects. We patch the data, and the client signs off on the patches. Now we are ready to perform the final data migration (because users have been working in the legacy system all this time, making updates). All users are then locked out of the legacy systems and we are given new data dumps to load. We spend another two days carefully running through the same checklist. How do we-or our client—know that everything is okay? - Another full round of UAT.

People will complain that building automated code takes too much time. You should not accept this view, automation saves time and produces better, more consistent results. Truth be told, even if the code really was a one-time run, automating it is not a time-consuming task. We have to code the transformations anyhow, so we may as well clean up and save the code.

Much like when building a rocket that will carry a robot to Mars "one-time", we need to hold out data migration code to the highest of standards. "One-time" is not an excuse for haphazardness.

This article is adapted from my book: Developing Data Migrations and Integrations with Salesforce.

Have a question that you feel will be a good topic for my Data Migration and Integration with Salesforce FAQ article series? Send me a message!

Have a question you would like to as see a part of my FAQ blog series? Email it to me! - Dave@SalesforceDataBlog.com