Data migration: what does it mean, and why is it such a big deal?

Data migration, in technical terms, refers to the process of extracting, transforming, and loading data from the existing systems into the new ERP platform. The task can be complex, considering the volume of information being processed from various sources.

ERP migration projects pose multiple challenges, the first of which is data integrity and redundancy. To avoid a longer migration process, it’s vital to ensure the information transferred to your new tool is not corrupted or duplicated.

Not all data is the same. We can differentiate several types of data:

• Entity records – these include Customers, Vendors, and related fields
• Item type-related records: these include components and styles that you want to track in the system
• Inventory records: these are actual inventory items derived from item types, so it's impossible to import inventory if the item types have not been properly set up in the system beforehand
• Historical transactions – order history, purchase order history, sales history.  These are the ones loaded with dynamite; proceed carefully!

I’m a start-up, starting fresh with “nothing” vs I want it all

There are cases when you don't want to import any data. But hold on for a moment. Is that really the case?

Well, as experienced jewelry ERP implementers, let us say that there is no such thing. There may be situations where only a very small amount of data needs to be imported, but there is no scenario where nothing needs to be imported.

In the case of migration, there are two opposite extremes.

1. One is when a given jewelry company believes that no data migration is necessary at all. The argument fails there because even a start-up company needs to import at least item types and possibly inventory data, otherwise there will be nothing to transact within the system. It's like deciding to cook a nice dinner for your partner but finding every page in the recipe book you picked up from the shelf is blank. Nice candles on the table will not save the night.

2. The other extreme case is when we want to migrate all historical data, going back 20 years. Why is it not recommended to do this?

Firstly, it is an incredibly huge task to review and clean up this massive dataset, and secondly, we burden the new ERP system with a large database from the outset. The larger the database, the slower the system, this is a paradigm. Just think about it: every time you open the system and search through 20 years of data, it needs to index it all. While there are optimization solutions for this, they don't provide a definitive solution for speed. So, let's think twice about how much historical data we want to migrate.

Indeed, less is sometimes more!

Assess and cleanse data

Anyone who has gone through this process once will shake their head and say, never again. They'd rather not even consider a new ERP. Obviously, this is just a joke, but it illustrates how big of a task this is.

Let's be clear: Preparing data for import cannot be the responsibility of the ERP solution provider because they do not know the "real situation," and the end result will not be what you expect!

Certainly, the ERP team can improve the data structure, such as technically incorrect data like incorrect SKUs, but they won't recognize if there's a duplication.

Even if it's a big task, never import data without first checking and correcting it! Without completing this step, it's like stepping out of the frying pan into the fire!

The steps of data migration into PIRO Fusion

Certainly, we also have a well-developed procedure for this process. Our most important tool for data migration is the Import tool (which we will soon completely overhaul) available in Fusion . But this is just the technical part.

The process itself looks like this:

• First, we need to prepare the appropriate fields and data points in the system. It's a mistake to think that the system already knows the same data structure as we do. Mostly, it does, and that's true, but our experience shows that every company uses a slightly different data structure, so custom fields are needed to tailor the data structure to each company. Once this is done, we can export an import sheet template from Fusion, which will contain the basic data points as well as the custom fields prepared for you. This explains why we can't provide you with import sheet templates right at the beginning of the project.
• In the second step, it's your turn to act. You need to export the data from the previous system to a CSV file and start converting it into the PIRO import sheet. This conversion provides a perfect opportunity to organize and cleanse the data mentioned earlier. Let's take a moment of silence to honor anyone tasked with sorting and cleansing a list containing 10,000 SKUs. They're a hero!
• In the third step, you need to validate the data's correctness and consistency. You can also do this with the PIRO import tool. This is an important step because there can be trivial errors in our data, such as an extra space in the SKU accidentally entered during copying. The system cannot catch every single error, so it's not appropriate to rely on the import tool to validate and only correct the erroneous data. As the saying goes, "a stitch in time saves nine." If we import incorrect data into the system and need to clean it up before we can re-import the correct data, everyone gets a little frustrated. The PIRO team because it's extra work, and you because it's extra cost.
• Finally, the PIRO team imports your data, and then it's time to celebrate. Cheers!

Do it correctly, at the right time

Finally, let's discuss how data migration will occur in multiple waves during the implementation.

There is very simple data, such as customers and suppliers. These can be imported almost anytime.

Then there are the complex data, like item types, where many factors are involved, numerous attributes need to be associated with them, and often images need to be associated with them. Usually, this can only begin once the system's background setup is ready for it.

Typically, inventory items are imported last, just before the go-live, because there's no point in seeing inventory quantities from months ago in the system. Sometimes, test inventory items can be imported during implementation, but the real import will happen directly before the go-live.

We hope this blog post has inspired you to tackle some data import. If you need a new ERP and can't avoid the migration part of it, why not approach it with a positive attitude and great enthusiasm?