Data Migration to Drupal Using Products from External Database - Guide

How can you perform a product data migration from an external database to Drupal using the tools available within the Migrate API? In this blog post, I’ll show you how to connect to the database, prepare the data structure, and use the migration tools available in Drush. This entry is aimed at people who have already had experience with migrations as well as those who are just getting started with them. I encourage you to read the article or watch the video of the “Nowoczesny Drupal” series (the video is in Polish).

In a previous blog article, we discussed importing data into Drupal from a CSV file. This time, we’ll focus on migrating from an external SQL database to the "products" content type in Drupal. This example shows how easy it is to import products from a database and how to deal with possible problems during the process itself.

Preparing the source for data migration

Our data is in an external database, in a table named "product." This table contains the following fields:

• Product ID - a unique identifier of the product,
• Title - the product name,
• Image - the path to the product image,
• Description - a brief description of the product,
• Category - the category to which the product belongs,
• Tags - additional tags assigned to the product.

For this article, I simplified the structure of this table by adding columns: "image," "category," and "tags." The IDs of categories, tags, and media were given directly in the table. This avoids the need for additional migrations of dictionaries and media files.

Setting up a data migration environment

To migrate product data, we first need to prepare the Drupal environment. For this guide, we’re using a clean installation of Droopler, to which we’ll add the necessary modules and create a custom module to handle the migration process.

Modules in Drupal needed for data migration

Data migration in Drupal is based on several modules that we need to activate. The first is Migrate, a basic module that is part of the core and provides basic data migration functions.

If we’re working with a larger database, the Migrate Plus and Migrate Tools modules will be useful. Migrate Plus is an extension that adds additional functionality to the migration process, and Migrate Tools is a toolkit that integrates migration with Drush, allowing you to manage the migration directly from the terminal.

Create your module for data migration

In addition, for this article, I created the Demigrate module. This is a module that was almost entirely generated using the Drupal Drush command - Devil Generate.

After it’s generated, it should be adjusted to our needs. We change the query and by the select field, instead of example, we give product:

The next step is to change the fields area by adding the appropriate fields from the table, namely "ID," "title," "image," "description," "category," and "tags."

I then generated a YAML file that maps fields from the source to fields in Drupal. This determines which fields in the database are to be assigned to the corresponding fields in Drupal. For example, the "title" field from the table will be assigned to the "title" field in Drupal, and "image" will be assigned to the "field_image" field.

Below, I show where in Drupal you can specify them:

Data migration process using Drupal Drush

When our module is ready, we can proceed with data migration. To be able to do this, we need to add a definition in setting.php to connect to the database.

Then we move to the terminal. Drupal, in conjunction with the Drush tool, offers several commands that greatly simplify this process. 

Checking the status of data migration

Before we perform the migration, we can check how many records are waiting to be imported by using the command:

drush migrate:status

This command displays the list of migrations with information about the number of records available for import. In our case, there are ten products in the table, which confirms the correctness of the prepared data.

Importing data

To import data, we use the command:

drush migrate:import --group=migration_products

This command allows you to import all products from the database into Drupal.

Once the import is complete, the products will appear in Drupal in the "Content" section.

If we’d like to import only a single record, we can use the "limit" option:

drush migrate:import --group=migration_products --limit=1

The data migration rollback

Sometimes, it happens that the migration goes wrong, for example, due to incorrect data in the product description. In this case, we can undo the entire migration using the command:

drush migrate:rollback --group=migration_products

After this operation, all imported records will be deleted from Drupal, and we can start the migration again, after correcting the data.

Update data after migration

Migration doesn’t always mean a one-time process. Often, after importing data, there is a need to update it. An example would be when changes have occurred in the old database, and we need to update the content in Drupal.

We can do this with the command:

drush migrate:import --group=migration_products --update

This command will update only those records that have been changed in the source database, without the need to import the entire dataset from scratch.

Data migration to Drupal - summary

Migrating data to Drupal may seem complicated, but with the use of Migrate API, Drupal Drush and tools such as Migrate Plus and Migrate Tools, the whole process becomes much simpler and more efficient. It’s important to properly prepare the data, generate a custom module, and know the basic Drush commands. With these Drupal tools, you can efficiently transfer data from external sources to Drupal, ensuring the continuity of the system and minimizing the risk of errors.

Importing from a database doesn't have to be difficult - just follow the steps above, and the process will go smoothly. If you have any questions, our data migration experts will be happy to help.