Migrate nodes and vocabulary from Drupal 7 to Drupal 8
In this blog we will explain a migration from drupal 7 to drupal 8. Everything you need to know before starting your first migration and troubles you will have to deal with.
First step is to download the needed modules - there are some drupal contrib modules that allow you to do a drupal migration in the most cleanest way, and the process will be up to drupal standards.
Drupal Migrate
Drupal Migrate UI (optional)
Migrate
Migrate Upgrade
Migrate Tools
Migrate Plus
These are:
What is migration and how it works
This is a common approach to a migration. We have a source database (doesn't necessarily need to be a database, it can be CSV, XML, JSON etc.) and we have a destination database which is a Drupal 8 database. As you can see on the diagram between these two databases (source and destination) we have a list of ordered operations.
The first one is a getting the data, it’s a query system which lets you get the data from the source database. Mapping goes next, where you can set which field from the source database should go (and where) to the destination database. The next step is processing, in this operation, we can change the data taken from the source db. For example, we need to change a format of the taken data to fit new structures in Drupal 8 (It is actually a preprocess for each row that is getting migrated). The last one are settings, it’s a part of the destination object that sets the data in the structures of the destination database (Drupal 8 database structure).
Migration: The process of moving content from one site to another. ‘A migration’ typically refers to all the content of a single content or entity type (in other words, one node type, one taxonomy, and so on).
Migration Group: A collection of Migrations with common traits
Source: The Drupal 6 or 7 database from which you’re drawing your content (or other weird source of data, if applicable)
Process: The process which Drupal code does to the data after it’s been loaded, in order to digest it into a format that Drupal 8 can work with.
Destination: The Drupal 8 site
KEY PHRASES:
Interestingly, each of those key phrases above corresponds directly to a code file that’s required for migration. Each Migration has a configuration (.yml) file, and each is individually tailored for the content of that entity. As config files, each of these is pretty independant and not reusable. However, we can also assign them to Migration Groups. Groups are also configuration (.yml) files. They allow us to declare common configurations once, and reuse them in each migration that belongs to that group.
The Source Plugin code is responsible for doing queries to the Source database, retrieving the data, and formatting it into PHP objects that can be worked on. The Process Plugin takes that data, does stuff to it, and passes it to the next step. The Destination Plugin then saves it in Drupal 8 format. Rinse, repeat.
On a Drupal-to-Drupal migration, around 75% of your time will be spent working in the Migration or Migration Group config, declaring the different Process Plugins to use. You may wind up writing one or more Process Plugins as part of your migration development, but a lot of really useful ones are included in Drupal core migration code and are documented here.
A few more are included with Migrate Plus.
Drupal 8 core has Source Plugins for all standard Drupal 6 and Drupal 7 entity types (node, taxonomy, user, etc.). The only time you’ll ever need to write a Source plugin is for a migration from a source other than Drupal 6 or 7, and many of these are already available as Contrib modules. Also included in Drupal core are Destination Plugins for all of the core entity types. Unless you’re using a custom entity in Drupal 8, and migrating data into that entity, you’ll probably never write a Destination Plugin.
Let’s start with our first migration
Title
Body
Field_tags (Term reference)
Field_image (Image field)
In this example we will migrate Blog content type with fields listed below:
Tags vocabulary with tag terms.
First step
Install vanila drupal 7 site, create vocabulary ‘Tags’ fill up with some optional terms and create an content type ‘Blog’ that contains mentioned fields. After that, create few nodes of blog content type and populate it with some dummy content. When you finish that export database (we will use this database as source).
Second step.
Install a vanila drupal 8 site, and also create a content type ‘Blog’ that contains mentioned fields. It would be nice if you could export the configuration after creation within a custom module that will be used for a custom migration or into a global config/sync folder.
SNote: During migration process you will often have to delete your configurations from ‘config’ table in drupal 8 database (this configuration is provided by your custom module). For every modification in your .yml config file you have to uninstall and then install module again.
In this example I’ve worked with a docker local environment. If you prefer working with docker then you can use this setup for drupal 8 migration.
Be aware that your will need two mysql / mariadb containers. One for drupal 7 database and another one for drupal 8 database.
version: "3"
services:
mariadb8:
image: wodby/mariadb:$MARIADB_TAG
container_name: "${PROJECT_NAME}_mariadb8"
stop_grace_period: 30s
environment:
MYSQL_ROOT_PASSWORD: $DB_ROOT_PASSWORD
MYSQL_DATABASE: $DB_NAME
MYSQL_USER: $DB_USER
MYSQL_PASSWORD: $DB_PASSWORD
volumes:
- ./databases:/var/lib/mysql/databases
# volumes:
# - ./mariadb-init:/docker-entrypoint-initdb.d # Place init .sql file(s) here.
# - /path/to/mariadb/data/on/host:/var/lib/mysql # Use bind mount
mariadb7:
image: wodby/mariadb:$MARIADB_TAG
container_name: "${PROJECT_NAME}_mariadb7"
stop_grace_period: 30s
environment:
MYSQL_ROOT_PASSWORD: $DB_ROOT_PASSWORD
MYSQL_DATABASE: drupal7
MYSQL_USER: drupal7
MYSQL_PASSWORD: drupal7
volumes:
- ./databases:/var/lib/mysql/databases
# volumes:
# - ./mariadb-init:/docker-entrypoint-initdb.d # Place init .sql file(s) here.
# - /path/to/mariadb/data/on/host:/var/lib/mysql # Use bind mount8
php:
image: wodby/drupal-php:$PHP_TAG
container_name: "${PROJECT_NAME}_php"
environment:
PHP_SENDMAIL_PATH: /usr/sbin/sendmail -t -i -S mailhog:1025
DB_HOST: $DB_HOST
DB_USER: $DB_USER
DB_PASSWORD: $DB_PASSWORD
DB_NAME: $DB_NAME
DB_DRIVER: $DB_DRIVER
COLUMNS: 80 # Set 80 columns for docker exec -it.
## Read instructions at https://wodby.com/stacks/drupal/docs/local/xdebug/
PHP_XDEBUG: 1
PHP_XDEBUG_DEFAULT_ENABLE: 1
# PHP_XDEBUG_REMOTE_CONNECT_BACK: 0
# PHP_IDE_CONFIG: serverName=my-ide
# PHP_XDEBUG_REMOTE_HOST: host.docker.internal # Docker 18.03+ & Linux/Mac/Win
# PHP_XDEBUG_REMOTE_HOST: 172.17.0.1 # Linux, Docker < 18.03
# PHP_XDEBUG_REMOTE_HOST: 10.254.254.254 # macOS, Docker < 18.03
# PHP_XDEBUG_REMOTE_HOST: 10.0.75.1 # Windows, Docker < 18.03
volumes:
- ./:/var/www/html
## For macOS users (https://wodby.com/stacks/drupal/docs/local/docker-for-mac/)
# - ./:/var/www/html:cached # User-guided caching
# - docker-sync:/var/www/html # Docker-sync
## For Xdebug profiler files
# - files:/mnt/files
nginx:
image: wodby/drupal-nginx
container_name: "${PROJECT_NAME}_nginx"
ports:
- "9999:80"
depends_on:
- php
environment:
# NGINX_PAGESPEED: "on"
NGINX_STATIC_OPEN_FILE_CACHE: "off"
NGINX_ERROR_LOG_LEVEL: debug
NGINX_BACKEND_HOST: php
NGINX_SERVER_ROOT: /var/www/html
# NGINX_VHOST_PRESET: $NGINX_VHOST_PRESET
# NGINX_DRUPAL_FILE_PROXY_URL: http://example.com
volumes:
- ./:/var/www/html
# For macOS users (https://wodby.com/stacks/drupal/docs/local/docker-for-mac/)
# - ./:/var/www/html:cached # User-guided caching
# - docker-sync:/var/www/html # Docker-sync
# labels:
# - 'traefik.backend=nginx'
# - 'traefik.port=80'
# - 'traefik.frontend.rule=Host:${PROJECT_BASE_URL}'
mailhog:
image: mailhog/mailhog
container_name: "${PROJECT_NAME}_mailhog"
As you can see mariadb8 will create databases folder in your site root. You should copy d7 database into databases folder, so you can easily import sql dump file.
Now when your containers are up you will be able to import drupal 7 database into mariadb7 container. (in this case it is mariadb7) .
Your settings.php file should looks like this:
$databases['default']['default'] = array(
'driver' => 'mysql',
'database' => 'drupal',
'username' => 'drupal',
'password' => 'drupal',
'host' => 'mariadb8',
'prefix' => '',
'port' => '3306',
);
$databases['old_drupal']['default'] = array (
'database' => 'drupal7',
'username' => 'drupal7',
'password' => 'drupal7',
'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
'driver' => 'mysql',
'host' => 'mariadb7',
);
Note that key ‘old_drupal’ points on drupal 7 database, also host is mariadb7 that points to a docker container that contains drupal 7 database.
Now is time to create migrate_custom module.
Structure of module folder :
Create migrate_custom.info.yml file.
name: My Migrate
description: Performs an example of a simple Drupal-to-Drupal migration.
package: Other
dependencies:
- migrate
- migrate_drupal
- migrate_plus
type: module
core: 8.x
Now we need to group migration sections. Create migrate_plus.migration_group.d7.yml file.
id: d7
label: D7 imports
description: Migrations importing from the legacy D7 ya_example site
source_type: Drupal 7
shared_configuration:
source:
key: old_drupal
In this file the most important point is the source key, it is the name of source database (from settings.php file).
After that, we will create a component for the vocabulary migrate_plus.migration.custom_taxonomy_vocabulary.yml
id: custom_taxonomy_vocabulary
label: Drupal 7 taxonomy vocabularies
migration_group: d7
dependencies:
enforced:
module:
- migrate_custom
source:
plugin: custom_taxonomy_vocabulary
process:
vid:
-
plugin: machine_name
source: machine_name
-
plugin: dedupe_entity
entity_type: taxonomy_vocabulary
field: vid
length: 32
label: name
name: name
description: description
hierarchy: hierarchy
module: module
weight: weight
destination:
plugin: entity:taxonomy_vocabulary
Pay attention on source plugin ‘custom_taxonomy_vocabulary’ , this plugin will be triggered when you start this migration.
Let’s create migration plugin for vocabulary. Create Vocabulary.php file into migrate_custom/src/Plugin/migrate/source folder.
/**
* @file
* Contains \Drupal\migrate_custom\Plugin\migrate\source\Vocabulary.
*/
namespace Drupal\migrate_custom\Plugin\migrate\source;
use Drupal\migrate\Row;
use Drupal\migrate\Plugin\migrate\source\SqlBase;
/**
* Drupal 7 vocabularies source from database.
*
* @MigrateSource(
* id = "custom_taxonomy_vocabulary",
* source_provider = "taxonomy"
* )
*/
class Vocabulary extends SqlBase {
/**
* {@inheritdoc}
*/
public function query() {
$query = $this->select('taxonomy_vocabulary', 'v')
->fields('v', array(
'vid',
'name',
'description',
'hierarchy',
'module',
'weight',
'machine_name'
));
return $query;
}
Query method will select all vocabularies from source database. You can modify this by your needs.
Then we need two more methods ‘fields()’ and ‘getIds()’.
/**
* {@inheritdoc}
*/
public function fields() {
return array(
'vid' => $this->t('The vocabulary ID.'),
'name' => $this->t('The name of the vocabulary.'),
'description' => $this->t('The description of the vocabulary.'),
'help' => $this->t('Help text to display for the vocabulary.'),
'relations' => $this->t('Whether or not related terms are enabled within the vocabulary. (0 = disabled, 1 = enabled)'),
'hierarchy' => $this->t('The type of hierarchy allowed within the vocabulary. (0 = disabled, 1 = single, 2 = multiple)'),
'weight' => $this->t('The weight of the vocabulary in relation to other vocabularies.'),
'parents' => $this->t("The Drupal term IDs of the term's parents."),
'node_types' => $this->t('The names of the node types the vocabulary may be used with.'),
);
}
/**
* {@inheritdoc}
*/
public function getIds() {
$ids['vid']['type'] = 'integer';
return $ids;
}
As you can see there is just a field definitions. Now we have a taxonomy_vocabulary ready for migration.
Drupal Migration API Source plugin
Lets create terms migration. Create migrate_plus.migration.custom_taxonomy_term.yml file
id: custom_taxonomy_term
label: Drupal 7 taxonomy terms
migration_group: d7
dependencies:
enforced:
module:
- migrate_custom
source:
plugin: custom_taxonomy_term
process:
tid: tid
vid:
plugin: migration_lookup
migration: custom_taxonomy_vocabulary
source: vid
name: name
description: description
weight: weight
parent:
-
plugin: skip_on_empty
method: process
-
plugin: migration_lookup
migration: custom_taxonomy_term
changed: timestamp
destination:
plugin: entity:taxonomy_term
migration_dependencies:
required:
- custom_taxonomy_vocabulary
Pay attention on migration_dependencies. If you run terms migration before vocabulary migration it will automatically trigger ‘custom_taxonomy_vocabulary’. It makes sense right?
As we have to write plugin for vocabulary, we also need to make plugin for terms. In migrate_custom/src/Plugin/migrate/source folder create Term.php file.
namespace Drupal\migrate_custom\Plugin\migrate\source;
use Drupal\migrate\Row;
use Drupal\migrate\Plugin\migrate\source\SqlBase;
/**
* Drupal 7 taxonomy terms source from database.
*
* @todo Support term_relation, term_synonym table if possible.
*
* @MigrateSource(
* id = "custom_taxonomy_term",
* source_provider = "taxonomy"
* )
*/
class Term extends SqlBase {
/**
* @file
* Contains \Drupal\migrate_custom\Plugin\migrate\source\Term.
*/
/**
* {@inheritdoc}
*/
public function query() {
$query = $this->select('taxonomy_term_data', 'td')
->fields('td', array('tid', 'vid', 'name', 'description', 'weight', 'format'))
->distinct();
return $query;
}
/**
* {@inheritdoc}
*/
public function fields() {
return array(
'tid' => $this->t('The term ID.'),
'vid' => $this->t('Existing term VID'),
'name' => $this->t('The name of the term.'),
'description' => $this->t('The term description.'),
'weight' => $this->t('Weight'),
'parent' => $this->t("The Drupal term IDs of the term's parents."),
);
}
/**
* {@inheritdoc}
*/
public function prepareRow(Row $row) {
// Find parents for this row.
$parents = $this->select('taxonomy_term_hierarchy', 'th')
->fields('th', array('parent', 'tid'))
->condition('tid', $row->getSourceProperty('tid'))
->execute()
->fetchCol();
$row->setSourceProperty('parent', $parents);
return parent::prepareRow($row);
}
/**
* {@inheritdoc}
*/
public function getIds() {
$ids['tid']['type'] = 'integer';
return $ids;
}
}
The new method is ‘prepareRow()’ You can look at this as row preprocess. So each row that is about to be migrated you can preprocess.
Because we have file field in our Blog node (field_image) we need to migrate files before start migrate Blog nodes.
Files migration could be real nightmare, but in this example we will show you how to do it simply.
First we need migrate_plus.migration.blog_file.yml config file.
# Every migration that references a file by Drupal 7 fid should specify this
# migration as an optional dependency.
id: blog_file
label: d7 blog files
audit: true
migration_group: d7
migration_tags:
- Drupal 7
- Content
source:
plugin: d7_file
scheme: public
constants:
# The tool configuring this migration must set source_base_path. It
# represents the fully qualified path relative to which URIs in the files
# table are specified, and must end with a /. See source_full_path
# configuration in this migration's process pipeline as an example.
source_base_path: 'sites/default/files/migrate_files'
process:
# If you are using this file to build a custom migration consider removing
# the fid field to allow incremental migrations.
fid: fid
filename: filename
source_full_path:
-
plugin: concat
delimiter: /
source:
- constants/source_base_path
- filepath
-
plugin: urlencode
uri:
plugin: file_copy
source:
- '@source_full_path'
- uri
filemime: filemime
# No need to migrate filesize, it is computed when file entities are saved.
# filesize: filesize
status: status
# Drupal 7 didn't keep track of the file's creation or update time -- all it
# had was the vague "timestamp" column. So we'll use it for both.
created: timestamp
changed: timestamp
uid: uid
destination:
plugin: entity:file
migration_dependencies:
required:
- d7_users
Pay attention on
source_base_path: 'sites/default/files/migrate_files'
This is a path where you should copy old drupal 7 files folder.
migration_dependencies:
required:
- d7_users
This part of code is not necessary. It means that blog_file migration needs user migration first.
Now it’s time to copy whole sites folder from old drupal 7 site to new migrate_files folder. The folder structure should looks like example below :
Now we can finally migrate our Blog nodes from drupal 7 to drupal 8 site.
Create migrate_plus.migration.custom_blog.yml config file.
id: custom_blog
label: Custom article node migration from Drupal 7
migration_group: d7
dependencies:
enforced:
module:
- migrate_custom
source:
plugin: d7_node
node_type: blog
destination:
plugin: entity:node
bundle: blog
process:
nid: nid
vid: vid
type: type
langcode:
plugin: static_map
bypass: true
source: language
map:
und: en
title: title
uid: uid
status: status
created: created
changed: changed
promote: promote
sticky: sticky
body:
plugin: iterator
source: body
process:
value: value
format:
-
plugin: static_map
bypass: true
source: format
map:
- null
-
plugin: skip_on_empty
method: process
-
plugin: migration
migration:
- d6_filter_format
- d7_filter_format
source: format
field_tags:
plugin: migration_lookup
source: field_tags
migration: custom_taxonomy_term
no_stub: true
field_image:
plugin: iterator
source: field_image
process:
target_id:
plugin: migration_lookup
migration: blog_file
source: fid
alt: alt
title: title
height: height
width: width
migration_dependencies:
required:
- d7_user
- blog_file
Now we are ready to run migration.
I would suggest you to update your drush to 9.x version.
Enable migrate_custom module. (drush en migrate_custom).
Next, you will need to install the Migration Group entity type (drush entup).
Type ‘drush migrate-status’ it will list up all available migrations.
‘drush migrate:import migration_id’ will run migration
‘drush migrate:rollback migration_id’ will rollback migration data.
Now when you type drush migrate-status you should see something similar this
Run migrations by order listed below :
- Vocabulary (drush migrate:import custom_taxonomy_vocabulary)
- Terms (drush migrate:import custom_taxonomy_term)
- Files (drush migrate:import blog_file)
- Blog (drush migrate:import custom_blog)
Suggestions and Additional stuff
If the execution of the migration fails, it’s state may continue to say "Importing". Running the migration in this state again will give us an error message "Migration xx is busy with another operation". To fix this you can stop and reset the migration with the drush migrate-reset-status [migration_id] command.
Debugging process
Migrate debugging may be very difficult, you can’t debug yml files, but you are able to debug plugin processes. There are a few debug methods that may help you in this case.
View messages of a migration execution
When migrations are executed with Drush using contributed Migrate Tools module, the messages of a migration can be used with drush migrate-messages <migration> or alias drush mmsg <migration>.
Printing debug information when using Drush
If you execute migrations using Drush, you can print debug information using drush_print(). This can be included for example in the prepareRow() method of the source plugin.
xDebug method
If you prefer xDebug tool you can debug source plugin if you follow these instructions below.
In migrate_custom.module file create hook_preprocess_page() or some other hook. Be sure that this ‘other’ hook will be triggered in browser. Call migrate service in this hook, like in the example.
function migrate_custom_preprocess_node(&$variables) {
$migration_id = 'custom_blog'; //put breakpoint here
$migration = \Drupal::service('plugin.manager.migration')
->createInstance($migration_id);
$executable = new MigrateExecutable($migration, new MigrateMessage());
$a = $executable->import();
}
Also you can automate your migration if you call migrate service in migrate_custom.install file.
use Drupal\migrate\MigrateExecutable;
use Drupal\migrate\MigrateMessage;
/**
* Migrate import for tags vocabulary.
*/
function migrate_custom_update_8104() {
$migration_id = 'custom_taxonomy_vocabulary';
$migration = \Drupal::service('plugin.manager.migration')
->createInstance($migration_id);
$executable = new MigrateExecutable($migration, new MigrateMessage());
$executable->import();
}
/**
* Migrate import for tags terms.
*/
function migrate_custom_update_8105() {
$migration_id = 'custom_taxonomy_term';
$migration = \Drupal::service('plugin.manager.migration')
->createInstance($migration_id);
$executable = new MigrateExecutable($migration, new MigrateMessage());
$executable->import();
}
/**
* Migrate import for node blog.
*/
function migrate_custom_update_8107() {
$migration_id = 'custom_blog';
$migration = \Drupal::service('plugin.manager.migration')
->createInstance($migration_id);
$executable = new MigrateExecutable($migration, new MigrateMessage());
$executable->import();
}
That’s it! We’ve suscessfully ran the import, you can download the module here:
Next, in part 2, we’ll be going over paragraph migration.
