Group migration by direct transfer

Introduced in GitLab 13.7.

Group migration by direct transfer is the evolution of migrating groups and projects using file exports. The goal is to have an easier way for the user to migrate a whole group, including projects, from one GitLab instance to another.

Design decisions

The following architectural diagram illustrates how the Group Migration works with a set of ETL Pipelines leveraging from the current GitLab APIs.

ETL

ETL, for extract, transform and load, is a data integration process that combines data from multiple data sources into a single, consistent data store that is loaded into a data warehouse or other target system.

Using ETL architecture makes the code more explicit and easier to follow, test and extend. The idea is to have one ETL pipeline for each relation to be imported.

API

The current project and group imports are file based, so they require an export step to generate the file to be imported.

Group migration by direct transfer leverages the GitLab API to speed the migration.

And, because we're on the road to GraphQL, Group migration by direct transfer can contribute to expanding GraphQL API coverage, which benefits both GitLab and its users.

Namespace

The migration process starts with the creation of a BulkImport record to keep track of the migration. From there all the code related to the GitLab Group Migration can be found under the new BulkImports namespace in all the application layers.