Skip to content

Update migration pipelines that use GraphQL API to extract data to use REST API or download ndjson files

As per @georgekoltsov comment

we should update existing migration pipelines that still use GraphQL API to extract data to use REST instead (to either use our REST API to fetch data or download ndjson files). This is used in 7-8 places.

More context from comment:

GraphQL is not completely abandoned and still used in a few places (very few) which provides a useful and straighforward way of fetching data from source GitLab.

For example:

  1. This is a first step in Group Migration. We fetch group information using GraphQL to create an empty group before importing things into it. For this step exporting an ndjson file is unnecessary.
  2. Same goes for Project Migration. Empty project information is fetched using GraphQL which I recently worked on.

This can be replaced by using REST API, of course. General rule of thumb I use is if data is 'flat' and does not have nested associations, GraphQL API can be used (or REST). In this case it's simpler to do 1 network request instead of downloading an ndjson, extracting/decompressing, etc.

I don't think we will be able to exclusively use ndjson and having different ways of 'data extraction' is good to have under your belt, since it's fairly easy to do in ETL. But it is true majority (if not all) of relations import for projects will be using NDJSON, simply because of the complexity of nested associations. This was the outcome of our transition to NDJSON in the first place #326757 (closed)

The only relation I can think of for Project Migration that can use GraphQL API is importing project members. We have similar MembersPipeline for Group Migration that fetched group members from GraphQL: https://gitlab.com/gitlab-org/gitlab/blob/c123c457d3aa404e0b8ca2e96607baf572ad405e/lib/bulk_imports/groups/pipelines/members_pipeline.rb#L1-L1 and for this GraphQL type I think we should be covered as the query is similar to the one for Groups.

  • Also, update developer documentation with the info about GraphQL/REST/NDJSON usage in migrating groups and projects by direct transfer.
Edited by Magdalena Frankiewicz
OSZAR »