Use pipelines to import PyPI packages

Problem to solve

There is no easy way to import packages to GitLab. For many customers, this is a blocker in adopting the GitLab package registry.

Proposal

As a Systems Administrator, I need a way to migrate my organization's Python packages to GitLab to migrate thousands of packages as efficiently as possible.

Benefits

• Save time on scripting and tooling
• Use of GitLab pipelines and templates, no need to learn a different tool.
• Get your team off Artifactory or Sonatype and save money/time.
• Import packages from any PyPI package registry to your GitLab project.

Limitations

The limitation will be directly those of child pipelines and CI/CD. See https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines and https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlab-cicd.

• The maximum gitlab-ci.yml file size is 5MB
• The MVC will likely limit the maximum number of packages imported

The only possible workaround for those limits is splitting the number of packages into chunks that fit the limits (Let's say that we can import 5000 packages max, then we could import 10 000 packages by using two pipelines).

Intended users

• Sidney (Systems Administrator)

Feature Usage Metrics

• Measure the total number of importer jobs run
• Total number of users running importer jobs

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.