Skip to content

Importer: Ensure specs produce no `ImportFailure`s

Problem

Last year we started to track individually failing relations by inserting them as ImportFailures instead of failing fast. Unfortunately this has led to several problems going under the radar. On master, there are several specs that currently pass, but where the imports that are being tested are actually partial failures because ImportFailures exist. Since we do not assert that there are no failures currently, these tests produce false negatives since they are green when they should be red instead.

Goal

We need to investigate and fix all existing ImportFailures in tests. Once fixed, we should then have any import spec that is a functional test case (i.e. where actual imports execute) assert that ImportFailure.all is always empty, so that we can detect regressions quickly and easily.

Approach

I suggest we add a test assertion such as expect(ImportFailure.all).to eq([]) to the following specs:

  • spec/lib/gitlab/import_export/project_tree_restorer_spec.rb
  • spec/lib/gitlab/import_export/import_export_equivalence_spec.rb

and perhaps others that I might be unaware of.

We need to investigate and fix the following errors so that we arrive at a green build with 0 import failures.

The project_tree_restorer_spec would currently fail on the following items:

+[#<ImportFailure:0x000055f16e263af8
       +  id: 1,
       +  relation_index: 0,
       +  project_id: 1,
       +  created_at: Wed, 19 Feb 2020 08:39:31 UTC +00:00,
       +  relation_key: [FILTERED],
       +  exception_class: "ActiveRecord::RecordNotUnique",
       +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
       +  exception_message:
       +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
       +   "DETAIL:  Key (project_id, source_branch, target_branch)=(1, feature, master) already exists.\n",
       +  retry_count: 0,
       +  group_id: nil,
       +  source: "process_relation_item!">]

The import_export_equivalence_spec spec would currently fail on the following items:

+[#<ImportFailure:0x000055f16cd64f80
        +  id: 18,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:06 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "Gitlab::Git::Repository::NoRepository",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "5:GetRepoPath: not a git repository '/home/git/gitlab/tmp/tests/repositories/@hashed/a2/18/a21855da08cb102d1d217c53dc5824a3a795c1c1a44e971bf01ab9da3a2acbbf.git'",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64c10
        +  id: 19,
        +  relation_index: 1,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:08 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveModel::UnknownAttributeError",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message: "unknown attribute 'diff_head_sha' for MergeRequest.",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64a80
        +  id: 20,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:08 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
        +   "DETAIL:  Key (project_id, source_branch, target_branch)=(68, feature, master) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd649b8
        +  id: 21,
        +  relation_index: 1,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:08 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordInvalid",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message: "Validation failed: Owner can't be blank",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd648f0
        +  id: 22,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordInvalid",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "Validation failed: Sentry issue   is already associated to a GitLab Issue. New issue will not be associated.",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64800
        +  id: 23,
        +  relation_index: 1,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 9) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64738
        +  id: 24,
        +  relation_index: 2,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 8) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64648
        +  id: 25,
        +  relation_index: 3,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 7) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64580
        +  id: 26,
        +  relation_index: 4,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 6) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd644b8
        +  id: 27,
        +  relation_index: 5,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 5) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd643f0
        +  id: 28,
        +  relation_index: 6,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 4) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd64328
        +  id: 29,
        +  relation_index: 7,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 3) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cd641e8
        +  id: 30,
        +  relation_index: 8,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 2) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdabf98
        +  id: 31,
        +  relation_index: 9,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_issues_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 1) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdabe58
        +  id: 32,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "Gitlab::Git::Repository::NoRepository",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "5:GetRepoPath: not a git repository '/home/git/gitlab/tmp/tests/repositories/@hashed/a2/18/a21855da08cb102d1d217c53dc5824a3a795c1c1a44e971bf01ab9da3a2acbbf.git'",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdabcf0
        +  id: 33,
        +  relation_index: 1,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 7) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdabc28
        +  id: 34,
        +  relation_index: 2,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 6) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdabae8
        +  id: 35,
        +  relation_index: 3,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 5) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdaba20
        +  id: 36,
        +  relation_index: 4,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 4) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab958
        +  id: 37,
        +  relation_index: 5,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 3) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab818
        +  id: 38,
        +  relation_index: 6,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:09 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 2) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab728
        +  id: 39,
        +  relation_index: 7,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_merge_requests_on_target_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (target_project_id, iid)=(68, 1) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab610
        +  id: 40,
        +  relation_index: 1,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_ci_pipelines_on_project_id_and_iid\"\n" +
        +   "DETAIL:  Key (project_id, iid)=(68, 1) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab548
        +  id: 41,
        +  relation_index: 3,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
        +   "DETAIL:  Key (project_id, source_branch, target_branch)=(68, feature, master) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab458
        +  id: 42,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordNotUnique",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message:
        +   "PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_external_pull_requests_on_project_and_branches\"\n" +
        +   "DETAIL:  Key (project_id, source_branch, target_branch)=(68, feature, master) already exists.\n",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab2f0
        +  id: 43,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordInvalid",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message: "Validation failed: Name has already been taken",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdab200
        +  id: 44,
        +  relation_index: 0,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordInvalid",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message: "Validation failed: Key has already been taken",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">,
        + #<ImportFailure:0x000055f16cdaaf80
        +  id: 45,
        +  relation_index: 1,
        +  project_id: 68,
        +  created_at: Wed, 19 Feb 2020 08:40:10 UTC +00:00,
        +  relation_key: [FILTERED],
        +  exception_class: "ActiveRecord::RecordInvalid",
        +  correlation_id_value: "6c09c08669890da38ceffaa948e521eb",
        +  exception_message: "Validation failed: Key has already been taken",
        +  retry_count: 0,
        +  group_id: nil,
        +  source: "process_relation_item!">]
Edited by Matthias Käppler
OSZAR »