LFS issue when using GItLab Geo and location-aware GIt URL
I will follow up on the logs. I will have to ask the customer if they would allow us to post the logs on this issue.
Summary
One of our customers is experiencing an issue when trying to push LFS file using the Location-Aware Git URL.
They are receiving the error file already closed. This doesn't happen when pushing directly to the external_url of the secondary node.
Steps to reproduce
- Configure GitLab Geo with Location-Aware Git URL: https://docs.gitlab.com/ee/administration/geo/replication/location_aware_git_url.html
- Try to push an LFS file using the location-aware git URL.
Example Project
What is the current bug behavior?
It continuously try to push the LFS file. If you look at the Git trace, it's being redirected to the external_url.
What is the expected correct behavior?
The push would be successful and automatically redirect to the Primary node.
Relevant logs and/or screenshots
Output of GIT_TRACE=1 git push when using location-aware Git URL:
11:34:51.767200 git.c:439 trace: built-in: git push
11:34:51.767492 run-command.c:663 trace: run_command: GIT_DIR=.git git-remote-https origin https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
Username for 'https://<PRIMARY_EXTERNAL_URL>': root
Password for 'https://root@<PRIMARY_EXTERNAL_URL>':
warning: redirecting to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/
11:35:00.929368 run-command.c:663 trace: run_command: .git/hooks/pre-push origin https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:00.935619 git.c:703 trace: exec: git-lfs pre-push origin https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:00.935743 run-command.c:663 trace: run_command: git-lfs pre-push origin https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:00.942448 trace git-lfs: exec: git 'version'
11:35:00.946945 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:35:00.950763 trace git-lfs: exec: git 'config' '-l'
11:35:00.952664 trace git-lfs: pre-push: refs/heads/master a842442c0286258bab0968ca701973f998763505 refs/heads/master b85a29efa4c89e357d17ddb53a116e96ea1528dc
11:35:01.850522 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/locks/verify
11:35:02.723890 trace git-lfs: HTTP: 302
11:35:02.723914 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/locks/verify to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:35:02.724418 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:35:04.370590 trace git-lfs: HTTP: 401
11:35:04.370794 trace git-lfs: setting repository access to basic
11:35:04.370946 trace git-lfs: exec: git 'config' '--replace-all' 'lfs.https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs.access' 'basic'
11:35:04.372688 trace git-lfs: setting repository access to basic
11:35:04.372710 trace git-lfs: exec: git 'config' '--replace-all' 'lfs.https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs.access' 'basic'
11:35:04.374115 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
11:35:04.374192 trace git-lfs: creds: git credential fill ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:04.375153 git.c:439 trace: built-in: git credential fill
Username for 'https://<CUSTOM_CLONE_HOST>.com': root
Password for 'https://root@<CUSTOM_CLONE_HOST>.com':
11:35:09.054081 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:09.054107 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/locks/verify
11:35:09.933074 trace git-lfs: HTTP: 302
11:35:09.933742 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/locks/verify to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:35:09.935746 trace git-lfs: creds: git credential fill ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:09.939754 git.c:439 trace: built-in: git credential fill
Username for 'https://<PRIMARY_EXTERNAL_URL>': root
Password for 'https://root@<PRIMARY_EXTERNAL_URL>':
11:35:14.292572 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:35:14.292611 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:35:16.023874 trace git-lfs: HTTP: 200
11:35:16.023924 trace git-lfs: creds: git credential approve ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:16.025387 git.c:439 trace: built-in: git credential approve
11:35:16.025661 trace git-lfs: creds: git credential approve ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:16.026991 git.c:439 trace: built-in: git credential approve
11:35:16.027407 trace git-lfs: HTTP: {"ours":[],"theirs":[]}
Locking support detected on remote "origin". Consider enabling it with:
$ git config lfs.https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs.locksverify true
11:35:16.036328 trace git-lfs: tq: running as batched queue, batch size of 100
11:35:16.036432 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
11:35:16.039533 trace git-lfs: tq: sending batch of size 1
11:35:16.039953 trace git-lfs: api: batch 1 files
11:35:16.040031 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:16.040191 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:16.040201 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:16.853579 trace git-lfs: HTTP: 302
11:35:16.853625 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:16.853998 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:16.854435 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:16.854483 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:17.606455 trace git-lfs: HTTP: 200
11:35:17.606644 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSmhOMlV3T0RNMFpDMW1aR1ptTFRSaE1qWXRPV1JsTXkweE4yTmtOV0UyTURVd1pUWWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU1UY
11:35:17.606678 trace git-lfs: HTTP: 3NJbTVpWmlJNk1UWXdOemt4TmpreE1pd2laWGh3SWpveE5qQTNPVEkwTVRFM2ZRLndmdmRSZzNVMjZQUElQY05uWUlFSUJfaHRiX2VZdlNpbEh1SnUxMHVMMFE=","Content-Type":"application/octet-stream"}}}}]}
11:35:17.606782 trace git-lfs: tq: starting transfer adapter "basic"
11:35:17.607238 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:18.276636 trace git-lfs: HTTP: 301 | 0 B/s
11:35:18.276671 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:18.276803 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:18.788104 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:18.788289 trace git-lfs: tq: enqueue retry #1 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:18.788312 trace git-lfs: tq: sending batch of size 1
11:35:18.788373 trace git-lfs: api: batch 1 files
11:35:18.788433 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:18.788560 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:18.788572 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:19.599331 trace git-lfs: HTTP: 302
11:35:19.599539 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:19.599911 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:19.600012 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:19.600076 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:20.334944 trace git-lfs: HTTP: 200
11:35:20.335869 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSTRaVE0wTXpJeE1pMHhOak5oTFRSaFkyTXRPREF5TkMwM1pXTm1NMlV6TnpOaFpXRWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU1qQ
11:35:20.336469 trace git-lfs: HTTP: XNJbTVpWmlJNk1UWXdOemt4TmpreE5Td2laWGh3SWpveE5qQTNPVEkwTVRJd2ZRLkxRQjVxeThHMm9naVd3a1hvOUJFT3lIMUlmMHJoRlM5NTFUelAxdU04c3c=","Content-Type":"application/octet-stream"}}}}]}
11:35:20.336772 trace git-lfs: tq: starting transfer adapter "basic"
11:35:20.337857 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:21.001628 trace git-lfs: HTTP: 301B | 48 KB/s
11:35:21.001673 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:21.001718 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:21.461122 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:21.461155 trace git-lfs: tq: enqueue retry #2 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:21.461173 trace git-lfs: tq: sending batch of size 1
11:35:21.461221 trace git-lfs: api: batch 1 files
11:35:21.461286 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:21.461309 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:21.461322 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:22.296359 trace git-lfs: HTTP: 302
11:35:22.296420 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:22.297162 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:22.297268 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:22.297310 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:23.120778 trace git-lfs: HTTP: 200
11:35:23.121075 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSmhOMkpqT1dRM05pMDRPRE5pTFRReVlUTXRPVEZqWkMwMk1ERXpNakZtWm1abVpURWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU1qS
11:35:23.121216 trace git-lfs: HTTP: XNJbTVpWmlJNk1UWXdOemt4TmpreE55d2laWGh3SWpveE5qQTNPVEkwTVRJeWZRLnlrNVByMFFXOXdxMVRjMG1UYTZRQjRRZC0wNVZmR1JvYVFqcDFPZ25BYU0=","Content-Type":"application/octet-stream"}}}}]}
11:35:23.121271 trace git-lfs: tq: starting transfer adapter "basic"
11:35:23.121446 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:23.767334 trace git-lfs: HTTP: 301B | 64 KB/s
11:35:23.767482 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:23.767562 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:24.221937 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:24.221956 trace git-lfs: tq: enqueue retry #3 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:24.222064 trace git-lfs: tq: sending batch of size 1
11:35:24.222309 trace git-lfs: api: batch 1 files
11:35:24.222379 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:24.222697 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:24.222709 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:25.025926 trace git-lfs: HTTP: 302
11:35:25.026263 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:25.027005 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:25.027315 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:25.027507 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:25.879659 trace git-lfs: HTTP: 200
11:35:25.879949 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSXpZbVUzWkRVMk9DMHhOR0U1TFRRNFlqTXRZV0ZpWkMxa01tVTBOakptWW1VeFlqZ2lMQ0pwWVhRaU9qRTJNRGM1TVRZNU1qV
11:35:25.880058 trace git-lfs: HTTP: XNJbTVpWmlJNk1UWXdOemt4TmpreU1Dd2laWGh3SWpveE5qQTNPVEkwTVRJMWZRLnpjV3VRU1lBdUFmeVRaYkhkcUVXS1RuUTdHeFZ0VDdKM0VvS2pfMVNYOUU=","Content-Type":"application/octet-stream"}}}}]}
11:35:25.880357 trace git-lfs: tq: starting transfer adapter "basic"
11:35:25.883077 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:26.552207 trace git-lfs: HTTP: 301B | 71 KB/s
11:35:26.552386 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:26.552544 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:27.002320 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:27.002617 trace git-lfs: tq: enqueue retry #4 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:27.002654 trace git-lfs: tq: sending batch of size 1
11:35:27.002758 trace git-lfs: api: batch 1 files
11:35:27.002875 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:27.003231 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:27.003259 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:27.836513 trace git-lfs: HTTP: 302
11:35:27.836744 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:27.837564 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:27.837585 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:27.837595 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:28.572623 trace git-lfs: HTTP: 200
11:35:28.572768 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSmlabVk1T0RJd05DMHhObVprTFRRM05tUXRPV0ZsWlMxaVlUVmtZVEJsT1dObFkyWWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU1qZ
11:35:28.572799 trace git-lfs: HTTP: 3NJbTVpWmlJNk1UWXdOemt4TmpreU15d2laWGh3SWpveE5qQTNPVEkwTVRJNGZRLjRvXzBMRXRsOEpfb05xWGVnUUdTMGtFVnpwZl9ObVB2NEdFVkViZk5aUms=","Content-Type":"application/octet-stream"}}}}]}
11:35:28.572885 trace git-lfs: tq: starting transfer adapter "basic"
11:35:28.573082 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:29.234300 trace git-lfs: HTTP: 301B | 79 KB/s
11:35:29.234483 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:29.234620 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:29.684238 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:29.684300 trace git-lfs: tq: enqueue retry #5 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:29.684330 trace git-lfs: tq: sending batch of size 1
11:35:29.684371 trace git-lfs: api: batch 1 files
11:35:29.684869 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:29.684952 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:29.684974 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:30.510962 trace git-lfs: HTTP: 302
11:35:30.510984 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:30.511240 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:30.511318 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:30.511365 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:31.247483 trace git-lfs: HTTP: 200
11:35:31.248352 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSTFNamM0TkdVeFl5MHpNbUppTFRRek56Z3RPVFF3WkMwM09XUTRZVEZoT0RVd09ETWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU16R
11:35:31.249057 trace git-lfs: HTTP: XNJbTVpWmlJNk1UWXdOemt4TmpreU5pd2laWGh3SWpveE5qQTNPVEkwTVRNeGZRLjhqQ0xaR1Y3RHRQWHJ1MEpYSFRSaWl6T3RUU2NqcUtuNWEyN1VCSk5DdW8=","Content-Type":"application/octet-stream"}}}}]}
11:35:31.249305 trace git-lfs: tq: starting transfer adapter "basic"
11:35:31.250606 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:31.910504 trace git-lfs: HTTP: 301B | 84 KB/s
11:35:31.910597 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:31.910729 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:32.367189 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:32.367517 trace git-lfs: tq: enqueue retry #6 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:32.367710 trace git-lfs: tq: sending batch of size 1
11:35:32.367919 trace git-lfs: api: batch 1 files
11:35:32.368150 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:32.368326 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:32.368459 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:33.186537 trace git-lfs: HTTP: 302
11:35:33.186778 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:33.187251 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:33.187272 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:33.187282 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:33.927891 trace git-lfs: HTTP: 200
11:35:33.928592 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSXdObVV4WkRJek1DMWpaV1kyTFRRMk56RXRZV0ZrWXkwek16ZzVaRGRpTTJSaVl6SWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU16T
11:35:33.929131 trace git-lfs: HTTP: XNJbTVpWmlJNk1UWXdOemt4TmpreU9Dd2laWGh3SWpveE5qQTNPVEkwTVRNemZRLlNIYVhzWVMxb1R6MWZ0a1NFdzlXVVhmYTRXWmx2UDVxRlZTMWVGU3JER1U=","Content-Type":"application/octet-stream"}}}}]}
11:35:33.929378 trace git-lfs: tq: starting transfer adapter "basic"
11:35:33.930306 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:34.598948 trace git-lfs: HTTP: 301B | 86 KB/s
11:35:34.598992 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:34.599049 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:35.049336 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:35.049537 trace git-lfs: tq: enqueue retry #7 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:35.049571 trace git-lfs: tq: sending batch of size 1
11:35:35.049620 trace git-lfs: api: batch 1 files
11:35:35.049701 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:35.049861 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:35.049877 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:35.864640 trace git-lfs: HTTP: 302
11:35:35.864666 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:35.864984 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:35.865005 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:35.865028 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:36.647780 trace git-lfs: HTTP: 200
11:35:36.647924 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSTNOMlU1TlRRNVppMHdaVGt6TFRRNE5qZ3RZakl3TVMwMFpUUXdaVEV6TVRBNE16Z2lMQ0pwWVhRaU9qRTJNRGM1TVRZNU16W
11:35:36.647985 trace git-lfs: HTTP: XNJbTVpWmlJNk1UWXdOemt4Tmprek1Td2laWGh3SWpveE5qQTNPVEkwTVRNMmZRLmxJSm04NFdWcDBkQm42a3ZKSlBwTHc2c09wNXh5RWpFSkY2bmhxUUJCeHc=","Content-Type":"application/octet-stream"}}}}]}
11:35:36.648190 trace git-lfs: tq: starting transfer adapter "basic"
11:35:36.649299 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:37.315964 trace git-lfs: HTTP: 301B | 87 KB/s
11:35:37.316017 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:37.316068 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:37.766972 trace git-lfs: tq: retrying object 3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
11:35:37.767002 trace git-lfs: tq: enqueue retry #8 for "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c" (size: 10467491)
11:35:37.767017 trace git-lfs: tq: sending batch of size 1
11:35:37.767091 trace git-lfs: api: batch 1 files
11:35:37.767162 trace git-lfs: creds: git credential cache ("https", "<CUSTOM_CLONE_HOST>.com", "")
11:35:37.767183 trace git-lfs: Filled credentials for https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git
11:35:37.767192 trace git-lfs: HTTP: POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch
11:35:38.585017 trace git-lfs: HTTP: 302
11:35:38.585086 trace git-lfs: api: redirect POST https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/info/lfs/objects/batch to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:38.585486 trace git-lfs: creds: git credential cache ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:35:38.585520 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:38.585536 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/objects/batch
11:35:39.319796 trace git-lfs: HTTP: 200
11:35:39.320224 trace git-lfs: HTTP: {"objects":[{"oid":"3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c","size":10467491,"actions":{"upload":{"href":"https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491","header":{"Authorization":"Basic cm9vdDpleUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKa1lYUmhJanA3SW1GamRHOXlJam9pY205dmRDSjlMQ0pxZEdraU9pSmlNamt4TUdWaU9DMWhNVEppTFRSbFlXSXRPVFpqT0MxaVlqUTBPR014Tnpka04ySWlMQ0pwWVhRaU9qRTJNRGM1TVRZNU16a
11:35:39.320496 trace git-lfs: HTTP: 3NJbTVpWmlJNk1UWXdOemt4Tmprek5Dd2laWGh3SWpveE5qQTNPVEkwTVRNNWZRLndGbTY0V0xpeE1hTjN4MVRYQzBpclFrRHp6WFlDbVBRcnlxNmdob2d3czg=","Content-Type":"application/octet-stream"}}}}]}
11:35:39.320604 trace git-lfs: tq: starting transfer adapter "basic"
11:35:39.320805 trace git-lfs: HTTP: PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491
11:35:39.985725 trace git-lfs: HTTP: 301B | 90 KB/s
11:35:39.985782 trace git-lfs: api: redirect PUT https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git/gitlab-lfs/objects/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c/10467491 to https://<SECONDARY_EXTERNAL_URL>/
11:35:39.985867 trace git-lfs: HTTP: PUT https://<SECONDARY_EXTERNAL_URL>/
11:35:40.447046 trace git-lfs: tq: refusing to retry "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c", too many retries (8)
11:35:40.447251 trace git-lfs: tq: refusing to retry "3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c", too many retries (8)
LFS: Put https://<SECONDARY_EXTERNAL_URL>/: read /home/<snipped>/Desktop/t/eu/.git/lfs/objects/3f/b7/3fb76ddccdc950088e478d91c09bf2b32227902ae5edd9da6d7b534d55adf53c: file already closed
error: failed to push some refs to 'https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.git'
<snipped>@<snipped>-vm:~/Desktop/t/eu$ https://<CUSTOM_CLONE_HOST>.com/root/lfs-test.gitOutput of GIT_TRACE=1 git push when using external_url of the Secondary node:
11:06:49.927309 git.c:439 trace: built-in: git push
11:06:49.927481 run-command.c:663 trace: run_command: GIT_DIR=.git git-remote-https origin https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git
Username for 'https://<PRIMARY_EXTERNAL_URL>': root
Password for 'https://root@<PRIMARY_EXTERNAL_URL>':
warning: redirecting to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/
11:06:57.205247 run-command.c:663 trace: run_command: .git/hooks/pre-push origin https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git
11:06:57.206928 git.c:703 trace: exec: git-lfs pre-push origin https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git
11:06:57.206971 run-command.c:663 trace: run_command: git-lfs pre-push origin https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git
11:06:57.209959 trace git-lfs: exec: git 'version'
11:06:57.212252 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:06:57.215104 trace git-lfs: exec: git 'config' '-l'
11:06:57.216915 trace git-lfs: pre-push: refs/heads/master b85a29efa4c89e357d17ddb53a116e96ea1528dc refs/heads/master 85c896de1121531da7d143876f4d369be7053f86
11:06:58.141211 trace git-lfs: HTTP: POST https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs/locks/verify
11:06:58.995888 trace git-lfs: HTTP: 302
11:06:58.996071 trace git-lfs: api: redirect POST https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs/locks/verify to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/lo
11:06:58.996496 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:07:00.643666 trace git-lfs: HTTP: 401
11:07:00.643740 trace git-lfs: setting repository access to basic
11:07:00.643765 trace git-lfs: exec: git 'config' '--replace-all' 'lfs.https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs.access' 'basic'
11:07:00.645841 trace git-lfs: setting repository access to basic
11:07:00.645865 trace git-lfs: exec: git 'config' '--replace-all' 'lfs.https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs.access' 'basic'
11:07:00.647862 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
11:07:00.647995 trace git-lfs: creds: git credential fill ("https", "<SECONDARY_EXTERNAL_URL>", "")
11:07:00.649229 git.c:439 trace: built-in: git credential fill
Username for 'https://<SECONDARY_EXTERNAL_URL>': root
Password for 'https://root@<SECONDARY_EXTERNAL_URL>':
11:07:13.895428 trace git-lfs: Filled credentials for https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git
11:07:13.895449 trace git-lfs: HTTP: POST https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs/locks/verify
11:07:14.773593 trace git-lfs: HTTP: 302
11:07:14.773669 trace git-lfs: api: redirect POST https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs/locks/verify to https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/lo
11:07:14.774014 trace git-lfs: creds: git credential fill ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:07:14.776894 git.c:439 trace: built-in: git credential fill
Username for 'https://<PRIMARY_EXTERNAL_URL>': root
Password for 'https://root@<PRIMARY_EXTERNAL_URL>':
11:07:20.639090 trace git-lfs: Filled credentials for https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:07:20.639168 trace git-lfs: HTTP: POST https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/info/lfs/locks/verify
11:07:22.383421 trace git-lfs: HTTP: 200
11:07:22.383483 trace git-lfs: creds: git credential approve ("https", "<PRIMARY_EXTERNAL_URL>", "")
11:07:22.384694 git.c:439 trace: built-in: git credential approve
11:07:22.385002 trace git-lfs: creds: git credential approve ("https", "<SECONDARY_EXTERNAL_URL>", "")
11:07:22.386099 git.c:439 trace: built-in: git credential approve
11:07:22.386386 trace git-lfs: HTTP: {"ours":[],"theirs":[]}
Locking support detected on remote "origin". Consider enabling it with:
$ git config lfs.https://<SECONDARY_EXTERNAL_URL>/root/lfs-test.git/info/lfs.locksverify true
11:07:22.386742 trace git-lfs: tq: running as batched queue, batch size of 100
11:07:22.386833 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
11:07:22.390575 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
11:07:22.390592 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
11:07:22.390660 trace git-lfs: filepathfilter: accepting "tmp"
11:07:22.392467 run-command.c:663 trace: run_command: git send-pack --stateless-rpc --helper-status --thin --progress https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/ --stdin
11:07:22.394125 git.c:439 trace: built-in: git send-pack --stateless-rpc --helper-status --thin --progress https://<PRIMARY_EXTERNAL_URL>/-/push_from_secondary/3/root/lfs-test.git/ --stdin
11:07:22.394618 run-command.c:663 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
11:07:22.395983 git.c:439 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 4, done.Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.27.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.2.1-ee Revision: 447f057d946 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https:// HTTP Clone URL: https:// SSH Clone URL: git@ Elasticsearch: no Geo: yes Geo node: Secondary Using LDAP: no Using Omniauth: yes Omniauth Providers: openid_connect, github GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) ...skipping... Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 1/2 ... yes 1/3 ... yes 1/4 ... yes 1/5 ... yes 1/6 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.27.0) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking Geo ...
GitLab Geo secondary database is correctly configured ... yes Database replication enabled? ... yes Database replication working? ... yes GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes GitLab Geo HTTP(S) connectivity ... * Can connect to the primary node ... yes GitLab Geo is available ... yes GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... yes, found a secondary node named "" HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... yes OpenSSH configured to use AuthorizedKeysCommand ... skipped Reason: Cannot access OpenSSH configuration file Try fixing it: This is expected if you are using SELinux. You may want to check configuration manually For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to disable writing to authorized_keys file ... no Try fixing it: You need to disable
Write to authorized_keys filein GitLab's Admin panel For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yesChecking Geo ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Workaround
For those on 13.6 and earlier, you could opt for a partial solution to location-aware Git URL by not customizing the HTTP remote URL as mentioned in this step https://docs.gitlab.com/ee/administration/geo/replication/location_aware_git_url.html#configure-git-clone-urls-to-use-the-special-git-url
The single Git URL would still work, but the Git-over-HTTP remote URL displayed in the UI would be the URL for the particular Geo site you are browsing.
Edited by Michael Kozono