View Issue Details

IDProjectCategoryView StatusLast Update
0016067Buildsysgit.centos.orgpublic2019-11-09 14:54
Reporterquanah 
PrioritynormalSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Summary0016067: With migration to new git.centos.org configuration, it is no longer possible to git fetch
DescriptionI have my own local fork of the OpenLDAP RPM repo from git.centos.org. I add a remote for the centos repository. Since the migration to the new git.centos.org, it is no longer possible to use git fetch to retrieve the updated refs etc
Steps To ReproduceAdd the git.centos.org remote:

git remote add centos-upstream https://git.centos.org/rpms/openldap.git
git fetch centos-upstream (At this point, it hangs forever)

Debugging shows:

GIT_TRACE=1 GIT_TRACE_PACK_ACCESS=1 git fetch -v --progress centos-upstream
15:50:40.254118 git.c:419 trace: built-in: git fetch -v --progress centos-upstream
15:50:40.256525 run-command.c:643 trace: run_command: GIT_DIR=.git git-remote-https centos-upstream https://git.centos.org/rpms/openldap.git
15:50:40.908100 run-command.c:643 trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin https://git.centos.org/rpms/openldap.git/
15:50:40.915182 git.c:419 trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin https://git.centos.org/rpms/openldap.git/
15:50:40.916827 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 3880
15:50:40.921629 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 1703
15:50:40.922190 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 12
15:50:40.922484 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 3193
15:50:40.922531 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 3025
15:50:40.922552 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 2519
15:50:40.922572 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 2858
15:50:40.922592 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 2687
15:50:40.922612 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 2186
15:50:40.922632 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 2016
15:50:40.922712 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 2353
15:50:40.923153 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 3360
15:50:40.923446 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 567
POST git-upload-pack (gzip 1166 to 550 bytes)
15:50:43.108756 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 885
15:50:43.109386 packfile.c:1592 .git/objects/pack/pack-005025242b48b2e8cf1e93d9215d803f12fa74ae.pack 195


(It is now 16:30, so 40 minutes later, no progress)


I can directly clone the repository w/o issue, so the problem seems specific to the fetch capability.
TagsNo tags attached.

Activities

quanah

quanah

2019-05-10 23:18

reporter   ~0034444

Just to note, it's still stuck at the same spot, all these hours later.
smooge

smooge

2019-05-14 17:35

reporter   ~0034464

What is the ip address you are coming from and what is the timezone for the logs to try and align with any logs?
quanah

quanah

2019-05-14 17:40

reporter   ~0034465

IP address should be 47.208.128.44 and the timezone in the log file above is GMT/Zulu time.

Thanks!
quanah

quanah

2019-05-14 17:41

reporter   ~0034466

Although it's possible that the IP may be different, as my ISP did reset the connection recently (I don't recall if it was before/after I ran the fetch command). If necessary, I can re-run it.
paelzer

paelzer

2019-06-03 07:40

reporter   ~0034582

Hi,
I wanted to mention that I'm affected by the same for quite a while (a few weeks maybe).
I also used to have remotes added and git fetch hang on them.

To check on the only obvious difference being no master branch I have played with -m and -t options but to no useful effect.
I have realized you can have centos remotes syncing, even multiple of them together like https://git.centos.org/rpms/qemu-kvm and https://git.centos.org/rpms/qemu-kvm-ma.git.
As long as there are no others.

It might be related to "which other" remotes are in the repository but I haven't found the detail that makes the difference.
quanah

quanah

2019-06-03 16:04

reporter   ~0034588

I only have the one remote. Still unable to fetch.
quanah

quanah

2019-06-27 15:06

reporter   ~0034748

Hello, anyone? This is a serious problem with the centos git migration. Can this please be examined and resolved? Thanks!
quanah

quanah

2019-07-22 13:41

reporter   ~0034863

Still unable to git fetch. Currently what I get is:

GIT_CURL_VERBOSE=1 GIT_TRACE=1 git fetch -vvv centos-upstream
13:39:41.902699 git.c:439 trace: built-in: git fetch -vvv centos-upstream
13:39:41.905264 run-command.c:663 trace: run_command: GIT_DIR=.git git-remote-https centos-upstream https://git.centos.org/rpms/openldap.git
* Couldn't find host git.centos.org in the .netrc file; using defaults
* Trying 8.43.84.211...
* TCP_NODELAY set
* Connected to git.centos.org (8.43.84.211) port 443 (#0)
* found 133 certificates in /etc/ssl/certs/ca-certificates.crt
* found 402 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: git.centos.org (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=North Carolina,L=Raleigh,O=Red Hat Inc.,CN=git.centos.org
* start date: Fri, 17 Nov 2017 00:00:00 GMT
* expire date: Tue, 12 Jan 2021 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA
* compression: NULL
* ALPN, server did not agree to a protocol
> GET /rpms/openldap.git/info/refs?service=git-upload-pack HTTP/1.1
Host: git.centos.org
User-Agent: git/2.22.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Mon, 22 Jul 2019 13:39:42 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
< x-xss-protection: 1; mode=block
< content-security-policy: default-src 'none'
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000
< cache-control: no-cache
< x-frame-options: DENY
< referrer-policy: no-referrer
< x-repospanner-nodename: centos01
< Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiWlRnMlltUTJaVEF5WldWbVlXRmxNalk0T1RNNU5qQXlZMlU1T1RnNU5UQXdOVE00TTJJMk53PT0ifX0.EBdOHg.QPseQ7_eQcKvBFrD9DVlpDy2bxA; Expires=
Thu, 22-Aug-2019 13:39:42 GMT; Secure; HttpOnly; Path=/
< Transfer-Encoding: chunked
< Content-Type: application/x-git-upload-pack-advertisement
<
* Connection #0 to host git.centos.org left intact
13:39:42.698249 run-command.c:663 trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin -v -v https://git.centos.org/rpms/openldap.git/
13:39:42.706014 git.c:439 trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin -v -v https://git.centos.org/rpms/openldap.git/
Server supports multi_ack_detailed
Server supports no-done
Server supports side-band-64k
Server supports allow-tip-sha1-in-want
Server supports allow-reachable-sha1-in-want
Server version is repoSpanner/0.5+10.e4df0439a0f696f71c1f2fee285dc2022d435839.el7.infra
Marking 6e577a6c4330e5f42c16d9e596c7e027db3c08a5 as complete
Marking c07fc219baac7a9b7efed657e6bbf2cb360d56e0 as complete
Marking 5f1ff0fa439f4db242330974a14c2457c01fad53 as complete
Marking 0ddf00d8ba812e0f088b44c8bdfed2ca0a2219e3 as complete
Marking 64434bd317cb08c08c47798120a1f09250f670a7 as complete
Marking 3b147ac26c8c89946c769c75944e25038025b338 as complete
Marking 066b15f7509827eca22614ca0f007109a1bcb898 as complete
Marking 465fbd8853cd05852896d1f0247cf9e271689cd0 as complete
Marking 7ea064495417c61a9e776c11a6d831dbf2394d65 as complete
Marking cb81eb15e762376df3e6224899892cbf5273109b as complete
Marking c86ef4c371a4799e9f4e1afe3da3630ed7c64b91 as complete
Marking 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17 as complete
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c4)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c5)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c5-plus)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c6)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c6-plus)
already have 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17 (refs/heads/c7)
already have 0da05883d57c32e55b7ccd04018d880ad867551b (refs/heads/c7-alt)
want 3b9fe0adfb3ecc590fe36b36e4561a708b892083 (refs/heads/c8)
have 6e577a6c4330e5f42c16d9e596c7e027db3c08a5
have c07fc219baac7a9b7efed657e6bbf2cb360d56e0
have 5f1ff0fa439f4db242330974a14c2457c01fad53
have 0ddf00d8ba812e0f088b44c8bdfed2ca0a2219e3
have 64434bd317cb08c08c47798120a1f09250f670a7
have 3b147ac26c8c89946c769c75944e25038025b338
have 066b15f7509827eca22614ca0f007109a1bcb898
have 465fbd8853cd05852896d1f0247cf9e271689cd0
have 7ea064495417c61a9e776c11a6d831dbf2394d65
have cb81eb15e762376df3e6224899892cbf5273109b
have c86ef4c371a4799e9f4e1afe3da3630ed7c64b91
have 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17
have 0da05883d57c32e55b7ccd04018d880ad867551b
have b710131f729eb3f63ea0f4631665574bd40b5612
have 17dfde8fba0b30412ef708fd5e4e3ade8d2e0c62
have dc1b8b9716fdacf5b71967857620e01e3cc0ae59
POST git-upload-pack (gzip 1166 to 550 bytes)
* Couldn't find host git.centos.org in the .netrc file; using defaults
* Found bundle for host git.centos.org: 0x556c455c7da0 [can pipeline]
* Re-using existing connection! (#0) with host git.centos.org
* Connected to git.centos.org (8.43.84.211) port 443 (#0)
> POST /rpms/openldap.git/git-upload-pack HTTP/1.1
Host: git.centos.org
User-Agent: git/2.22.0
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 550

* upload completely sent off: 550 out of 550 bytes
< HTTP/1.1 200 OK
< Date: Mon, 22 Jul 2019 13:39:42 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
< x-xss-protection: 1; mode=block
< content-security-policy: default-src 'none'
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000
< cache-control: no-cache
< x-frame-options: DENY
< referrer-policy: no-referrer
< x-repospanner-nodename: centos01
< Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiT0dZME5qTm1NakptTVdaaU5HVTRZalZoTUdObE5EbGlPV0V4TVRKaE9XSXhOMlV3TTJZMVlnPT0ifX0.EBdOIA.0LpNZmDV-PVyny0D6ip_zTgERIs; Expires=Thu, 22-Aug-2019 13:39:44 GMT; Secure; HttpOnly; Path=/
< Content-Length: 285
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host git.centos.org left intact
got ack 3 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17
got ack 3 0da05883d57c32e55b7ccd04018d880ad867551b
got ack 4 b710131f729eb3f63ea0f4631665574bd40b5612
got ack 4 17dfde8fba0b30412ef708fd5e4e3ade8d2e0c62
got ack 4 dc1b8b9716fdacf5b71967857620e01e3cc0ae59
done


Then nothing further occurs. This makes it virtually impossible to maintain my forked repository. When will there be an update on this, it's been over 2 months since CentOS broke this.
halovan

halovan

2019-08-12 19:48

reporter   ~0034968

I wanted to mention that I, too, seem to be affected by the same issue for a few months now. We track multiple repositories, and about two months ago, saw the same issue.

Our work around has been to remove the repo locally and redo the clone and add locally tracked repo manually in order to perform what used to be a simple fetch. Getting this fixed would allow us to get back to a normal workflow.
quanah

quanah

2019-10-02 20:51

reporter   ~0035299

Hello,

It's been nearly 5 months since this report about the fact that the CentOS git server is fundamentally broken. When will it be fixed?
arrfab

arrfab

2019-10-02 21:12

administrator   ~0035300

Have you already reported that upstream (so on repospanner) ?
I see those two bugs :
- https://github.com/repoSpanner/repoSpanner/issues/69 (pointing back to here)
- https://github.com/repoSpanner/repoSpanner/issues/79
quanah

quanah

2019-10-02 21:50

reporter   ~0035301

@arrfab No, I was originally pointed at filing a bug here. It was not mentioned that one would need to file additional bugs elsewhere.
arrfab

arrfab

2019-10-11 11:58

administrator   ~0035439

Last edited: 2019-10-11 12:00

View 2 revisions

@quanah : When I discussed with a repospanner developer, we wanted to test one scenario, but first wanted to acknowledge this issue and so be able to reproduce it , but I can't : it works on my side.
Here is how I tried to reproduce your issue with openldap :

[arrfab@git ]$ time git clone https://src.fedoraproject.org/rpms/openldap.git
Cloning into 'openldap'...
remote: Counting objects: 2384, done.
remote: Compressing objects: 100% (1730/1730), done.
remote: Total 2384 (delta 1460), reused 997 (delta 620)
Receiving objects: 100% (2384/2384), 1.72 MiB | 555.00 KiB/s, done.
Resolving deltas: 100% (1460/1460), done.

real 0m3.591s
user 0m0.226s
sys 0m0.114s
[arrfab@git]$ cd openldap/
[arrfab@git openldap]$ git remote add centos-upstream https://git.centos.org/rpms/openldap.git
[arrfab@git openldap]$ time git fetch centos-upstream
remote: Welcome to repoSpanner 0.5+11.c6a026e428baaa9b34c8fd54a6f1c0a6f2ae062e.el7.infra, node centos01.rpms.fedoraproject.org
remote: Building packfile
remote: Packfile built, sending
remote: Packfile sent
Receiving objects: 100% (182/182), 453.01 KiB | 0 bytes/s, done.
From https://git.centos.org/rpms/openldap
 * [new branch] c5-plus -> centos-upstream/c5-plus
 * [new branch] c7 -> centos-upstream/c7
 * [new branch] c7-alt -> centos-upstream/c7-alt
 * [new branch] c8-beta -> centos-upstream/c8-beta
 * [new branch] c4 -> centos-upstream/c4
 * [new branch] c6 -> centos-upstream/c6
 * [new branch] c6-plus -> centos-upstream/c6-plus
 * [new branch] c5 -> centos-upstream/c5
 * [new branch] c8 -> centos-upstream/c8

real 1m2.207s
user 0m0.094s
sys 0m0.100s

So I just added fedora as origin and then added centos as other remote.
I wanted to test a theory about switching in repospanner default branch to c7 (as yes, there is still no master branch, as long as Fedora will not have migrated their infra to repospanner cluster), but it works without.
I'll still do the default branch change in repospanner to see if that makes any diff, but it works on my side without.

smooge

smooge

2019-10-11 12:19

reporter   ~0035440

Could you add the command line steps you do to set up this problem.
1. You had a copy of openldap already.. how did you create it
2. You added the remote repository.. how did you do that (as the options may lead us to seeing this problem)
3. You run git fetch .. do you have anything in your .git/config we need also.
quanah

quanah

2019-10-11 15:50

reporter   ~0035445

@smooge - My copy of openldap came from the original (pre-migration) CentOS Git repo which I then pushed to our local gitlab instance

I added the remote repository using the command line. How I did this is literally in the first comment of this bug:

git remote add centos-upstream https://git.centos.org/rpms/openldap.git

There's nothing special in my .git/config:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = git@gitlab.symas.net:Symas/rheldap.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
[remote "centos-upstream"]
        url = https://git.centos.org/rpms/openldap.git
        fetch = +refs/heads/*:refs/remotes/centos-upstream/*


As noted, this issue affects multiple other individuals, not just myself.

Additionally, I have no issues with the Fedora repo (I just added it as an upstream, and git fetch worked correctly). This issue is purely linked to the CentOS git repo:

quanah@ub18:~/git/symas/rheldap$ git remote add fedora-upstream https://src.fedoraproject.org/rpms/openldap.git
quanah@ub18:~/git/symas/rheldap$ git fetch fedora-upstream
warning: no common commits
remote: Counting objects: 2384, done.
remote: Compressing objects: 100% (1730/1730), done.
remote: Total 2384 (delta 1461), reused 997 (delta 620)
Receiving objects: 100% (2384/2384), 1.71 MiB | 3.61 MiB/s, done.
Resolving deltas: 100% (1461/1461), done.
From https://src.fedoraproject.org/rpms/openldap
 * [new branch] FC-4-split -> fedora-upstream/FC-4-split
 * [new branch] f10 -> fedora-upstream/f10
 * [new branch] f11 -> fedora-upstream/f11

....

git fetch with CentOS still hangs indefinitely:

quanah@ub18:~/git/symas/rheldap$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git fetch -vvv centos-upstream
15:47:50.020007 git.c:440 trace: built-in: git fetch -vvv centos-upstream
15:47:50.021946 run-command.c:663 trace: run_command: GIT_DIR=.git git-remote-https centos-upstream https://git.centos.org/rpms/openldap.git
* Couldn't find host git.centos.org in the .netrc file; using defaults
* Trying 8.43.84.211...
* TCP_NODELAY set
* Connected to git.centos.org (8.43.84.211) port 443 (#0)
* found 133 certificates in /etc/ssl/certs/ca-certificates.crt
* found 402 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: git.centos.org (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=North Carolina,L=Raleigh,O=Red Hat Inc.,CN=git.centos.org
* start date: Fri, 17 Nov 2017 00:00:00 GMT
* expire date: Tue, 12 Jan 2021 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA
* compression: NULL
* ALPN, server did not agree to a protocol
> GET /rpms/openldap.git/info/refs?service=git-upload-pack HTTP/1.1
Host: git.centos.org
User-Agent: git/2.23.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Fri, 11 Oct 2019 15:47:50 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
< x-xss-protection: 1; mode=block
< Content-Security-Policy: default-src 'self';script-src 'self' 'nonce-YIRP2bZOBs8kgXnOTnquf4zqF'; style-src 'self' 'nonce-YIRP2bZOBs8kgXnOTnquf4zqF'; object-src 'none';base
-uri 'self';img-src 'self' https:;connect-src 'self';frame-src ;frame-ancestors ;
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000
< cache-control: no-cache
< x-frame-options: DENY
< referrer-policy: no-referrer
< x-repospanner-nodename: centos01
< Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiTmpVMllXUTFNV1F6WWpCak1ETTRNbVF6T1RoaE9HWXdNVFZqWVRsa1l6Qm1ZalkzTm1Rd05nPT0ifX0.EII1pg.eUC_gt9c4x0TXpABOZZ2B96D6Zw; Expires=Mon, 11-Nov-2019 15:47:50 GMT; Secure; HttpOnly; Path=/
< Transfer-Encoding: chunked
< Content-Type: application/x-git-upload-pack-advertisement
<
* Connection #0 to host git.centos.org left intact
15:47:50.802271 run-command.c:663 trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin -v -v https://git.centos.org/rpms/openldap.git/
15:47:50.810407 git.c:440 trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin -v -v https://git.centos.org/rpms/openldap.git/
Server version is repoSpanner/0.5+11.c6a026e428baaa9b34c8fd54a6f1c0a6f2ae062e.el7.infra
Server supports multi_ack_detailed
Server supports no-done
Server supports side-band-64k
Server supports allow-tip-sha1-in-want
Server supports allow-reachable-sha1-in-want
Marking 3eaf4e4abf3727053824a33a3b090219fd2b5ce7 as complete
Marking 7dc5c202a6ed815c8d67b0ea0a034f904df900af as complete
Marking 4b5ed195dacd43b079a051c7375db728872c49fd as complete
Marking f088a3b84f8b6650f0c4d30c7a3b7103d446b690 as complete
Marking 5836b1e6d6c99c92b67c37a3f90ddaa9095e7838 as complete
Marking 08ec53a757f64db10e7e3337c72f70fe1e7b0a96 as complete
Marking 13d36a07c5b882e1f00c0dbd5eacc17410ef2c75 as complete
Marking ff124be61b841b6f1f0b90ea67d1493cc0f416a8 as complete
Marking 166d0acab24fbc07397e7817dab64af6d1f49987 as complete
Marking 40576d2e82549b8bcf61c226b89f9bf82d2e88fd as complete
Marking aaf56dfd5b991dec6639c2beb6abd83243a5a031 as complete
Marking f1c4207742b7ff5cc1275a782c039ae6485b3d72 as complete
Marking 1fb1d20a05d9e0ea377ee508af884801761ed72d as complete
Marking 6e577a6c4330e5f42c16d9e596c7e027db3c08a5 as complete
Marking c07fc219baac7a9b7efed657e6bbf2cb360d56e0 as complete
Marking 5f1ff0fa439f4db242330974a14c2457c01fad53 as complete
Marking 0ddf00d8ba812e0f088b44c8bdfed2ca0a2219e3 as complete
Marking 64434bd317cb08c08c47798120a1f09250f670a7 as complete
Marking 3b147ac26c8c89946c769c75944e25038025b338 as complete
Marking 066b15f7509827eca22614ca0f007109a1bcb898 as complete
Marking 465fbd8853cd05852896d1f0247cf9e271689cd0 as complete
Marking 7ea064495417c61a9e776c11a6d831dbf2394d65 as complete
Marking cb81eb15e762376df3e6224899892cbf5273109b as complete
Marking c86ef4c371a4799e9f4e1afe3da3630ed7c64b91 as complete
Marking d91e3752b493c28137d6fb5e5d4113a45f8f6dd9 as complete
Marking f5aae857a3e61a6391a10181a8dbf14f5cc57b10 as complete
Marking 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17 as complete
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c4)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c5)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c5-plus)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c6)
want a37f561028822f3373dd2ebfe07ad1fbb6bff0a4 (refs/heads/c6-plus)
already have 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17 (refs/heads/c7)
already have 0da05883d57c32e55b7ccd04018d880ad867551b (refs/heads/c7-alt)
want 3b9fe0adfb3ecc590fe36b36e4561a708b892083 (refs/heads/c8)
want 57672de28558c2cea2230362e345b0a7c9263d3f (refs/heads/c8-beta)
have 3eaf4e4abf3727053824a33a3b090219fd2b5ce7
have 7dc5c202a6ed815c8d67b0ea0a034f904df900af
have 4b5ed195dacd43b079a051c7375db728872c49fd
have f088a3b84f8b6650f0c4d30c7a3b7103d446b690
have 5836b1e6d6c99c92b67c37a3f90ddaa9095e7838
have 08ec53a757f64db10e7e3337c72f70fe1e7b0a96
have 13d36a07c5b882e1f00c0dbd5eacc17410ef2c75
have ff124be61b841b6f1f0b90ea67d1493cc0f416a8
have 166d0acab24fbc07397e7817dab64af6d1f49987
have 40576d2e82549b8bcf61c226b89f9bf82d2e88fd
have aaf56dfd5b991dec6639c2beb6abd83243a5a031
have f1c4207742b7ff5cc1275a782c039ae6485b3d72
have 1fb1d20a05d9e0ea377ee508af884801761ed72d
have 6e577a6c4330e5f42c16d9e596c7e027db3c08a5
have c07fc219baac7a9b7efed657e6bbf2cb360d56e0
have 5f1ff0fa439f4db242330974a14c2457c01fad53
POST git-upload-pack (gzip 1216 to 572 bytes)
* Couldn't find host git.centos.org in the .netrc file; using defaults
* Found bundle for host git.centos.org: 0x562d9cdb7510 [can pipeline]
* Re-using existing connection! (#0) with host git.centos.org
* Connected to git.centos.org (8.43.84.211) port 443 (#0)
> POST /rpms/openldap.git/git-upload-pack HTTP/1.1
Host: git.centos.org
User-Agent: git/2.23.0
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 572

* upload completely sent off: 572 out of 572 bytes
< HTTP/1.1 200 OK
< Date: Fri, 11 Oct 2019 15:47:50 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
< x-xss-protection: 1; mode=block
< Content-Security-Policy: default-src 'self';script-src 'self' 'nonce-5q0Xo04RWjA2AYexD2Tu36f1J'; style-src 'self' 'nonce-5q0Xo04RWjA2AYexD2Tu36f1J'; object-src 'none';base-uri 'self';img-src 'self' https:;connect-src 'self';frame-src ;frame-ancestors ;
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000
< cache-control: no-cache
< x-frame-options: DENY
< referrer-policy: no-referrer
< x-repospanner-nodename: centos01
< Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiTkRJMVpHRTNNekZrTVRreE5tSTNaV1ZpWkdWaU0yRmhOVGhsTVRRd05qWXdNVFJtTlRNME9BPT0ifX0.EII1qA.L78R2pbgR6W9oCXfBhT8Xjq_iuc; Expires=Mon, 11-Nov-2019 15:47:52 GMT; Secure; HttpOnly; Path=/
< Content-Length: 8
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host git.centos.org left intact
have 0ddf00d8ba812e0f088b44c8bdfed2ca0a2219e3
have 64434bd317cb08c08c47798120a1f09250f670a7
have 3b147ac26c8c89946c769c75944e25038025b338
have 066b15f7509827eca22614ca0f007109a1bcb898
have 465fbd8853cd05852896d1f0247cf9e271689cd0
have 7ea064495417c61a9e776c11a6d831dbf2394d65
have cb81eb15e762376df3e6224899892cbf5273109b
have c86ef4c371a4799e9f4e1afe3da3630ed7c64b91
have d91e3752b493c28137d6fb5e5d4113a45f8f6dd9
have f5aae857a3e61a6391a10181a8dbf14f5cc57b10
have 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17
have 0da05883d57c32e55b7ccd04018d880ad867551b
have b710131f729eb3f63ea0f4631665574bd40b5612
have f6fb5e8a930930b0a3c2c04bdc4678eaa02acdcd
have 8bd8644add18a25e0f761cf0eb0d23082d9d4dee
have 3b59a4668dc01d8c6d97b16ab51dd42a97553c83
POST git-upload-pack (gzip 1216 to 573 bytes)
* Couldn't find host git.centos.org in the .netrc file; using defaults
* Found bundle for host git.centos.org: 0x562d9cdb7510 [can pipeline]
* Re-using existing connection! (#0) with host git.centos.org
* Connected to git.centos.org (8.43.84.211) port 443 (#0)
> POST /rpms/openldap.git/git-upload-pack HTTP/1.1
Host: git.centos.org
User-Agent: git/2.23.0
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 573

* upload completely sent off: 573 out of 573 bytes
< HTTP/1.1 200 OK
< Date: Fri, 11 Oct 2019 15:47:52 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
< x-xss-protection: 1; mode=block
< Content-Security-Policy: default-src 'self';script-src 'self' 'nonce-QSKy43m2lRgCDrhPYFERy9I4U'; style-src 'self' 'nonce-QSKy43m2lRgCDrhPYFERy9I4U'; object-src 'none';base-uri 'self';img-src 'self' https:;connect-src 'self';frame-src ;frame-ancestors ;
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000
< cache-control: no-cache
< x-frame-options: DENY
< referrer-policy: no-referrer
< x-repospanner-nodename: centos01
< Set-Cookie: pagure=eyJfcGVybWFuZW50Ijp0cnVlLCJjc3JmIjp7IiBiIjoiWm1aak5XTXdPVFF5WXpNNE1EZGpaakF6TlRBd056Um1NemxqWlRZeE0yTm1OalZsTlRVNE1nPT0ifX0.EII1qg.xyLChNTSsPX-KxxQCU4g4f8-QRc; Expires=Mon, 11-Nov-2019 15:47:54 GMT; Secure; HttpOnly; Path=/
< Content-Length: 340
< Content-Type: text/plain; charset=utf-8
<
got ack 3 17ff3c55e3176ad9e96c37bcc7871f88d0ad7e17* Connection #0 to host git.centos.org left intact

got ack 3 0da05883d57c32e55b7ccd04018d880ad867551b
got ack 4 b710131f729eb3f63ea0f4631665574bd40b5612
got ack 4 f6fb5e8a930930b0a3c2c04bdc4678eaa02acdcd
got ack 4 8bd8644add18a25e0f761cf0eb0d23082d9d4dee
got ack 4 3b59a4668dc01d8c6d97b16ab51dd42a97553c83
done



After this point, it just hangs forever.
quanah

quanah

2019-10-14 14:50

reporter   ~0035462

Just to note, it's still stuck at the same spot > 48 hours later.
arrfab

arrfab

2019-10-14 15:27

administrator   ~0035463

it worked on our side so maybe something differerent : tested from a stock centos 7 node, so using git 1.8.3.1
But it seems you use newer git (User-Agent: git/2.23.0) .. @rbarlow : do you think it can be the issue ?
I have quite some other tasks to work on, but I'll try to find free time to replicate on a different git version (or better : can you do and try to reproduce with a version like we have on centos 7, to confirm/deny that it can be the issue)
quanah

quanah

2019-10-14 16:11

reporter   ~0035464

Note that I cannot reproduce if I start with a copy of the repo from the new git server, it's only broken with my repo having originated from the original centos git server. I may just have to abandon my current repo and start over.
bowlofeggs

bowlofeggs

2019-10-14 16:43

reporter   ~0035465

@arrfab I am honestly not familiar enough with the git protocol to comment about whether the client version is significant here.
halovan

halovan

2019-10-15 14:03

reporter   ~0035484

I was attempting to fetch updates on httpd, and it's been stuck at the same spot for the past 4 days. User-Agent: git/1.8.3.1

Our issues are with repos from both the original server and the new server. We track over a dozen repos, and the issue seems to randomly re-appear on one or two at a time. Comparing the git logs from the repo that hangs to a fresh checkout of the same repo, it seems the only differences are updates to non-c7 branches.

I tried to fetch individual branches in the same repo and found some return as expected and others will hang. For instance:
 * git fetch centos-upstream --> This is what we normally run and hangs forever
 * git fetch centos-upstream c7 --> This seems to work correctly
 * git fetch centos-upstream c8-stream-2.4 --> Hangs just like the original command

For the systemd repo, we see the same results at above, but 'git fetch centos-upstream c7-sig-altarch-fasttrack' causes the hang.

This makes me wonder -- is it possible that the issue is not due to the new server, but with the new branches added to the repos? I can't be certain, but I started noticing these issues around the time the master branch went away and the new c8 branches were introduced. Also, @quanah posted the issue on May 10, just 3 days after the c8 branch was committed. It could be related.

@quanah If you fetch just the branches you are working with, does it work? This is going to be our work around until the devs fix the issue.
quanah

quanah

2019-10-15 17:50

reporter   ~0035489

@halovan - Your right, I can git fetch a specific (old) branch:

quanah@ub18:~/git/symas/rheldap$ git fetch centos-upstream c7
From https://git.centos.org/rpms/openldap
 * branch c7 -> FETCH_HEAD
 * [new branch] c7 -> centos-upstream/c7

But fetching new branches is what hangs:

quanah@ub18:~/git/symas/rheldap$ git fetch centos-upstream c8

<hang>
quanah

quanah

2019-11-06 17:54

reporter   ~0035650

Hm...

 git clone https://git.centos.org/rpms/openldap.git
Cloning into 'openldap'...
remote: Welcome to repoSpanner 0.5+11.c6a026e428baaa9b34c8fd54a6f1c0a6f2ae062e.el7.infra, node centos01.rpms.fedoraproject.org
remote: Building packfile
remote: Packfile built, sending
remote: Packfile sent
Receiving objects: 100% (207/207), 481.14 KiB | 1.03 MiB/s, done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.
bowlofeggs

bowlofeggs

2019-11-06 17:55

reporter   ~0035651

Hi quanah,

It did clone correctly, but simply failed to check out a branch (because there is no master, and no otherwise default branch). You should be able to check out a branch by using git checkout <branch_name>, however.
quanah

quanah

2019-11-06 20:42

reporter   ~0035652

@bowlofeggs I understand it cloned correctly. That's not the point. The point is that I still cannot pull any new branches since master got deleted, and I'm wondering (as others have speculated) that the deletion of the master branch is what broke things, since it's only branches added SINCE then that cannot get used.
bowlofeggs

bowlofeggs

2019-11-06 21:49

reporter   ~0035653

Hi quanah,

I am able to checkout the branches. For example:

$ git clone https://git.centos.org/rpms/openldap.git
Cloning into 'openldap'...
remote: Welcome to repoSpanner 0.5+11.c6a026e428baaa9b34c8fd54a6f1c0a6f2ae062e.el7.infra, node centos01.rpms.fedoraproject.org
remote: Building packfile
remote: Packfile built, sending
remote: Packfile sent
Receiving objects: 100% (207/207), 481.14 KiB | 2.56 MiB/s, done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git branch -r
  origin/c4
  origin/c5
  origin/c5-plus
  origin/c6
  origin/c6-plus
  origin/c7
  origin/c7-alt
  origin/c7-beta
  origin/c8
  origin/c8-beta
$ git checkout c8
Branch 'c8' set up to track remote branch 'c8' from 'origin'.
Switched to a new branch 'c8'

Or do you mean something else?
quanah

quanah

2019-11-06 22:25

reporter   ~0035654

@bowlofeggs -- Again, this has nothing to do with checking out the branches. I suggest you go read back from the start.
halovan

halovan

2019-11-09 14:54

reporter   ~0035666

@quanah I had wondered if the missing master branch was part of the issue. It disappeared at the same time the issues started happening.

Since giving you the tip about fetching specific old branches, that command has stopped working on one of the repos we're tracking. I'm typically able to re-clone the repo and re-fetch the old branch, but at this point, it would probably be easier if they were to fix the issue rather than having to keep concocting workarounds.

@bowlofeggs It's been an intermittent issue that's been difficult to replicate, but my guess is, if you clone openldap and checkout the c7 branch, then wait for an update to the c8 branch, once the c8 branch has an update pushed to it, you'll no longer be able to fetch from the c7 branch. I don't think it's quite that straightforward, but there is a correlation to fetches breaking and updates being pushed to other branches.

Issue History

Date Modified Username Field Change
2019-05-10 16:30 quanah New Issue
2019-05-10 23:18 quanah Note Added: 0034444
2019-05-14 17:35 smooge Note Added: 0034464
2019-05-14 17:40 quanah Note Added: 0034465
2019-05-14 17:41 quanah Note Added: 0034466
2019-06-03 07:40 paelzer Note Added: 0034582
2019-06-03 16:04 quanah Note Added: 0034588
2019-06-27 15:06 quanah Note Added: 0034748
2019-07-22 13:41 quanah Note Added: 0034863
2019-08-12 19:48 halovan Note Added: 0034968
2019-10-02 20:51 quanah Note Added: 0035299
2019-10-02 21:12 arrfab Note Added: 0035300
2019-10-02 21:50 quanah Note Added: 0035301
2019-10-11 11:58 arrfab Note Added: 0035439
2019-10-11 11:59 arrfab Status new => feedback
2019-10-11 12:00 arrfab Note Edited: 0035439 View Revisions
2019-10-11 12:19 smooge Note Added: 0035440
2019-10-11 15:50 quanah Note Added: 0035445
2019-10-11 15:50 quanah Status feedback => assigned
2019-10-14 14:50 quanah Note Added: 0035462
2019-10-14 15:27 arrfab Note Added: 0035463
2019-10-14 16:11 quanah Note Added: 0035464
2019-10-14 16:43 bowlofeggs Note Added: 0035465
2019-10-15 14:03 halovan Note Added: 0035484
2019-10-15 17:50 quanah Note Added: 0035489
2019-11-06 17:54 quanah Note Added: 0035650
2019-11-06 17:55 bowlofeggs Note Added: 0035651
2019-11-06 20:42 quanah Note Added: 0035652
2019-11-06 21:49 bowlofeggs Note Added: 0035653
2019-11-06 22:25 quanah Note Added: 0035654
2019-11-09 14:54 halovan Note Added: 0035666