View Issue Details

IDProjectCategoryView StatusLast Update
0015791BuildsysCi.centos.org Ecosystem Testingpublic2019-02-18 23:49
Reporterbowlofeggs 
PrioritynormalSeverityminorReproducibilityrandom
Status closedResolutionunable to reproduce 
Summary0015791: Sometimes reporting status to GitHub fails
DescriptionGreetings!

I sometimes observe pull requests on Bodhi that say they are failing certain tests, but when I look at the flow graph in Jenkins the tests actually passed, but the task that reports it to GitHub is marked as a failure. This does not happen too frequently, which is why I marked it as minor severity. My "thumb in the air" estimate is that this happens to something like 2-5% of my pull requests (i.e., I have no collected any real data, but I'd qualify it as "doesn't happen too often, but happens often enough to notice").

Here is an example:

https://github.com/fedora-infra/bodhi/pull/2975#issuecomment-461058117

If you look at the console output from that job, there are some long, backwards Java tracebacks that indicate a 502 from the GitHub server (i.e., this might just be a GitHub problem, but I wonder if Jenkins just needs to retry a few times before giving up in failure?):

java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/repos/fedora-infra/bodhi/statuses/360304ed42957ac936ae216675b5ebe87537580a
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
    at org.kohsuke.github.Requester.parse(Requester.java:607)
Caused: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/repos/fedora-infra/bodhi/statuses/360304ed42957ac936ae216675b5ebe87537580a
    at sun.reflect.GeneratedConstructorAccessor7452.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1939)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1938)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1508)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
    at org.kohsuke.github.Requester.parse(Requester.java:617)
Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: 502, message: 'Bad Gateway' for URL: https://api.github.com/repos/fedora-infra/bodhi/statuses/360304ed42957ac936ae216675b5ebe87537580a
    at org.kohsuke.github.Requester.parse(Requester.java:638)
    at org.kohsuke.github.Requester.parse(Requester.java:599)
    at org.kohsuke.github.Requester._to(Requester.java:277)
Also: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/users/fedora-infra
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
        at org.kohsuke.github.Requester.parse(Requester.java:607)
    Caused: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/users/fedora-infra
        at sun.reflect.GeneratedConstructorAccessor7452.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1939)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1938)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1508)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
        at org.kohsuke.github.Requester.parse(Requester.java:617)
    Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: 502, message: 'Bad Gateway' for URL: https://api.github.com/users/fedora-infra
        at org.kohsuke.github.Requester.parse(Requester.java:638)
        at org.kohsuke.github.Requester.parse(Requester.java:599)
        at org.kohsuke.github.Requester._to(Requester.java:277)
    Caused: org.kohsuke.github.HttpException: {
  "message": "Server Error"
}

        at org.kohsuke.github.Requester.handleApiError(Requester.java:695)
        at org.kohsuke.github.Requester._to(Requester.java:298)
        at org.kohsuke.github.Requester.to(Requester.java:239)
        at org.kohsuke.github.GitHub.getUser(GitHub.java:397)
        at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep.getRepoIfValid(GitHubStatusNotificationStep.java:269)
        at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep.access$100(GitHubStatusNotificationStep.java:79)
        at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep$Execution.run(GitHubStatusNotificationStep.java:373)
Also: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/repos/fedora-infra/bodhi/statuses/360304ed42957ac936ae216675b5ebe87537580a
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
        at org.kohsuke.github.Requester.parse(Requester.java:607)
    Caused: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/repos/fedora-infra/bodhi/statuses/360304ed42957ac936ae216675b5ebe87537580a
        at sun.reflect.GeneratedConstructorAccessor7452.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1939)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1938)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1508)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
        at org.kohsuke.github.Requester.parse(Requester.java:617)
    Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: 502, message: 'Bad Gateway' for URL: https://api.github.com/repos/fedora-infra/bodhi/statuses/360304ed42957ac936ae216675b5ebe87537580a
        at org.kohsuke.github.Requester.parse(Requester.java:638)
        at org.kohsuke.github.Requester.parse(Requester.java:599)
        at org.kohsuke.github.Requester._to(Requester.java:277)
    Also: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/repos/fedora-infra/bodhi/commits/360304ed42957ac936ae216675b5ebe87537580a
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
            at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
            at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
            at org.kohsuke.github.Requester.parse(Requester.java:607)
        Caused: java.io.IOException: Server returned HTTP response code: 502 for URL: https://api.github.com/repos/fedora-infra/bodhi/commits/360304ed42957ac936ae216675b5ebe87537580a
            at sun.reflect.GeneratedConstructorAccessor7452.newInstance(Unknown Source)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
            at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944)
            at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1939)
            at java.security.AccessController.doPrivileged(Native Method)
            at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1938)
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1508)
            at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
            at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
            at org.kohsuke.github.Requester.parse(Requester.java:617)
        Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: 502, message: 'Bad Gateway' for URL: https://api.github.com/repos/fedora-infra/bodhi/commits/360304ed42957ac936ae216675b5ebe87537580a
            at org.kohsuke.github.Requester.parse(Requester.java:638)
            at org.kohsuke.github.Requester.parse(Requester.java:599)
            at org.kohsuke.github.Requester._to(Requester.java:277)
        Caused: org.kohsuke.github.HttpException: {
  "message": "Server Error"
}

            at org.kohsuke.github.Requester.handleApiError(Requester.java:695)
            at org.kohsuke.github.Requester._to(Requester.java:298)
            at org.kohsuke.github.Requester.to(Requester.java:239)
            at org.kohsuke.github.GHRepository.getCommit(GHRepository.java:952)
            at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep$Execution.run(GitHubStatusNotificationStep.java:377)
        Caused: java.lang.IllegalArgumentException: The specified commit does not exist in the specified repository
            at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep$Execution.run(GitHubStatusNotificationStep.java:379)
    Caused: org.kohsuke.github.HttpException: {
  "message": "Server Error"
}

Caused: org.kohsuke.github.HttpException: {
  "message": "Server Error"
}

    at org.kohsuke.github.Requester.handleApiError(Requester.java:695)
    at org.kohsuke.github.Requester._to(Requester.java:298)
    at org.kohsuke.github.Requester.to(Requester.java:239)
    at org.kohsuke.github.GHRepository.createCommitStatus(GHRepository.java:1083)
    at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep$Execution.run(GitHubStatusNotificationStep.java:381)
    at org.jenkinsci.plugins.pipeline.githubstatusnotification.GitHubStatusNotificationStep$Execution.run(GitHubStatusNotificationStep.java:355)
    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
    at hudson.security.ACL.impersonate(ACL.java:290)
    at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Perhaps it is possible for me to get my Groovyscript to catch this and retry as a workaround?
TagsNo tags attached.

Activities

bowlofeggs

bowlofeggs

2019-02-08 14:39

reporter   ~0033792

Maybe some slightly useful but still kinda finger-in-the-air data about frequency: I think I witnessed this happen 3 times in the past 4*24 hours. I'm not sure how many jobs ran during that period.
bstinson

bstinson

2019-02-18 23:49

administrator   ~0033865

I'm going to close this for now. We updated the github api plugin today, hopefully that helps with the error checking.

Issue History

Date Modified Username Field Change
2019-02-06 15:24 bowlofeggs New Issue
2019-02-08 14:39 bowlofeggs Note Added: 0033792
2019-02-18 23:49 bstinson Status new => closed
2019-02-18 23:49 bstinson Resolution open => unable to reproduce
2019-02-18 23:49 bstinson Note Added: 0033865