(Catching up on a backlog of emails, hence my belated reply) I just checked the spark-prs app engine logs and it appears that our JIRA API calls are failing due to a CAPTCHA check (same issue as before). Based on https://jira.atlassian.com/browse/JRASERVER-40362, it sounds like this is a fairly common problem.
I manually logged into the 'apachespark' JIRA account and completed the CPATCHA, so *hopefully* things should be temporarily unbroken. To permanently fix this issue, we might need to use OAuth tokens for connecting to JIRA (instead of basic username + password auth). It looks like the Python JIRA library supports this ( https://jira.readthedocs.io/en/master/examples.html#oauth) and I found some promising-looking instructions on how to generate the OAuth tokens: https://www.redradishtech.com/display/KB/How+to+write+a+Python+script+authenticating+with+Jira+via+OAuth . However, it looks like you need to be a JIRA administrator in order to configure the applink so I can't fix this by myself. It would be great if the native GitHub <-> JIRA integration could meet our needs; this probably either didn't exist or wasn't configurable by us when we first wrote our own integration / sync script. On Wed, Apr 29, 2020 at 6:21 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: > Let actually me just take a look by myself and bring some updates soon. > > 2020년 4월 30일 (목) 오전 9:13, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > >> WDYT @Josh Rosen <joshro...@databricks.com>? >> Seems >> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L131-L142 >> this >> isn't working anymore. >> Does it make sense to move it to native Jira-GitHub integration >> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html> >> ? >> It won't change JIRA status as we used to do but it might be better from >> a cursory look. However, maybe I missed some context. >> >> >> 2020년 4월 30일 (목) 오전 2:46, Nicholas Chammas <nicholas.cham...@gmail.com>님이 >> 작성: >> >>> Not sure what you mean. The native integration will auto-link from a >>> Jira ticket to the PRs that mention that ticket. I don't think it will >>> update the ticket's status, though. >>> >>> Would you like me to file a ticket with Infra and see what they say? >>> >>> On Tue, Apr 28, 2020 at 12:21 AM Hyukjin Kwon <gurwls...@gmail.com> >>> wrote: >>> >>>> Maybe it's time to switch. Do you know if we can still link the JIRA >>>> against Github? >>>> The script used to change the status of JIRA too but it stopped working >>>> for a long time - I suspect this isn't a big deal. >>>> >>>> 2020년 4월 25일 (토) 오전 10:31, Nicholas Chammas <nicholas.cham...@gmail.com>님이 >>>> 작성: >>>> >>>>> Have we asked Infra recently about enabling the native Jira-GitHub >>>>> integration >>>>> <https://confluence.atlassian.com/adminjiracloud/connect-jira-cloud-to-github-814188429.html>? >>>>> Maybe we can deprecate the part of this script that updates Jira tickets >>>>> with links to the PR and rely on the native integration instead. We use it >>>>> at my day job, for example. >>>>> >>>>> On Fri, Apr 24, 2020 at 12:39 AM Hyukjin Kwon <gurwls...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Seems like this github_jira_sync.py >>>>>> <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> >>>>>> script >>>>>> seems stopped working completely now. >>>>>> >>>>>> https://issues.apache.org/jira/browse/SPARK-31532 <> >>>>>> https://github.com/apache/spark/pull/28316 >>>>>> https://issues.apache.org/jira/browse/SPARK-31529 <> >>>>>> https://github.com/apache/spark/pull/28315 >>>>>> https://issues.apache.org/jira/browse/SPARK-31528 <> >>>>>> https://github.com/apache/spark/pull/28313 >>>>>> >>>>>> Josh, would you mind taking a look please when you find some time? >>>>>> There is a bunch of JIRAs now, and it is very confusing which JIRA is >>>>>> in progress with a PR or not. >>>>>> >>>>>> >>>>>> 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: >>>>>> >>>>>>> Just FYI, I had to come up with a better JQL to filter out the JIRAs >>>>>>> that already have linked PRs. >>>>>>> In case it helps someone, I use this JQL now to look through the >>>>>>> open JIRAs: >>>>>>> >>>>>>> project = SPARK AND >>>>>>> status = Open AND >>>>>>> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *") >>>>>>> ORDER BY created DESC, priority DESC, updated DESC >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: >>>>>>> >>>>>>>> That's a great explanation. Thanks I didn't know that. >>>>>>>> >>>>>>>> Josh, do you know who I should ping on this? >>>>>>>> >>>>>>>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <dongjoon.h...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, Hyukjin. >>>>>>>>> >>>>>>>>> In short, there are two bots. And, the current situation happens >>>>>>>>> when only one bot with `dev/github_jira_sync.py` works. >>>>>>>>> >>>>>>>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status >>>>>>>>> change because it only use `add_remote_link` and `add_comment` API. >>>>>>>>> I know only this bot (in Apache Spark repository repo) >>>>>>>>> >>>>>>>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID >>>>>>>>> `githubbot` (Name: `ASF GitHub Bot`). >>>>>>>>> And, the other bot's activity is done under JIRA ID `apachespark` >>>>>>>>> (Name: `Apache Spark`). >>>>>>>>> The other bot is the one which Josh mentioned before. (in >>>>>>>>> `databricks/spark-pr-dashboard` repo). >>>>>>>>> >>>>>>>>> The root cause will be the same. The API key used by the bot is >>>>>>>>> rejected by Apache JIRA and forwarded to CAPCHAR. >>>>>>>>> >>>>>>>>> Bests, >>>>>>>>> Dongjoon. >>>>>>>>> >>>>>>>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gurwls...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> Seems this issue is re-happening again. Seems the PR link is >>>>>>>>>> properly created in the corresponding JIRA but it doesn't change the >>>>>>>>>> JIRA's >>>>>>>>>> status from OPEN to IN-PROGRESS. >>>>>>>>>> >>>>>>>>>> See, for instance, >>>>>>>>>> >>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28443 >>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28440 >>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28436 >>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28434 >>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28433 >>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-28431 >>>>>>>>>> >>>>>>>>>> Josh and Dongjoon, do you guys maybe have any idea? >>>>>>>>>> >>>>>>>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gurwls...@gmail.com>님이 >>>>>>>>>> 작성: >>>>>>>>>> >>>>>>>>>>> Thank you so much Josh .. !! >>>>>>>>>>> >>>>>>>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <rosenvi...@gmail.com>님이 >>>>>>>>>>> 작성: >>>>>>>>>>> >>>>>>>>>>>> The code for this runs in http://spark-prs.appspot.com (see >>>>>>>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137 >>>>>>>>>>>> ) >>>>>>>>>>>> >>>>>>>>>>>> I checked the AppEngine logs and it looks like we're getting >>>>>>>>>>>> error responses, possibly due to a credentials issue: >>>>>>>>>>>> >>>>>>>>>>>> Exception when starting progress on JIRA issue SPARK-27355 ( >>>>>>>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>) >>>>>>>>>>>>> Traceback (most recent call last): File >>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py", >>>>>>>>>>>>> line 138 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in update_pr start_issue_progress("%s-%s" % >>>>>>>>>>>>> (app.config['JIRA_PROJECT'], >>>>>>>>>>>>> issue_number)) File >>>>>>>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'], >>>>>>>>>>>>> issue_number)) >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py", >>>>>>>>>>>>> line 27 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in start_issue_progress jira_client = get_jira_client() File >>>>>>>>>>>>> jira_client = get_jira_client() >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py", >>>>>>>>>>>>> line 18 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File >>>>>>>>>>>>> app.config['JIRA_PASSWORD'])) >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py", >>>>>>>>>>>>> line 472 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in __init__ si = self.server_info() File >>>>>>>>>>>>> si = self.server_info() >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py", >>>>>>>>>>>>> line 2133 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in server_info j = self._get_json('serverInfo') File >>>>>>>>>>>>> j = self._get_json('serverInfo') >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py", >>>>>>>>>>>>> line 2549 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in _get_json r = self._session.get(url, params=params) File >>>>>>>>>>>>> r = self._session.get(url, params=params) >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py", >>>>>>>>>>>>> line 151 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in get return self.__verb('GET', url, **kwargs) File >>>>>>>>>>>>> return self.__verb('GET', url, **kwargs) >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py", >>>>>>>>>>>>> line 147 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File >>>>>>>>>>>>> raise_on_error(response, verb=verb, **kwargs) >>>>>>>>>>>>> File >>>>>>>>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py", >>>>>>>>>>>>> line 57 >>>>>>>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>>>>>>>> in raise_on_error r.status_code, error, r.url, request=request, >>>>>>>>>>>>> response=r, >>>>>>>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url: >>>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text: >>>>>>>>>>>>> CAPTCHA_CHALLENGE; login-url= >>>>>>>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, >>>>>>>>>>>>> error, r.url, request=request, response=r, **kwargs) >>>>>>>>>>>>> JIRAError: JiraError HTTP 403 url: >>>>>>>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo >>>>>>>>>>>>> text: CAPTCHA_CHALLENGE; login-url= >>>>>>>>>>>>> https://issues.apache.org/jira/login.jsp >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, >>>>>>>>>>>> so I used the credentials to manually log in and complete the >>>>>>>>>>>> challenge. >>>>>>>>>>>> >>>>>>>>>>>> Hopefully that's enough to fix things, but to prevent >>>>>>>>>>>> re-occurrence we might need to change the login credential type >>>>>>>>>>>> from >>>>>>>>>>>> username + password to instead use OAuth tokens. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon < >>>>>>>>>>>> gurwls...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are >>>>>>>>>>>>> being rapidly increased (from around 2400 to 2600) >>>>>>>>>>>>> >>>>>>>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gurwls...@gmail.com>님이 >>>>>>>>>>>>> 작성: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running >>>>>>>>>>>>>> correctly somewhere. >>>>>>>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" >>>>>>>>>>>>>> when >>>>>>>>>>>>>> somebody opens a PR against a JIRA. >>>>>>>>>>>>>> Looks now it only leaves a link and does not change JIRA's >>>>>>>>>>>>>> status. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can someone else who knows where it's running can check this? >>>>>>>>>>>>>> >>>>>>>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever >>>>>>>>>>>>>> since this happened, this makes (at least to me) duplicately >>>>>>>>>>>>>> check the >>>>>>>>>>>>>> JIRAs. >>>>>>>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not >>>>>>>>>>>>>> duplicated because JIRAs having PRs have different status, "IN >>>>>>>>>>>>>> PROGRESS" >>>>>>>>>>>>>> but now all JIRAs have "OPEN" status. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>
