Re: Spark Packaging Jenkins
> > 1) https://issues.apache.org/jira/browse/SPARK-26565 > 2) create world-wide campaign to stop bash scripting > 3) re-create these builds w/o calling release-build.sh at all (preferred)
Re: Spark Packaging Jenkins
after spending the last three hours unraveling how dev/create-release/release-build.sh works, i think i figured out what needs to happen. 1) https://issues.apache.org/jira/browse/SPARK-26565 2) create world-wide campaign to stop bash scripting my brain... it hurts. On Sun, Jan 6, 2019 at 1:07 PM Dongjoon Hyun wrote: > Thank you, Felix and Shane. > > +1 for packaging except signing in Jenkins. That sounds perfect for the > next releases (2.2.3, 2.3.3, 3.0.0). > > BTW, thanks to Shane, the followings are recovered from today. > > 1. For all live branches (master to branch-2.2), Jenkins Packaging jobs > are recovered to build and packaging. Only there exist signing errors. > > - Master: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2181/console > - branch-2.4: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.4-package/95/console > - branch-2.3: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.3-package/230/console > - branch-2.2: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.2-package/240/console > > 2. Daily snapshot maven publishing is recovered. This is helpful for > partial module testing. > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/ > > Thank you all. Everything becomes healthier during 2.2.3 release. > This will reduce the burden of 2.3.3/3.0.0 releases. :D > > Bests, > Dongjoon. > > > On Sun, Jan 6, 2019 at 11:42 AM Felix Cheung > wrote: > >> Awesome Shane! >> >> >> -- >> *From:* shane knapp >> *Sent:* Sunday, January 6, 2019 11:38 AM >> *To:* Felix Cheung >> *Cc:* Dongjoon Hyun; Wenchen Fan; dev >> *Subject:* Re: Spark Packaging Jenkins >> >> noted. i like the idea of building (but not signing) the release and >> will update the job(s) this week. >> >> On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung >> wrote: >> >>> https://spark.apache.org/release-process.html >>> >>> Look for do-release-docker.sh script >>> >>> >>> -- >>> *From:* Felix Cheung >>> *Sent:* Sunday, January 6, 2019 11:17 AM >>> *To:* Dongjoon Hyun; Wenchen Fan >>> *Cc:* dev; shane knapp >>> *Subject:* Re: Spark Packaging Jenkins >>> >>> The release process doc should have been updated on this - as mentioned >>> we do not use Jenkins for release signing (take this offline if further >>> discussion is needed) >>> >>> The release build on Jenkins can still be useful for pre-validating the >>> release build process (without actually signing it) >>> >>> >>> -- >>> *From:* Dongjoon Hyun >>> *Sent:* Saturday, January 5, 2019 9:46 PM >>> *To:* Wenchen Fan >>> *Cc:* dev; shane knapp >>> *Subject:* Re: Spark Packaging Jenkins >>> >>> Thank you, Wenchen. >>> >>> I see. I'll update the doc and proceed to the next step manually as you >>> advise. And it seems that we can stop the outdated Jenkins jobs, too. >>> >>> Bests, >>> Dongjoon. >>> >>> On Sat, Jan 5, 2019 at 20:15 Wenchen Fan wrote: >>> >>>> IIRC there was a change to the release process: we stop using the >>>> shared gpg key on Jenkins, but use the personal key of the release manager. >>>> I'm not sure Jenkins can help testing package anymore. >>>> >>>> BTW release manager needs to run the packaging script by himself. If >>>> there is a problem, the release manager will find it out sooner or later. >>>> >>>> >>>> >>>> On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun >>>> wrote: >>>> >>>>> Hi, All. >>>>> >>>>> It turns out that `gpg signing` is the next huddle in Spark Packaging >>>>> Jenkins. >>>>> Since 2.4.0 release, is there something changed in our Jenkins machine? >>>>> >>>>> gpg: skipped >>>>> "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": >>>>> No secret key >>>>> gpg: signing failed: No secret key >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>>
Re: Spark Packaging Jenkins
Thank you, Felix and Shane. +1 for packaging except signing in Jenkins. That sounds perfect for the next releases (2.2.3, 2.3.3, 3.0.0). BTW, thanks to Shane, the followings are recovered from today. 1. For all live branches (master to branch-2.2), Jenkins Packaging jobs are recovered to build and packaging. Only there exist signing errors. - Master: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2181/console - branch-2.4: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.4-package/95/console - branch-2.3: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.3-package/230/console - branch-2.2: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.2-package/240/console 2. Daily snapshot maven publishing is recovered. This is helpful for partial module testing. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/ Thank you all. Everything becomes healthier during 2.2.3 release. This will reduce the burden of 2.3.3/3.0.0 releases. :D Bests, Dongjoon. On Sun, Jan 6, 2019 at 11:42 AM Felix Cheung wrote: > Awesome Shane! > > > -- > *From:* shane knapp > *Sent:* Sunday, January 6, 2019 11:38 AM > *To:* Felix Cheung > *Cc:* Dongjoon Hyun; Wenchen Fan; dev > *Subject:* Re: Spark Packaging Jenkins > > noted. i like the idea of building (but not signing) the release and will > update the job(s) this week. > > On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung > wrote: > >> https://spark.apache.org/release-process.html >> >> Look for do-release-docker.sh script >> >> >> -- >> *From:* Felix Cheung >> *Sent:* Sunday, January 6, 2019 11:17 AM >> *To:* Dongjoon Hyun; Wenchen Fan >> *Cc:* dev; shane knapp >> *Subject:* Re: Spark Packaging Jenkins >> >> The release process doc should have been updated on this - as mentioned >> we do not use Jenkins for release signing (take this offline if further >> discussion is needed) >> >> The release build on Jenkins can still be useful for pre-validating the >> release build process (without actually signing it) >> >> >> ------ >> *From:* Dongjoon Hyun >> *Sent:* Saturday, January 5, 2019 9:46 PM >> *To:* Wenchen Fan >> *Cc:* dev; shane knapp >> *Subject:* Re: Spark Packaging Jenkins >> >> Thank you, Wenchen. >> >> I see. I'll update the doc and proceed to the next step manually as you >> advise. And it seems that we can stop the outdated Jenkins jobs, too. >> >> Bests, >> Dongjoon. >> >> On Sat, Jan 5, 2019 at 20:15 Wenchen Fan wrote: >> >>> IIRC there was a change to the release process: we stop using the shared >>> gpg key on Jenkins, but use the personal key of the release manager. I'm >>> not sure Jenkins can help testing package anymore. >>> >>> BTW release manager needs to run the packaging script by himself. If >>> there is a problem, the release manager will find it out sooner or later. >>> >>> >>> >>> On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun >>> wrote: >>> >>>> Hi, All. >>>> >>>> It turns out that `gpg signing` is the next huddle in Spark Packaging >>>> Jenkins. >>>> Since 2.4.0 release, is there something changed in our Jenkins machine? >>>> >>>> gpg: skipped >>>> "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": >>>> No secret key >>>> gpg: signing failed: No secret key >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> >>>> On Fri, Jan 4, 2019 at 11:52 AM shane knapp >>>> wrote: >>>> >>>>> https://issues.apache.org/jira/browse/SPARK-26537 >>>>> >>>>> On Fri, Jan 4, 2019 at 11:31 AM shane knapp >>>>> wrote: >>>>> >>>>>> this may push in to early next week... these builds were set up >>>>>> before my time, and i'm currently unraveling how they all work before >>>>>> pushing a commit to fix stuff. >>>>>> >>>>>> nothing like some code archaeology to make my friday more exciting! >>>>>> :) >>>>>> >>>>>> shane >>>>>> >>>>>> On Fri, Jan 4,
Re: Spark Packaging Jenkins
Awesome Shane! From: shane knapp Sent: Sunday, January 6, 2019 11:38 AM To: Felix Cheung Cc: Dongjoon Hyun; Wenchen Fan; dev Subject: Re: Spark Packaging Jenkins noted. i like the idea of building (but not signing) the release and will update the job(s) this week. On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung mailto:felixcheun...@hotmail.com>> wrote: https://spark.apache.org/release-process.html Look for do-release-docker.sh script From: Felix Cheung mailto:felixcheun...@hotmail.com>> Sent: Sunday, January 6, 2019 11:17 AM To: Dongjoon Hyun; Wenchen Fan Cc: dev; shane knapp Subject: Re: Spark Packaging Jenkins The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed) The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it) From: Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> Sent: Saturday, January 5, 2019 9:46 PM To: Wenchen Fan Cc: dev; shane knapp Subject: Re: Spark Packaging Jenkins Thank you, Wenchen. I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too. Bests, Dongjoon. On Sat, Jan 5, 2019 at 20:15 Wenchen Fan mailto:cloud0...@gmail.com>> wrote: IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore. BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later. On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Hi, All. It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins. Since 2.4.0 release, is there something changed in our Jenkins machine? gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key gpg: signing failed: No secret key Bests, Dongjoon. On Fri, Jan 4, 2019 at 11:52 AM shane knapp mailto:skn...@berkeley.edu>> wrote: https://issues.apache.org/jira/browse/SPARK-26537 On Fri, Jan 4, 2019 at 11:31 AM shane knapp mailto:skn...@berkeley.edu>> wrote: this may push in to early next week... these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff. nothing like some code archaeology to make my friday more exciting! :) shane On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Thank you, Shane! Bests, Dongjoon. On Fri, Jan 4, 2019 at 10:50 AM shane knapp mailto:skn...@berkeley.edu>> wrote: yeah, i'll get on that today. thanks for the heads up. On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Hi, All As a part of release process, we need to check Packaging/Compile/Test Jenkins status. http://spark.apache.org/release-process.html 1. Spark Packaging: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ 2. Spark QA Compile: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ 3. Spark QA Test: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git). But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox. Can we fix this in this week? Bests, Dongjoon. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Spark Packaging Jenkins
noted. i like the idea of building (but not signing) the release and will update the job(s) this week. On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung wrote: > https://spark.apache.org/release-process.html > > Look for do-release-docker.sh script > > > -- > *From:* Felix Cheung > *Sent:* Sunday, January 6, 2019 11:17 AM > *To:* Dongjoon Hyun; Wenchen Fan > *Cc:* dev; shane knapp > *Subject:* Re: Spark Packaging Jenkins > > The release process doc should have been updated on this - as mentioned we > do not use Jenkins for release signing (take this offline if further > discussion is needed) > > The release build on Jenkins can still be useful for pre-validating the > release build process (without actually signing it) > > > -- > *From:* Dongjoon Hyun > *Sent:* Saturday, January 5, 2019 9:46 PM > *To:* Wenchen Fan > *Cc:* dev; shane knapp > *Subject:* Re: Spark Packaging Jenkins > > Thank you, Wenchen. > > I see. I'll update the doc and proceed to the next step manually as you > advise. And it seems that we can stop the outdated Jenkins jobs, too. > > Bests, > Dongjoon. > > On Sat, Jan 5, 2019 at 20:15 Wenchen Fan wrote: > >> IIRC there was a change to the release process: we stop using the shared >> gpg key on Jenkins, but use the personal key of the release manager. I'm >> not sure Jenkins can help testing package anymore. >> >> BTW release manager needs to run the packaging script by himself. If >> there is a problem, the release manager will find it out sooner or later. >> >> >> >> On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun >> wrote: >> >>> Hi, All. >>> >>> It turns out that `gpg signing` is the next huddle in Spark Packaging >>> Jenkins. >>> Since 2.4.0 release, is there something changed in our Jenkins machine? >>> >>> gpg: skipped >>> "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": >>> No secret key >>> gpg: signing failed: No secret key >>> >>> Bests, >>> Dongjoon. >>> >>> >>> On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: >>> >>>> https://issues.apache.org/jira/browse/SPARK-26537 >>>> >>>> On Fri, Jan 4, 2019 at 11:31 AM shane knapp >>>> wrote: >>>> >>>>> this may push in to early next week... these builds were set up >>>>> before my time, and i'm currently unraveling how they all work before >>>>> pushing a commit to fix stuff. >>>>> >>>>> nothing like some code archaeology to make my friday more exciting! :) >>>>> >>>>> shane >>>>> >>>>> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun >>>>> wrote: >>>>> >>>>>> Thank you, Shane! >>>>>> >>>>>> Bests, >>>>>> Dongjoon. >>>>>> >>>>>> On Fri, Jan 4, 2019 at 10:50 AM shane knapp >>>>>> wrote: >>>>>> >>>>>>> yeah, i'll get on that today. thanks for the heads up. >>>>>>> >>>>>>> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun < >>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, All >>>>>>>> >>>>>>>> As a part of release process, we need to check >>>>>>>> Packaging/Compile/Test Jenkins status. >>>>>>>> >>>>>>>> http://spark.apache.org/release-process.html >>>>>>>> >>>>>>>> 1. Spark Packaging: >>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>>>>>>> 2. Spark QA Compile: >>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>>>>>>> 3. Spark QA Test: >>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>>>>>>> >>>>>>>> Currently, (2) and (3) are working because it uses GitHub ( >>>>>>>> https://github.com/apache/spark.git). >>>>>>>> But, (1) seems to be broken because it's looking for old repo( >>>>>>>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) >>>>>>>> instead of new GitBox. >>>>>>>> >>>>>>>> Can we fix this in this week? >>>>>>>> >>>>>>>> Bests, >>>>>>>> Dongjoon. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Shane Knapp >>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>>> https://rise.cs.berkeley.edu >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> Shane Knapp >>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>> https://rise.cs.berkeley.edu >>>>> >>>> >>>> >>>> -- >>>> Shane Knapp >>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>> https://rise.cs.berkeley.edu >>>> >>> -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Spark Packaging Jenkins
https://spark.apache.org/release-process.html Look for do-release-docker.sh script From: Felix Cheung Sent: Sunday, January 6, 2019 11:17 AM To: Dongjoon Hyun; Wenchen Fan Cc: dev; shane knapp Subject: Re: Spark Packaging Jenkins The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed) The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it) From: Dongjoon Hyun Sent: Saturday, January 5, 2019 9:46 PM To: Wenchen Fan Cc: dev; shane knapp Subject: Re: Spark Packaging Jenkins Thank you, Wenchen. I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too. Bests, Dongjoon. On Sat, Jan 5, 2019 at 20:15 Wenchen Fan mailto:cloud0...@gmail.com>> wrote: IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore. BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later. On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Hi, All. It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins. Since 2.4.0 release, is there something changed in our Jenkins machine? gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key gpg: signing failed: No secret key Bests, Dongjoon. On Fri, Jan 4, 2019 at 11:52 AM shane knapp mailto:skn...@berkeley.edu>> wrote: https://issues.apache.org/jira/browse/SPARK-26537 On Fri, Jan 4, 2019 at 11:31 AM shane knapp mailto:skn...@berkeley.edu>> wrote: this may push in to early next week... these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff. nothing like some code archaeology to make my friday more exciting! :) shane On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Thank you, Shane! Bests, Dongjoon. On Fri, Jan 4, 2019 at 10:50 AM shane knapp mailto:skn...@berkeley.edu>> wrote: yeah, i'll get on that today. thanks for the heads up. On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Hi, All As a part of release process, we need to check Packaging/Compile/Test Jenkins status. http://spark.apache.org/release-process.html 1. Spark Packaging: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ 2. Spark QA Compile: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ 3. Spark QA Test: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git). But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox. Can we fix this in this week? Bests, Dongjoon. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Spark Packaging Jenkins
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed) The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it) From: Dongjoon Hyun Sent: Saturday, January 5, 2019 9:46 PM To: Wenchen Fan Cc: dev; shane knapp Subject: Re: Spark Packaging Jenkins Thank you, Wenchen. I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too. Bests, Dongjoon. On Sat, Jan 5, 2019 at 20:15 Wenchen Fan mailto:cloud0...@gmail.com>> wrote: IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore. BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later. On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Hi, All. It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins. Since 2.4.0 release, is there something changed in our Jenkins machine? gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key gpg: signing failed: No secret key Bests, Dongjoon. On Fri, Jan 4, 2019 at 11:52 AM shane knapp mailto:skn...@berkeley.edu>> wrote: https://issues.apache.org/jira/browse/SPARK-26537 On Fri, Jan 4, 2019 at 11:31 AM shane knapp mailto:skn...@berkeley.edu>> wrote: this may push in to early next week... these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff. nothing like some code archaeology to make my friday more exciting! :) shane On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Thank you, Shane! Bests, Dongjoon. On Fri, Jan 4, 2019 at 10:50 AM shane knapp mailto:skn...@berkeley.edu>> wrote: yeah, i'll get on that today. thanks for the heads up. On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: Hi, All As a part of release process, we need to check Packaging/Compile/Test Jenkins status. http://spark.apache.org/release-process.html 1. Spark Packaging: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ 2. Spark QA Compile: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ 3. Spark QA Test: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git). But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox. Can we fix this in this week? Bests, Dongjoon. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Spark Packaging Jenkins
Thank you, Wenchen. I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too. Bests, Dongjoon. On Sat, Jan 5, 2019 at 20:15 Wenchen Fan wrote: > IIRC there was a change to the release process: we stop using the shared > gpg key on Jenkins, but use the personal key of the release manager. I'm > not sure Jenkins can help testing package anymore. > > BTW release manager needs to run the packaging script by himself. If there > is a problem, the release manager will find it out sooner or later. > > > > On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun > wrote: > >> Hi, All. >> >> It turns out that `gpg signing` is the next huddle in Spark Packaging >> Jenkins. >> Since 2.4.0 release, is there something changed in our Jenkins machine? >> >> gpg: skipped >> "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": >> No secret key >> gpg: signing failed: No secret key >> >> Bests, >> Dongjoon. >> >> >> On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: >> >>> https://issues.apache.org/jira/browse/SPARK-26537 >>> >>> On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: >>> >>>> this may push in to early next week... these builds were set up before >>>> my time, and i'm currently unraveling how they all work before pushing a >>>> commit to fix stuff. >>>> >>>> nothing like some code archaeology to make my friday more exciting! :) >>>> >>>> shane >>>> >>>> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun >>>> wrote: >>>> >>>>> Thank you, Shane! >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> On Fri, Jan 4, 2019 at 10:50 AM shane knapp >>>>> wrote: >>>>> >>>>>> yeah, i'll get on that today. thanks for the heads up. >>>>>> >>>>>> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun < >>>>>> dongjoon.h...@gmail.com> wrote: >>>>>> >>>>>>> Hi, All >>>>>>> >>>>>>> As a part of release process, we need to check >>>>>>> Packaging/Compile/Test Jenkins status. >>>>>>> >>>>>>> http://spark.apache.org/release-process.html >>>>>>> >>>>>>> 1. Spark Packaging: >>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>>>>>> 2. Spark QA Compile: >>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>>>>>> 3. Spark QA Test: >>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>>>>>> >>>>>>> Currently, (2) and (3) are working because it uses GitHub ( >>>>>>> https://github.com/apache/spark.git). >>>>>>> But, (1) seems to be broken because it's looking for old repo( >>>>>>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) >>>>>>> instead of new GitBox. >>>>>>> >>>>>>> Can we fix this in this week? >>>>>>> >>>>>>> Bests, >>>>>>> Dongjoon. >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Shane Knapp >>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>> https://rise.cs.berkeley.edu >>>>>> >>>>> >>>> >>>> -- >>>> Shane Knapp >>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>> https://rise.cs.berkeley.edu >>>> >>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >>
Re: Spark Packaging Jenkins
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore. BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later. On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun wrote: > Hi, All. > > It turns out that `gpg signing` is the next huddle in Spark Packaging > Jenkins. > Since 2.4.0 release, is there something changed in our Jenkins machine? > > gpg: skipped > "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": > No secret key > gpg: signing failed: No secret key > > Bests, > Dongjoon. > > > On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: > >> https://issues.apache.org/jira/browse/SPARK-26537 >> >> On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: >> >>> this may push in to early next week... these builds were set up before >>> my time, and i'm currently unraveling how they all work before pushing a >>> commit to fix stuff. >>> >>> nothing like some code archaeology to make my friday more exciting! :) >>> >>> shane >>> >>> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun >>> wrote: >>> >>>> Thank you, Shane! >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> On Fri, Jan 4, 2019 at 10:50 AM shane knapp >>>> wrote: >>>> >>>>> yeah, i'll get on that today. thanks for the heads up. >>>>> >>>>> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun >>>>> wrote: >>>>> >>>>>> Hi, All >>>>>> >>>>>> As a part of release process, we need to check Packaging/Compile/Test >>>>>> Jenkins status. >>>>>> >>>>>> http://spark.apache.org/release-process.html >>>>>> >>>>>> 1. Spark Packaging: >>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>>>>> 2. Spark QA Compile: >>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>>>>> 3. Spark QA Test: >>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>>>>> >>>>>> Currently, (2) and (3) are working because it uses GitHub ( >>>>>> https://github.com/apache/spark.git). >>>>>> But, (1) seems to be broken because it's looking for old repo( >>>>>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead >>>>>> of new GitBox. >>>>>> >>>>>> Can we fix this in this week? >>>>>> >>>>>> Bests, >>>>>> Dongjoon. >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Shane Knapp >>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>> https://rise.cs.berkeley.edu >>>>> >>>> >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> >
Re: Spark Packaging Jenkins
Hi, All. It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins. Since 2.4.0 release, is there something changed in our Jenkins machine? gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key gpg: signing failed: No secret key Bests, Dongjoon. On Fri, Jan 4, 2019 at 11:52 AM shane knapp wrote: > https://issues.apache.org/jira/browse/SPARK-26537 > > On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: > >> this may push in to early next week... these builds were set up before >> my time, and i'm currently unraveling how they all work before pushing a >> commit to fix stuff. >> >> nothing like some code archaeology to make my friday more exciting! :) >> >> shane >> >> On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun >> wrote: >> >>> Thank you, Shane! >>> >>> Bests, >>> Dongjoon. >>> >>> On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: >>> >>>> yeah, i'll get on that today. thanks for the heads up. >>>> >>>> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun >>>> wrote: >>>> >>>>> Hi, All >>>>> >>>>> As a part of release process, we need to check Packaging/Compile/Test >>>>> Jenkins status. >>>>> >>>>> http://spark.apache.org/release-process.html >>>>> >>>>> 1. Spark Packaging: >>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>>>> 2. Spark QA Compile: >>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>>>> 3. Spark QA Test: >>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>>>> >>>>> Currently, (2) and (3) are working because it uses GitHub ( >>>>> https://github.com/apache/spark.git). >>>>> But, (1) seems to be broken because it's looking for old repo( >>>>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead >>>>> of new GitBox. >>>>> >>>>> Can we fix this in this week? >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>> >>>> -- >>>> Shane Knapp >>>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>> https://rise.cs.berkeley.edu >>>> >>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu >
Re: Spark Packaging Jenkins
https://issues.apache.org/jira/browse/SPARK-26537 On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: > this may push in to early next week... these builds were set up before my > time, and i'm currently unraveling how they all work before pushing a > commit to fix stuff. > > nothing like some code archaeology to make my friday more exciting! :) > > shane > > On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun > wrote: > >> Thank you, Shane! >> >> Bests, >> Dongjoon. >> >> On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: >> >>> yeah, i'll get on that today. thanks for the heads up. >>> >>> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun >>> wrote: >>> Hi, All As a part of release process, we need to check Packaging/Compile/Test Jenkins status. http://spark.apache.org/release-process.html 1. Spark Packaging: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ 2. Spark QA Compile: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ 3. Spark QA Test: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ Currently, (2) and (3) are working because it uses GitHub ( https://github.com/apache/spark.git). But, (1) seems to be broken because it's looking for old repo( https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox. Can we fix this in this week? Bests, Dongjoon. >>> >>> -- >>> Shane Knapp >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> https://rise.cs.berkeley.edu >>> >> > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Spark Packaging Jenkins
this may push in to early next week... these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff. nothing like some code archaeology to make my friday more exciting! :) shane On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun wrote: > Thank you, Shane! > > Bests, > Dongjoon. > > On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: > >> yeah, i'll get on that today. thanks for the heads up. >> >> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun >> wrote: >> >>> Hi, All >>> >>> As a part of release process, we need to check Packaging/Compile/Test >>> Jenkins status. >>> >>> http://spark.apache.org/release-process.html >>> >>> 1. Spark Packaging: >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >>> 2. Spark QA Compile: >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >>> 3. Spark QA Test: >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >>> >>> Currently, (2) and (3) are working because it uses GitHub ( >>> https://github.com/apache/spark.git). >>> But, (1) seems to be broken because it's looking for old repo( >>> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of >>> new GitBox. >>> >>> Can we fix this in this week? >>> >>> Bests, >>> Dongjoon. >>> >>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Re: Spark Packaging Jenkins
Thank you, Shane! Bests, Dongjoon. On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: > yeah, i'll get on that today. thanks for the heads up. > > On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun > wrote: > >> Hi, All >> >> As a part of release process, we need to check Packaging/Compile/Test >> Jenkins status. >> >> http://spark.apache.org/release-process.html >> >> 1. Spark Packaging: >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ >> 2. Spark QA Compile: >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ >> 3. Spark QA Test: >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ >> >> Currently, (2) and (3) are working because it uses GitHub ( >> https://github.com/apache/spark.git). >> But, (1) seems to be broken because it's looking for old repo( >> https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of >> new GitBox. >> >> Can we fix this in this week? >> >> Bests, >> Dongjoon. >> >> > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technical Lead > https://rise.cs.berkeley.edu >
Re: Spark Packaging Jenkins
yeah, i'll get on that today. thanks for the heads up. On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun wrote: > Hi, All > > As a part of release process, we need to check Packaging/Compile/Test > Jenkins status. > > http://spark.apache.org/release-process.html > > 1. Spark Packaging: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ > 2. Spark QA Compile: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ > 3. Spark QA Test: > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ > > Currently, (2) and (3) are working because it uses GitHub ( > https://github.com/apache/spark.git). > But, (1) seems to be broken because it's looking for old repo( > https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of > new GitBox. > > Can we fix this in this week? > > Bests, > Dongjoon. > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
Spark Packaging Jenkins
Hi, All As a part of release process, we need to check Packaging/Compile/Test Jenkins status. http://spark.apache.org/release-process.html 1. Spark Packaging: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/ 2. Spark QA Compile: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/ 3. Spark QA Test: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/ Currently, (2) and (3) are working because it uses GitHub ( https://github.com/apache/spark.git). But, (1) seems to be broken because it's looking for old repo( https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox. Can we fix this in this week? Bests, Dongjoon.
