Some updates. For nightly build we are still suffering from the python problem... Anyway, using python2 directly can solve the problem.
I'm still working on it. Thanks. 张铎(Duo Zhang) <[email protected]> 于2022年2月9日周三 09:34写道： > The new Jenkins jobs are still unavailable due to lack of python support. > > If this still can not be resolved tonight, I will rollback to ci-hadoop > and try migration next time. > > Thanks. > > 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 21:35写道： > >> Hi Gavin, the ssh transfer plugin is fine, but now we miss python on >> hbase11-20. >> >> I've filed INFRA-22855, PTAL. >> >> Thanks. >> >> 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 20:19写道： >> >>> Thank you Gavin. WIll take a look soon! >>> >>> Gavin McDonald <[email protected]> 于2022年2月8日周二 18:31写道： >>> >>>> Give it a try now please >>>> >>>> On Sat, Feb 5, 2022 at 8:55 AM 张铎(Duo Zhang) <[email protected]> >>>> wrote: >>>> >>>> > Hi, Gavin, any luck on the ssh transfer plugin? >>>> > >>>> > Thanks. >>>> > >>>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月29日周六 14:53写道： >>>> > >>>> >> So any updates here Gavin? >>>> >> >>>> >> Is the ssh transfer plugin available now? >>>> >> >>>> >> Thanks. >>>> >> >>>> >> Sean Busbey <[email protected]> 于2022年1月23日周日 09:04写道： >>>> >> >>>> >>> It looks like I currently do have access to manage the ci-hbase >>>> >>> controller. >>>> >>> >>>> >>> >>>> >>> Could you check that the git-websites label handling where a common >>>> >>> machine gets leased to the ci-hbase controller is set up properly? >>>> or give >>>> >>> me a pointer on how I would check this myself? >>>> >>> >>>> >>> I started trying to migrate our website generation job again and >>>> it's >>>> >>> been waiting for a node with the git-websites label to show up for >>>> ~8 hours >>>> >>> at this point. >>>> >>> >>>> >>> https://ci-hbase.apache.org/job/hbase_generate_website/3/ >>>> >>> >>>> >>> >>>> >>> >>>> >>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald < >>>> [email protected]> >>>> >>> wrote: >>>> >>> >>>> >>>> Hi, >>>> >>>> Timestamper plugin installed. >>>> >>>> >>>> >>>> By the way I thought I had given the PMC members permissions to >>>> manage >>>> >>>> Jenkins >>>> >>>> on ci-hbase including installing plugins etc, can someone check? >>>> >>>> >>>> >>>> >>>> >>>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) < >>>> [email protected]> >>>> >>>> wrote: >>>> >>>> >>>> >>>> > We faced a strange issue.. >>>> >>>> > >>>> >>>> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: >>>> >>>> startup failed: >>>> >>>> > WorkflowScript: 29: Invalid option type "timestamps". Valid >>>> option >>>> >>>> types: [buildDiscarder, catchError, checkoutToSubdirectory, >>>> >>>> disableConcurrentBuilds, disableResume, durabilityHint, >>>> >>>> githubProjectProperty, newContainerPerStage, overrideIndexTriggers, >>>> >>>> parallelsAlwaysFailFast, preserveStashes, quietPeriod, >>>> rateLimitBuilds, >>>> >>>> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, >>>> timeout, >>>> >>>> waitUntil, warnError, withChecks, withContext, withCredentials, >>>> withEnv, >>>> >>>> wrap, ws] @ line 29, column 5. >>>> >>>> > timestamps() >>>> >>>> > ^ >>>> >>>> > >>>> >>>> > 1 error >>>> >>>> > >>>> >>>> > at >>>> >>>> >>>> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >>>> >>>> > at >>>> >>>> >>>> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >>>> >>>> > at >>>> >>>> >>>> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >>>> >>>> > at >>>> >>>> >>>> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >>>> >>>> > at >>>> >>>> >>>> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >>>> >>>> > at >>>> >>>> >>>> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >>>> >>>> > at >>>> >>>> >>>> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >>>> >>>> > at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >>>> >>>> > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >>>> >>>> > at >>>> >>>> >>>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >>>> >>>> > at >>>> >>>> >>>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >>>> >>>> > at >>>> >>>> >>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >>>> >>>> > at >>>> >>>> >>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >>>> >>>> > at >>>> >>>> >>>> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >>>> >>>> > at >>>> >>>> hudson.model.ResourceController.execute(ResourceController.java:97) >>>> >>>> > at hudson.model.Executor.run(Executor.java:429) >>>> >>>> > >>>> >>>> > >>>> >>>> > After googling, it seems a restart of jenkins can fix it, or we >>>> do not >>>> >>>> > have timestamper plugin installed? Mind taking a look Gavin? >>>> >>>> > >>>> >>>> > Thanks. >>>> >>>> > >>>> >>>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >>>> >>>> > >>>> >>>> >> Oh, just saw that we could do this by ourselves. >>>> >>>> >> >>>> >>>> >> Let me have a try. >>>> >>>> >> >>>> >>>> >> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >>>> >>>> >> >>>> >>>> >>> OK, so while copying you could keep the old jenkins builds? >>>> >>>> >>> That's awesome. >>>> >>>> >>> >>>> >>>> >>> Then please help 'copying' HBase-Find-Flaky-Tests and >>>> >>>> HBase-Flaky-Tests >>>> >>>> >>> to ci-hbase too? We need the history builds to calucate the >>>> flaky >>>> >>>> test list >>>> >>>> >>> so if all the old build data can be kept then it will help us >>>> a lot. >>>> >>>> >>> >>>> >>>> >>> Thanks~ >>>> >>>> >>> >>>> >>>> >>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >>>> >>>> >>> >>>> >>>> >>>> Thank you Gavin. >>>> >>>> >>>> >>>> >>>> >>>> Let me first move some light weighted job first to see if >>>> >>>> everything is >>>> >>>> >>>> OK. >>>> >>>> >>>> >>>> >>>> >>>> HBase Nightly, some flaky related jobs, and also the main pre >>>> >>>> commit >>>> >>>> >>>> job for HBase are tied together. We need to move them at once. >>>> >>>> >>>> >>>> >>>> >>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >>>> >>>> >>>> >>>> >>>> >>>>> Thanks for your patience >>>> >>>> >>>>> >>>> >>>> >>>>> I have 'copied' over a job - >>>> >>>> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >>>> >>>> >>>>> >>>> >>>> >>>>> which of course made me aware of what plugins I needed to >>>> install. >>>> >>>> >>>>> >>>> >>>> >>>>> Feel free to delete that, or use it and delete the old, and >>>> let me >>>> >>>> >>>>> know if have any issues >>>> >>>> >>>>> with any other missing plugins. >>>> >>>> >>>>> >>>> >>>> >>>>> >>>> >>>> >>>>> >>>> >>>> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >>>> >>>> [email protected]> >>>> >>>> >>>>> wrote: >>>> >>>> >>>>> >>>> >>>> >>>>>> I’ve also replied on the jira issue. >>>> >>>> >>>>>> >>>> >>>> >>>>>> You can try to copy a job to the new Jenkins. >>>> >>>> >>>>>> >>>> >>>> >>>>>> Thanks. >>>> >>>> >>>>>> >>>> >>>> >>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： >>>> >>>> >>>>>> >>>> >>>> >>>>>>> Im taking a look ... >>>> >>>> >>>>>>> >>>> >>>> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey < >>>> [email protected]> >>>> >>>> >>>>>>> wrote: >>>> >>>> >>>>>>> >>>> >>>> >>>>>>> > Hi Gavin >>>> >>>> >>>>>>> > >>>> >>>> >>>>>>> > I went to start migrating jobs to the ci-hbase >>>> controller, >>>> >>>> but the >>>> >>>> >>>>>>> new job >>>> >>>> >>>>>>> > workflow doesn't include pipelines or multibranch >>>> pipelines. >>>> >>>> We >>>> >>>> >>>>>>> use both of >>>> >>>> >>>>>>> > these job types for nearly all of our work. Can they be >>>> >>>> enabled? >>>> >>>> >>>>>>> > >>>> >>>> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >>>> >>>> >>>>>>> [email protected]> >>>> >>>> >>>>>>> > wrote: >>>> >>>> >>>>>>> > >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >>>> >>>> >>>>>>> >> > Thank you Gavin. >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > We have some urls in our jenkins files which >>>> references the >>>> >>>> >>>>>>> artifacts of >>>> >>>> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not >>>> work. >>>> >>>> For >>>> >>>> >>>>>>> example, >>>> >>>> >>>>>>> >> here: >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >>>> >>>> >>>> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will >>>> file a >>>> >>>> HBase >>>> >>>> >>>>>>> issue >>>> >>>> >>>>>>> >> to do >>>> >>>> >>>>>>> >> > this. >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > And will report any problems on INFRA_22760 while >>>> >>>> migrating. >>>> >>>> >>>>>>> When the >>>> >>>> >>>>>>> >> > migration is half done, you can move for example, 5 >>>> hbase >>>> >>>> nodes >>>> >>>> >>>>>>> from >>>> >>>> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is >>>> finally >>>> >>>> done, >>>> >>>> >>>>>>> you can >>>> >>>> >>>>>>> >> move >>>> >>>> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to >>>> ci-hbase. >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > WDYT? >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >> However you want to do it is fine with me :) >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >> You have 10 nodes on there right now to get you started. >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >> Let me know how you get on >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > Thanks. >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 >>>> >>>> 16:16写道： >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> > > Hi All, >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > I created an Infra ticket for tracking. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > For now, I left your current nodes hbase1 to >>>> hbase10 in >>>> >>>> >>>>>>> ci-hadoop.a.o >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > I have created for you 10 new nodes so far out of >>>> our >>>> >>>> Hetzner >>>> >>>> >>>>>>> DC >>>> >>>> >>>>>>> >> > > of a similar specifications to the hbase1-10. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Those are hbase11-20 and are available right now for >>>> >>>> testing >>>> >>>> >>>>>>> on >>>> >>>> >>>>>>> >> > > ci-hbase.apache.org >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > I do not know your workflow well enough to decide >>>> the >>>> >>>> next >>>> >>>> >>>>>>> steps for >>>> >>>> >>>>>>> >> you >>>> >>>> >>>>>>> >> > > for testing. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Options include: >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto >>>> ci-hbase >>>> >>>> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of >>>> ci-hadoop >>>> >>>> onto >>>> >>>> >>>>>>> ci-hbase >>>> >>>> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs >>>> over >>>> >>>> for >>>> >>>> >>>>>>> testing. >>>> >>>> >>>>>>> >> > > - I like copy, however quite a few projects jobs >>>> >>>> would >>>> >>>> >>>>>>> not be >>>> >>>> >>>>>>> >> > > appropriate >>>> >>>> >>>>>>> >> > > to have the same job running twice, including >>>> >>>> those that >>>> >>>> >>>>>>> >> deploy, or >>>> >>>> >>>>>>> >> > > those that >>>> >>>> >>>>>>> >> > > build from PRs and continue a workflow to >>>> report >>>> >>>> back, >>>> >>>> >>>>>>> or >>>> >>>> >>>>>>> >> approve >>>> >>>> >>>>>>> >> > > for merge. >>>> >>>> >>>>>>> >> > > So for a copy, would need to be a job that >>>> just >>>> >>>> 'builds >>>> >>>> >>>>>>> and >>>> >>>> >>>>>>> >> tests' >>>> >>>> >>>>>>> >> > > with no outflow. >>>> >>>> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org >>>> to >>>> >>>> test >>>> >>>> >>>>>>> the nodes >>>> >>>> >>>>>>> >> > > setup and the >>>> >>>> >>>>>>> >> > > main controller setup, without affecting current >>>> >>>> jobs. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Obviously, you may to tweak version controlled >>>> script >>>> >>>> files >>>> >>>> >>>>>>> such as a >>>> >>>> >>>>>>> >> > > JenkinsFile, which makes the copy options harder >>>> unless >>>> >>>> >>>>>>> testing from a >>>> >>>> >>>>>>> >> > > branch created for the purpose. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > I may be overthinking things, and option 1 , the >>>> >>>> simplest and >>>> >>>> >>>>>>> most >>>> >>>> >>>>>>> >> > > straight forward from my end would also suit you >>>> guys >>>> >>>> and we >>>> >>>> >>>>>>> deal >>>> >>>> >>>>>>> >> with any >>>> >>>> >>>>>>> >> > > breakage as it happens (I'm thinking of >>>> integrations with >>>> >>>> >>>>>>> other >>>> >>>> >>>>>>> >> services, >>>> >>>> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed >>>> that are >>>> >>>> >>>>>>> needed etc.) >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Let me know please ASAP how you want to proceed so >>>> we >>>> >>>> can get >>>> >>>> >>>>>>> using >>>> >>>> >>>>>>> >> these >>>> >>>> >>>>>>> >> > > nodes, after which I can then proceed to move >>>> hbase1-10 >>>> >>>> nodes >>>> >>>> >>>>>>> over >>>> >>>> >>>>>>> >> also. >>>> >>>> >>>>>>> >> > > And then start adding more into the mix. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It >>>> is >>>> >>>> >>>>>>> available on >>>> >>>> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are >>>> shared and >>>> >>>> >>>>>>> leased out to >>>> >>>> >>>>>>> >> > > controllers as needed upon request of a job. >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Thanks >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > Gav... >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >>>> >>>> >>>>>>> >> > > > Thanks you Gavin. >>>> >>>> >>>>>>> >> > > > >>>> >>>> >>>>>>> >> > > > You can do it in your convenience, just let me >>>> know so >>>> >>>> I >>>> >>>> >>>>>>> could >>>> >>>> >>>>>>> >> change the >>>> >>>> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at >>>> least >>>> >>>> our >>>> >>>> >>>>>>> pre >>>> >>>> >>>>>>> >> commit job >>>> >>>> >>>>>>> >> > > > still uses the Hadoop label. >>>> >>>> >>>>>>> >> > > > >>>> >>>> >>>>>>> >> > > > And another thing is about the >>>> hbase_generate_website >>>> >>>> job, >>>> >>>> >>>>>>> it needs >>>> >>>> >>>>>>> >> to be >>>> >>>> >>>>>>> >> > > > executed on the nodes under the git-websites >>>> label. >>>> >>>> I'm not >>>> >>>> >>>>>>> sure if >>>> >>>> >>>>>>> >> the >>>> >>>> >>>>>>> >> > > > label is still available after the migration. >>>> >>>> >>>>>>> >> > > > >>>> >>>> >>>>>>> >> > > > Gavin McDonald <[email protected]> >>>> 于2021年12月8日周三 >>>> >>>> >>>>>>> 17:28写道： >>>> >>>> >>>>>>> >> > > > >>>> >>>> >>>>>>> >> > > > > Hi HBase devs. >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > This is to let you know that Infra has created a >>>> >>>> >>>>>>> dedicated Jenkins >>>> >>>> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > You have new incoming donated agents which will >>>> get >>>> >>>> >>>>>>> connected to >>>> >>>> >>>>>>> >> this >>>> >>>> >>>>>>> >> > > new >>>> >>>> >>>>>>> >> > > > > controller by Infra. >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > You also have 10 or so agents connected to >>>> >>>> >>>>>>> >> > > https://ci-hadoop.apache.org - >>>> >>>> >>>>>>> >> > > > > I >>>> >>>> >>>>>>> >> > > > > would like to move these agents - and all of >>>> your >>>> >>>> jobs - >>>> >>>> >>>>>>> over to >>>> >>>> >>>>>>> >> the >>>> >>>> >>>>>>> >> > > new >>>> >>>> >>>>>>> >> > > > > controller as soon as possible. >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > You may or may not need to reconfigure your >>>> jobs and >>>> >>>> or >>>> >>>> >>>>>>> >> JenkinsFile or >>>> >>>> >>>>>>> >> > > > > other. >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > Can I get the go ahead to move these at my >>>> >>>> convenience or >>>> >>>> >>>>>>> suggest >>>> >>>> >>>>>>> >> > > please a >>>> >>>> >>>>>>> >> > > > > timeline for Infra to move these agents. >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > Thanks! >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > -- >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > > *Gavin McDonald* >>>> >>>> >>>>>>> >> > > > > Systems Administrator >>>> >>>> >>>>>>> >> > > > > ASF Infrastructure Team >>>> >>>> >>>>>>> >> > > > > >>>> >>>> >>>>>>> >> > > > >>>> >>>> >>>>>>> >> > > >>>> >>>> >>>>>>> >> > >>>> >>>> >>>>>>> >> >>>> >>>> >>>>>>> > >>>> >>>> >>>>>>> >>>> >>>> >>>>>>> -- >>>> >>>> >>>>>>> >>>> >>>> >>>>>>> *Gavin McDonald* >>>> >>>> >>>>>>> Systems Administrator >>>> >>>> >>>>>>> ASF Infrastructure Team >>>> >>>> >>>>>>> >>>> >>>> >>>>>> >>>> >>>> >>>>> >>>> >>>> >>>>> -- >>>> >>>> >>>>> >>>> >>>> >>>>> *Gavin McDonald* >>>> >>>> >>>>> Systems Administrator >>>> >>>> >>>>> ASF Infrastructure Team >>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> >>>> *Gavin McDonald* >>>> >>>> Systems Administrator >>>> >>>> ASF Infrastructure Team >>>> >>>> >>>> >>> >>>> >>>> -- >>>> >>>> *Gavin McDonald* >>>> Systems Administrator >>>> ASF Infrastructure Team >>>> >>>
