The new Jenkins jobs are still unavailable due to lack of python support. If this still can not be resolved tonight, I will rollback to ci-hadoop and try migration next time.
Thanks. 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 21:35写道： > Hi Gavin, the ssh transfer plugin is fine, but now we miss python on > hbase11-20. > > I've filed INFRA-22855, PTAL. > > Thanks. > > 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 20:19写道： > >> Thank you Gavin. WIll take a look soon! >> >> Gavin McDonald <[email protected]> 于2022年2月8日周二 18:31写道： >> >>> Give it a try now please >>> >>> On Sat, Feb 5, 2022 at 8:55 AM 张铎(Duo Zhang) <[email protected]> >>> wrote: >>> >>> > Hi, Gavin, any luck on the ssh transfer plugin? >>> > >>> > Thanks. >>> > >>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月29日周六 14:53写道： >>> > >>> >> So any updates here Gavin? >>> >> >>> >> Is the ssh transfer plugin available now? >>> >> >>> >> Thanks. >>> >> >>> >> Sean Busbey <[email protected]> 于2022年1月23日周日 09:04写道： >>> >> >>> >>> It looks like I currently do have access to manage the ci-hbase >>> >>> controller. >>> >>> >>> >>> >>> >>> Could you check that the git-websites label handling where a common >>> >>> machine gets leased to the ci-hbase controller is set up properly? >>> or give >>> >>> me a pointer on how I would check this myself? >>> >>> >>> >>> I started trying to migrate our website generation job again and it's >>> >>> been waiting for a node with the git-websites label to show up for >>> ~8 hours >>> >>> at this point. >>> >>> >>> >>> https://ci-hbase.apache.org/job/hbase_generate_website/3/ >>> >>> >>> >>> >>> >>> >>> >>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald <[email protected] >>> > >>> >>> wrote: >>> >>> >>> >>>> Hi, >>> >>>> Timestamper plugin installed. >>> >>>> >>> >>>> By the way I thought I had given the PMC members permissions to >>> manage >>> >>>> Jenkins >>> >>>> on ci-hbase including installing plugins etc, can someone check? >>> >>>> >>> >>>> >>> >>>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) < >>> [email protected]> >>> >>>> wrote: >>> >>>> >>> >>>> > We faced a strange issue.. >>> >>>> > >>> >>>> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: >>> >>>> startup failed: >>> >>>> > WorkflowScript: 29: Invalid option type "timestamps". Valid option >>> >>>> types: [buildDiscarder, catchError, checkoutToSubdirectory, >>> >>>> disableConcurrentBuilds, disableResume, durabilityHint, >>> >>>> githubProjectProperty, newContainerPerStage, overrideIndexTriggers, >>> >>>> parallelsAlwaysFailFast, preserveStashes, quietPeriod, >>> rateLimitBuilds, >>> >>>> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, >>> timeout, >>> >>>> waitUntil, warnError, withChecks, withContext, withCredentials, >>> withEnv, >>> >>>> wrap, ws] @ line 29, column 5. >>> >>>> > timestamps() >>> >>>> > ^ >>> >>>> > >>> >>>> > 1 error >>> >>>> > >>> >>>> > at >>> >>>> >>> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >>> >>>> > at >>> >>>> >>> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >>> >>>> > at >>> >>>> >>> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >>> >>>> > at >>> >>>> >>> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >>> >>>> > at >>> >>>> >>> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >>> >>>> > at >>> >>>> >>> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >>> >>>> > at >>> >>>> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >>> >>>> > at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >>> >>>> > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >>> >>>> > at >>> >>>> >>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >>> >>>> > at >>> >>>> >>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >>> >>>> > at >>> >>>> >>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >>> >>>> > at >>> >>>> >>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >>> >>>> > at >>> >>>> >>> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >>> >>>> > at >>> >>>> hudson.model.ResourceController.execute(ResourceController.java:97) >>> >>>> > at hudson.model.Executor.run(Executor.java:429) >>> >>>> > >>> >>>> > >>> >>>> > After googling, it seems a restart of jenkins can fix it, or we >>> do not >>> >>>> > have timestamper plugin installed? Mind taking a look Gavin? >>> >>>> > >>> >>>> > Thanks. >>> >>>> > >>> >>>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >>> >>>> > >>> >>>> >> Oh, just saw that we could do this by ourselves. >>> >>>> >> >>> >>>> >> Let me have a try. >>> >>>> >> >>> >>>> >> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >>> >>>> >> >>> >>>> >>> OK, so while copying you could keep the old jenkins builds? >>> >>>> >>> That's awesome. >>> >>>> >>> >>> >>>> >>> Then please help 'copying' HBase-Find-Flaky-Tests and >>> >>>> HBase-Flaky-Tests >>> >>>> >>> to ci-hbase too? We need the history builds to calucate the >>> flaky >>> >>>> test list >>> >>>> >>> so if all the old build data can be kept then it will help us a >>> lot. >>> >>>> >>> >>> >>>> >>> Thanks~ >>> >>>> >>> >>> >>>> >>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >>> >>>> >>> >>> >>>> >>>> Thank you Gavin. >>> >>>> >>>> >>> >>>> >>>> Let me first move some light weighted job first to see if >>> >>>> everything is >>> >>>> >>>> OK. >>> >>>> >>>> >>> >>>> >>>> HBase Nightly, some flaky related jobs, and also the main pre >>> >>>> commit >>> >>>> >>>> job for HBase are tied together. We need to move them at once. >>> >>>> >>>> >>> >>>> >>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >>> >>>> >>>> >>> >>>> >>>>> Thanks for your patience >>> >>>> >>>>> >>> >>>> >>>>> I have 'copied' over a job - >>> >>>> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >>> >>>> >>>>> >>> >>>> >>>>> which of course made me aware of what plugins I needed to >>> install. >>> >>>> >>>>> >>> >>>> >>>>> Feel free to delete that, or use it and delete the old, and >>> let me >>> >>>> >>>>> know if have any issues >>> >>>> >>>>> with any other missing plugins. >>> >>>> >>>>> >>> >>>> >>>>> >>> >>>> >>>>> >>> >>>> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >>> >>>> [email protected]> >>> >>>> >>>>> wrote: >>> >>>> >>>>> >>> >>>> >>>>>> I’ve also replied on the jira issue. >>> >>>> >>>>>> >>> >>>> >>>>>> You can try to copy a job to the new Jenkins. >>> >>>> >>>>>> >>> >>>> >>>>>> Thanks. >>> >>>> >>>>>> >>> >>>> >>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： >>> >>>> >>>>>> >>> >>>> >>>>>>> Im taking a look ... >>> >>>> >>>>>>> >>> >>>> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey < >>> [email protected]> >>> >>>> >>>>>>> wrote: >>> >>>> >>>>>>> >>> >>>> >>>>>>> > Hi Gavin >>> >>>> >>>>>>> > >>> >>>> >>>>>>> > I went to start migrating jobs to the ci-hbase controller, >>> >>>> but the >>> >>>> >>>>>>> new job >>> >>>> >>>>>>> > workflow doesn't include pipelines or multibranch >>> pipelines. >>> >>>> We >>> >>>> >>>>>>> use both of >>> >>>> >>>>>>> > these job types for nearly all of our work. Can they be >>> >>>> enabled? >>> >>>> >>>>>>> > >>> >>>> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >>> >>>> >>>>>>> [email protected]> >>> >>>> >>>>>>> > wrote: >>> >>>> >>>>>>> > >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >>> >>>> >>>>>>> >> > Thank you Gavin. >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > We have some urls in our jenkins files which >>> references the >>> >>>> >>>>>>> artifacts of >>> >>>> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not >>> work. >>> >>>> For >>> >>>> >>>>>>> example, >>> >>>> >>>>>>> >> here: >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >>> >>>> >>> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will >>> file a >>> >>>> HBase >>> >>>> >>>>>>> issue >>> >>>> >>>>>>> >> to do >>> >>>> >>>>>>> >> > this. >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > And will report any problems on INFRA_22760 while >>> >>>> migrating. >>> >>>> >>>>>>> When the >>> >>>> >>>>>>> >> > migration is half done, you can move for example, 5 >>> hbase >>> >>>> nodes >>> >>>> >>>>>>> from >>> >>>> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is >>> finally >>> >>>> done, >>> >>>> >>>>>>> you can >>> >>>> >>>>>>> >> move >>> >>>> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to >>> ci-hbase. >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > WDYT? >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >> However you want to do it is fine with me :) >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >> You have 10 nodes on there right now to get you started. >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >> Let me know how you get on >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > Thanks. >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 >>> >>>> 16:16写道： >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> > > Hi All, >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > I created an Infra ticket for tracking. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 >>> in >>> >>>> >>>>>>> ci-hadoop.a.o >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > I have created for you 10 new nodes so far out of our >>> >>>> Hetzner >>> >>>> >>>>>>> DC >>> >>>> >>>>>>> >> > > of a similar specifications to the hbase1-10. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Those are hbase11-20 and are available right now for >>> >>>> testing >>> >>>> >>>>>>> on >>> >>>> >>>>>>> >> > > ci-hbase.apache.org >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > I do not know your workflow well enough to decide the >>> >>>> next >>> >>>> >>>>>>> steps for >>> >>>> >>>>>>> >> you >>> >>>> >>>>>>> >> > > for testing. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Options include: >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto >>> ci-hbase >>> >>>> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of >>> ci-hadoop >>> >>>> onto >>> >>>> >>>>>>> ci-hbase >>> >>>> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs >>> over >>> >>>> for >>> >>>> >>>>>>> testing. >>> >>>> >>>>>>> >> > > - I like copy, however quite a few projects jobs >>> >>>> would >>> >>>> >>>>>>> not be >>> >>>> >>>>>>> >> > > appropriate >>> >>>> >>>>>>> >> > > to have the same job running twice, including >>> >>>> those that >>> >>>> >>>>>>> >> deploy, or >>> >>>> >>>>>>> >> > > those that >>> >>>> >>>>>>> >> > > build from PRs and continue a workflow to >>> report >>> >>>> back, >>> >>>> >>>>>>> or >>> >>>> >>>>>>> >> approve >>> >>>> >>>>>>> >> > > for merge. >>> >>>> >>>>>>> >> > > So for a copy, would need to be a job that just >>> >>>> 'builds >>> >>>> >>>>>>> and >>> >>>> >>>>>>> >> tests' >>> >>>> >>>>>>> >> > > with no outflow. >>> >>>> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org >>> to >>> >>>> test >>> >>>> >>>>>>> the nodes >>> >>>> >>>>>>> >> > > setup and the >>> >>>> >>>>>>> >> > > main controller setup, without affecting current >>> >>>> jobs. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Obviously, you may to tweak version controlled script >>> >>>> files >>> >>>> >>>>>>> such as a >>> >>>> >>>>>>> >> > > JenkinsFile, which makes the copy options harder >>> unless >>> >>>> >>>>>>> testing from a >>> >>>> >>>>>>> >> > > branch created for the purpose. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > I may be overthinking things, and option 1 , the >>> >>>> simplest and >>> >>>> >>>>>>> most >>> >>>> >>>>>>> >> > > straight forward from my end would also suit you guys >>> >>>> and we >>> >>>> >>>>>>> deal >>> >>>> >>>>>>> >> with any >>> >>>> >>>>>>> >> > > breakage as it happens (I'm thinking of integrations >>> with >>> >>>> >>>>>>> other >>> >>>> >>>>>>> >> services, >>> >>>> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed >>> that are >>> >>>> >>>>>>> needed etc.) >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Let me know please ASAP how you want to proceed so we >>> >>>> can get >>> >>>> >>>>>>> using >>> >>>> >>>>>>> >> these >>> >>>> >>>>>>> >> > > nodes, after which I can then proceed to move >>> hbase1-10 >>> >>>> nodes >>> >>>> >>>>>>> over >>> >>>> >>>>>>> >> also. >>> >>>> >>>>>>> >> > > And then start adding more into the mix. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is >>> >>>> >>>>>>> available on >>> >>>> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared >>> and >>> >>>> >>>>>>> leased out to >>> >>>> >>>>>>> >> > > controllers as needed upon request of a job. >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Thanks >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > Gav... >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >>> >>>> >>>>>>> >> > > > Thanks you Gavin. >>> >>>> >>>>>>> >> > > > >>> >>>> >>>>>>> >> > > > You can do it in your convenience, just let me >>> know so >>> >>>> I >>> >>>> >>>>>>> could >>> >>>> >>>>>>> >> change the >>> >>>> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at >>> least >>> >>>> our >>> >>>> >>>>>>> pre >>> >>>> >>>>>>> >> commit job >>> >>>> >>>>>>> >> > > > still uses the Hadoop label. >>> >>>> >>>>>>> >> > > > >>> >>>> >>>>>>> >> > > > And another thing is about the >>> hbase_generate_website >>> >>>> job, >>> >>>> >>>>>>> it needs >>> >>>> >>>>>>> >> to be >>> >>>> >>>>>>> >> > > > executed on the nodes under the git-websites label. >>> >>>> I'm not >>> >>>> >>>>>>> sure if >>> >>>> >>>>>>> >> the >>> >>>> >>>>>>> >> > > > label is still available after the migration. >>> >>>> >>>>>>> >> > > > >>> >>>> >>>>>>> >> > > > Gavin McDonald <[email protected]> >>> 于2021年12月8日周三 >>> >>>> >>>>>>> 17:28写道： >>> >>>> >>>>>>> >> > > > >>> >>>> >>>>>>> >> > > > > Hi HBase devs. >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > This is to let you know that Infra has created a >>> >>>> >>>>>>> dedicated Jenkins >>> >>>> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > You have new incoming donated agents which will >>> get >>> >>>> >>>>>>> connected to >>> >>>> >>>>>>> >> this >>> >>>> >>>>>>> >> > > new >>> >>>> >>>>>>> >> > > > > controller by Infra. >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > You also have 10 or so agents connected to >>> >>>> >>>>>>> >> > > https://ci-hadoop.apache.org - >>> >>>> >>>>>>> >> > > > > I >>> >>>> >>>>>>> >> > > > > would like to move these agents - and all of your >>> >>>> jobs - >>> >>>> >>>>>>> over to >>> >>>> >>>>>>> >> the >>> >>>> >>>>>>> >> > > new >>> >>>> >>>>>>> >> > > > > controller as soon as possible. >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > You may or may not need to reconfigure your jobs >>> and >>> >>>> or >>> >>>> >>>>>>> >> JenkinsFile or >>> >>>> >>>>>>> >> > > > > other. >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > Can I get the go ahead to move these at my >>> >>>> convenience or >>> >>>> >>>>>>> suggest >>> >>>> >>>>>>> >> > > please a >>> >>>> >>>>>>> >> > > > > timeline for Infra to move these agents. >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > Thanks! >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > -- >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > > *Gavin McDonald* >>> >>>> >>>>>>> >> > > > > Systems Administrator >>> >>>> >>>>>>> >> > > > > ASF Infrastructure Team >>> >>>> >>>>>>> >> > > > > >>> >>>> >>>>>>> >> > > > >>> >>>> >>>>>>> >> > > >>> >>>> >>>>>>> >> > >>> >>>> >>>>>>> >> >>> >>>> >>>>>>> > >>> >>>> >>>>>>> >>> >>>> >>>>>>> -- >>> >>>> >>>>>>> >>> >>>> >>>>>>> *Gavin McDonald* >>> >>>> >>>>>>> Systems Administrator >>> >>>> >>>>>>> ASF Infrastructure Team >>> >>>> >>>>>>> >>> >>>> >>>>>> >>> >>>> >>>>> >>> >>>> >>>>> -- >>> >>>> >>>>> >>> >>>> >>>>> *Gavin McDonald* >>> >>>> >>>>> Systems Administrator >>> >>>> >>>>> ASF Infrastructure Team >>> >>>> >>>>> >>> >>>> >>>> >>> >>>> >>> >>>> -- >>> >>>> >>> >>>> *Gavin McDonald* >>> >>>> Systems Administrator >>> >>>> ASF Infrastructure Team >>> >>>> >>> >>> >>> >>> -- >>> >>> *Gavin McDonald* >>> Systems Administrator >>> ASF Infrastructure Team >>> >>
