But on ci-hadoop we could still use it? You mean that due to a CVE, we can not install the SSH plugin any more because the download source is unavailable?
Then I do not think we could move forward for now as we highly depend on uploading artifacts to nightlies. Let me revert the changes for now. Please respond here if the sshTransfer method is available on ci-hbase, then we could start the migration again. Thanks Gavin. Gavin McDonald <[email protected]> 于2022年1月22日周六 17:57写道： > As of a few days ago. the Publish over SSH plugin was made unavailable due > to a CVE. > Looking into it but hopefully a simple fix that will be rolled out soon > > On Sat, Jan 22, 2022 at 10:06 AM 张铎(Duo Zhang) <[email protected]> > wrote: > > > Ah, another problem... > > > > java.lang.NoSuchMethodError: No such DSL method 'sshTransfer' found > among steps > > > > > > Seems on the new jenkins we do not have the support for transfering data > > to nightlies... > > > > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 17:04写道： > > > >> Thank you Gavin. The builds are started now. > >> > >> Let's wait for the result. > >> > >> And on installing plugins, I haven't seen any system management related > >> button on the https://ci-hbase.apache.org/ page... > >> On the https://jenkins-ccos.apache.org/job/masters/job/HBase/ page, > >> click the 'Config Files' button, there is a 'Manage Jenkins' button but > I > >> have no permission to click it... > >> > >> Another thing is hbase20 is offline because of readonly filesystem. > PTAL. > >> > >> Thanks a lot. > >> > >> Gavin McDonald <[email protected]> 于2022年1月22日周六 16:42写道： > >> > >>> Hi, > >>> Timestamper plugin installed. > >>> > >>> By the way I thought I had given the PMC members permissions to manage > >>> Jenkins > >>> on ci-hbase including installing plugins etc, can someone check? > >>> > >>> > >>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) <[email protected]> > >>> wrote: > >>> > >>>> We faced a strange issue.. > >>>> > >>>> rg.codehaus.groovy.control.MultipleCompilationErrorsException: > startup failed: > >>>> WorkflowScript: 29: Invalid option type "timestamps". Valid option > types: [buildDiscarder, catchError, checkoutToSubdirectory, > disableConcurrentBuilds, disableResume, durabilityHint, > githubProjectProperty, newContainerPerStage, overrideIndexTriggers, > parallelsAlwaysFailFast, preserveStashes, quietPeriod, rateLimitBuilds, > retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, > waitUntil, warnError, withChecks, withContext, withCredentials, withEnv, > wrap, ws] @ line 29, column 5. > >>>> timestamps() > >>>> ^ > >>>> > >>>> 1 error > >>>> > >>>> at > org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) > >>>> at > org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) > >>>> at > org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) > >>>> at > org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) > >>>> at > org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) > >>>> at > groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) > >>>> at > groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) > >>>> at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) > >>>> at groovy.lang.GroovyShell.parse(GroovyShell.java:700) > >>>> at > org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) > >>>> at > org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) > >>>> at > org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) > >>>> at > org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) > >>>> at > org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) > >>>> at > hudson.model.ResourceController.execute(ResourceController.java:97) > >>>> at hudson.model.Executor.run(Executor.java:429) > >>>> > >>>> > >>>> After googling, it seems a restart of jenkins can fix it, or we do not > >>>> have timestamper plugin installed? Mind taking a look Gavin? > >>>> > >>>> Thanks. > >>>> > >>>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： > >>>> > >>>>> Oh, just saw that we could do this by ourselves. > >>>>> > >>>>> Let me have a try. > >>>>> > >>>>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： > >>>>> > >>>>>> OK, so while copying you could keep the old jenkins builds? > >>>>>> That's awesome. > >>>>>> > >>>>>> Then please help 'copying' HBase-Find-Flaky-Tests > >>>>>> and HBase-Flaky-Tests to ci-hbase too? We need the history builds to > >>>>>> calucate the flaky test list so if all the old build data can be > kept then > >>>>>> it will help us a lot. > >>>>>> > >>>>>> Thanks~ > >>>>>> > >>>>>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： > >>>>>> > >>>>>>> Thank you Gavin. > >>>>>>> > >>>>>>> Let me first move some light weighted job first to see if > everything > >>>>>>> is OK. > >>>>>>> > >>>>>>> HBase Nightly, some flaky related jobs, and also the main pre > commit > >>>>>>> job for HBase are tied together. We need to move them at once. > >>>>>>> > >>>>>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： > >>>>>>> > >>>>>>>> Thanks for your patience > >>>>>>>> > >>>>>>>> I have 'copied' over a job - > >>>>>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ > >>>>>>>> > >>>>>>>> which of course made me aware of what plugins I needed to install. > >>>>>>>> > >>>>>>>> Feel free to delete that, or use it and delete the old, and let me > >>>>>>>> know if have any issues > >>>>>>>> with any other missing plugins. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < > >>>>>>>> [email protected]> wrote: > >>>>>>>> > >>>>>>>>> I’ve also replied on the jira issue. > >>>>>>>>> > >>>>>>>>> You can try to copy a job to the new Jenkins. > >>>>>>>>> > >>>>>>>>> Thanks. > >>>>>>>>> > >>>>>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： > >>>>>>>>> > >>>>>>>>>> Im taking a look ... > >>>>>>>>>> > >>>>>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey <[email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> > Hi Gavin > >>>>>>>>>> > > >>>>>>>>>> > I went to start migrating jobs to the ci-hbase controller, but > >>>>>>>>>> the new job > >>>>>>>>>> > workflow doesn't include pipelines or multibranch pipelines. > We > >>>>>>>>>> use both of > >>>>>>>>>> > these job types for nearly all of our work. Can they be > enabled? > >>>>>>>>>> > > >>>>>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < > >>>>>>>>>> [email protected]> > >>>>>>>>>> > wrote: > >>>>>>>>>> > > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: > >>>>>>>>>> >> > Thank you Gavin. > >>>>>>>>>> >> > > >>>>>>>>>> >> > We have some urls in our jenkins files which references the > >>>>>>>>>> artifacts of > >>>>>>>>>> >> > the jenkins jobs, so a simple copy or move can not work. > For > >>>>>>>>>> example, > >>>>>>>>>> >> here: > >>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>>> >> > >>>>>>>>>> > https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 > >>>>>>>>>> >> > > >>>>>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will file a > >>>>>>>>>> HBase issue > >>>>>>>>>> >> to do > >>>>>>>>>> >> > this. > >>>>>>>>>> >> > > >>>>>>>>>> >> > And will report any problems on INFRA_22760 while > migrating. > >>>>>>>>>> When the > >>>>>>>>>> >> > migration is half done, you can move for example, 5 hbase > >>>>>>>>>> nodes from > >>>>>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is finally > >>>>>>>>>> done, you can > >>>>>>>>>> >> move > >>>>>>>>>> >> > all the remaining hbase nodes from ci-hadoop to ci-hbase. > >>>>>>>>>> >> > > >>>>>>>>>> >> > WDYT? > >>>>>>>>>> >> > >>>>>>>>>> >> However you want to do it is fine with me :) > >>>>>>>>>> >> > >>>>>>>>>> >> You have 10 nodes on there right now to get you started. > >>>>>>>>>> >> > >>>>>>>>>> >> Let me know how you get on > >>>>>>>>>> >> > >>>>>>>>>> >> > > >>>>>>>>>> >> > Thanks. > >>>>>>>>>> >> > > >>>>>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 > 16:16写道： > >>>>>>>>>> >> > > >>>>>>>>>> >> > > Hi All, > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > I created an Infra ticket for tracking. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 in > >>>>>>>>>> ci-hadoop.a.o > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > I have created for you 10 new nodes so far out of our > >>>>>>>>>> Hetzner DC > >>>>>>>>>> >> > > of a similar specifications to the hbase1-10. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Those are hbase11-20 and are available right now for > >>>>>>>>>> testing on > >>>>>>>>>> >> > > ci-hbase.apache.org > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > I do not know your workflow well enough to decide the > next > >>>>>>>>>> steps for > >>>>>>>>>> >> you > >>>>>>>>>> >> > > for testing. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Options include: > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto ci-hbase > >>>>>>>>>> >> > > 2. I can 'move' one or a number of jobs out of ci-hadoop > >>>>>>>>>> onto ci-hbase > >>>>>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs over > >>>>>>>>>> for testing. > >>>>>>>>>> >> > > - I like copy, however quite a few projects jobs > would > >>>>>>>>>> not be > >>>>>>>>>> >> > > appropriate > >>>>>>>>>> >> > > to have the same job running twice, including those > >>>>>>>>>> that > >>>>>>>>>> >> deploy, or > >>>>>>>>>> >> > > those that > >>>>>>>>>> >> > > build from PRs and continue a workflow to report > >>>>>>>>>> back, or > >>>>>>>>>> >> approve > >>>>>>>>>> >> > > for merge. > >>>>>>>>>> >> > > So for a copy, would need to be a job that just > >>>>>>>>>> 'builds and > >>>>>>>>>> >> tests' > >>>>>>>>>> >> > > with no outflow. > >>>>>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org to > >>>>>>>>>> test the nodes > >>>>>>>>>> >> > > setup and the > >>>>>>>>>> >> > > main controller setup, without affecting current > jobs. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Obviously, you may to tweak version controlled script > >>>>>>>>>> files such as a > >>>>>>>>>> >> > > JenkinsFile, which makes the copy options harder unless > >>>>>>>>>> testing from a > >>>>>>>>>> >> > > branch created for the purpose. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > I may be overthinking things, and option 1 , the simplest > >>>>>>>>>> and most > >>>>>>>>>> >> > > straight forward from my end would also suit you guys and > >>>>>>>>>> we deal > >>>>>>>>>> >> with any > >>>>>>>>>> >> > > breakage as it happens (I'm thinking of integrations with > >>>>>>>>>> other > >>>>>>>>>> >> services, > >>>>>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed that are > >>>>>>>>>> needed etc.) > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Let me know please ASAP how you want to proceed so we can > >>>>>>>>>> get using > >>>>>>>>>> >> these > >>>>>>>>>> >> > > nodes, after which I can then proceed to move hbase1-10 > >>>>>>>>>> nodes over > >>>>>>>>>> >> also. > >>>>>>>>>> >> > > And then start adding more into the mix. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is > >>>>>>>>>> available on > >>>>>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared and > >>>>>>>>>> leased out to > >>>>>>>>>> >> > > controllers as needed upon request of a job. > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Thanks > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > Gav... > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: > >>>>>>>>>> >> > > > Thanks you Gavin. > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > You can do it in your convenience, just let me know so > I > >>>>>>>>>> could > >>>>>>>>>> >> change the > >>>>>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at least > >>>>>>>>>> our pre > >>>>>>>>>> >> commit job > >>>>>>>>>> >> > > > still uses the Hadoop label. > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > And another thing is about the hbase_generate_website > >>>>>>>>>> job, it needs > >>>>>>>>>> >> to be > >>>>>>>>>> >> > > > executed on the nodes under the git-websites label. I'm > >>>>>>>>>> not sure if > >>>>>>>>>> >> the > >>>>>>>>>> >> > > > label is still available after the migration. > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > Gavin McDonald <[email protected]> 于2021年12月8日周三 > >>>>>>>>>> 17:28写道： > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > > Hi HBase devs. > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > This is to let you know that Infra has created a > >>>>>>>>>> dedicated Jenkins > >>>>>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > You have new incoming donated agents which will get > >>>>>>>>>> connected to > >>>>>>>>>> >> this > >>>>>>>>>> >> > > new > >>>>>>>>>> >> > > > > controller by Infra. > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > You also have 10 or so agents connected to > >>>>>>>>>> >> > > https://ci-hadoop.apache.org - > >>>>>>>>>> >> > > > > I > >>>>>>>>>> >> > > > > would like to move these agents - and all of your > jobs > >>>>>>>>>> - over to > >>>>>>>>>> >> the > >>>>>>>>>> >> > > new > >>>>>>>>>> >> > > > > controller as soon as possible. > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > You may or may not need to reconfigure your jobs and > or > >>>>>>>>>> >> JenkinsFile or > >>>>>>>>>> >> > > > > other. > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > Can I get the go ahead to move these at my > convenience > >>>>>>>>>> or suggest > >>>>>>>>>> >> > > please a > >>>>>>>>>> >> > > > > timeline for Infra to move these agents. > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > Thanks! > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > -- > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > *Gavin McDonald* > >>>>>>>>>> >> > > > > Systems Administrator > >>>>>>>>>> >> > > > > ASF Infrastructure Team > >>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > >>>>>>>>>> >> > > >>>>>>>>>> >> > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> > >>>>>>>>>> *Gavin McDonald* > >>>>>>>>>> Systems Administrator > >>>>>>>>>> ASF Infrastructure Team > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> > >>>>>>>> *Gavin McDonald* > >>>>>>>> Systems Administrator > >>>>>>>> ASF Infrastructure Team > >>>>>>>> > >>>>>>> > >>> > >>> -- > >>> > >>> *Gavin McDonald* > >>> Systems Administrator > >>> ASF Infrastructure Team > >>> > >> > > -- > > *Gavin McDonald* > Systems Administrator > ASF Infrastructure Team >
