On Sat, Jan 22, 2022 at 12:33 PM 张铎(Duo Zhang) <[email protected]> wrote:
> But on ci-hadoop we could still use it? > yes, the CVE mentions one needs overall administer permissions to exploit it, so Im not going to remove it. You mean that due to a CVE, we can not install the SSH plugin any more > because the download source is unavailable? > Correct. > Then I do not think we could move forward for now as we highly depend on > uploading artifacts to nightlies. > > Let me revert the changes for now. Please respond here if the sshTransfer > method is available on ci-hbase, then we could start the migration again. > Ok will do. I am exploring to see if we can get this fixed sooner rather than later > Thanks Gavin. > > Gavin McDonald <[email protected]> 于2022年1月22日周六 17:57写道： > >> As of a few days ago. the Publish over SSH plugin was made unavailable >> due >> to a CVE. >> Looking into it but hopefully a simple fix that will be rolled out soon >> >> On Sat, Jan 22, 2022 at 10:06 AM 张铎(Duo Zhang) <[email protected]> >> wrote: >> >> > Ah, another problem... >> > >> > java.lang.NoSuchMethodError: No such DSL method 'sshTransfer' found >> among steps >> > >> > >> > Seems on the new jenkins we do not have the support for transfering data >> > to nightlies... >> > >> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 17:04写道： >> > >> >> Thank you Gavin. The builds are started now. >> >> >> >> Let's wait for the result. >> >> >> >> And on installing plugins, I haven't seen any system management related >> >> button on the https://ci-hbase.apache.org/ page... >> >> On the https://jenkins-ccos.apache.org/job/masters/job/HBase/ page, >> >> click the 'Config Files' button, there is a 'Manage Jenkins' button >> but I >> >> have no permission to click it... >> >> >> >> Another thing is hbase20 is offline because of readonly filesystem. >> PTAL. >> >> >> >> Thanks a lot. >> >> >> >> Gavin McDonald <[email protected]> 于2022年1月22日周六 16:42写道： >> >> >> >>> Hi, >> >>> Timestamper plugin installed. >> >>> >> >>> By the way I thought I had given the PMC members permissions to manage >> >>> Jenkins >> >>> on ci-hbase including installing plugins etc, can someone check? >> >>> >> >>> >> >>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) <[email protected]> >> >>> wrote: >> >>> >> >>>> We faced a strange issue.. >> >>>> >> >>>> rg.codehaus.groovy.control.MultipleCompilationErrorsException: >> startup failed: >> >>>> WorkflowScript: 29: Invalid option type "timestamps". Valid option >> types: [buildDiscarder, catchError, checkoutToSubdirectory, >> disableConcurrentBuilds, disableResume, durabilityHint, >> githubProjectProperty, newContainerPerStage, overrideIndexTriggers, >> parallelsAlwaysFailFast, preserveStashes, quietPeriod, rateLimitBuilds, >> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, >> waitUntil, warnError, withChecks, withContext, withCredentials, withEnv, >> wrap, ws] @ line 29, column 5. >> >>>> timestamps() >> >>>> ^ >> >>>> >> >>>> 1 error >> >>>> >> >>>> at >> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >> >>>> at >> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >> >>>> at >> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >> >>>> at >> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >> >>>> at >> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >> >>>> at >> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >> >>>> at >> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >> >>>> at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >> >>>> at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >> >>>> at >> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >> >>>> at >> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >> >>>> at >> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >> >>>> at >> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >> >>>> at >> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >> >>>> at >> hudson.model.ResourceController.execute(ResourceController.java:97) >> >>>> at hudson.model.Executor.run(Executor.java:429) >> >>>> >> >>>> >> >>>> After googling, it seems a restart of jenkins can fix it, or we do >> not >> >>>> have timestamper plugin installed? Mind taking a look Gavin? >> >>>> >> >>>> Thanks. >> >>>> >> >>>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >> >>>> >> >>>>> Oh, just saw that we could do this by ourselves. >> >>>>> >> >>>>> Let me have a try. >> >>>>> >> >>>>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >> >>>>> >> >>>>>> OK, so while copying you could keep the old jenkins builds? >> >>>>>> That's awesome. >> >>>>>> >> >>>>>> Then please help 'copying' HBase-Find-Flaky-Tests >> >>>>>> and HBase-Flaky-Tests to ci-hbase too? We need the history builds >> to >> >>>>>> calucate the flaky test list so if all the old build data can be >> kept then >> >>>>>> it will help us a lot. >> >>>>>> >> >>>>>> Thanks~ >> >>>>>> >> >>>>>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >> >>>>>> >> >>>>>>> Thank you Gavin. >> >>>>>>> >> >>>>>>> Let me first move some light weighted job first to see if >> everything >> >>>>>>> is OK. >> >>>>>>> >> >>>>>>> HBase Nightly, some flaky related jobs, and also the main pre >> commit >> >>>>>>> job for HBase are tied together. We need to move them at once. >> >>>>>>> >> >>>>>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >> >>>>>>> >> >>>>>>>> Thanks for your patience >> >>>>>>>> >> >>>>>>>> I have 'copied' over a job - >> >>>>>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >> >>>>>>>> >> >>>>>>>> which of course made me aware of what plugins I needed to >> install. >> >>>>>>>> >> >>>>>>>> Feel free to delete that, or use it and delete the old, and let >> me >> >>>>>>>> know if have any issues >> >>>>>>>> with any other missing plugins. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >> >>>>>>>> [email protected]> wrote: >> >>>>>>>> >> >>>>>>>>> I’ve also replied on the jira issue. >> >>>>>>>>> >> >>>>>>>>> You can try to copy a job to the new Jenkins. >> >>>>>>>>> >> >>>>>>>>> Thanks. >> >>>>>>>>> >> >>>>>>>>> Gavin McDonald <[email protected]>于2022年1月20日 周四18:45写道： >> >>>>>>>>> >> >>>>>>>>>> Im taking a look ... >> >>>>>>>>>> >> >>>>>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey <[email protected] >> > >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>> > Hi Gavin >> >>>>>>>>>> > >> >>>>>>>>>> > I went to start migrating jobs to the ci-hbase controller, >> but >> >>>>>>>>>> the new job >> >>>>>>>>>> > workflow doesn't include pipelines or multibranch pipelines. >> We >> >>>>>>>>>> use both of >> >>>>>>>>>> > these job types for nearly all of our work. Can they be >> enabled? >> >>>>>>>>>> > >> >>>>>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >> >>>>>>>>>> [email protected]> >> >>>>>>>>>> > wrote: >> >>>>>>>>>> > >> >>>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >> >>>>>>>>>> >> > Thank you Gavin. >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > We have some urls in our jenkins files which references >> the >> >>>>>>>>>> artifacts of >> >>>>>>>>>> >> > the jenkins jobs, so a simple copy or move can not work. >> For >> >>>>>>>>>> example, >> >>>>>>>>>> >> here: >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> >> >> >>>>>>>>>> >> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will file a >> >>>>>>>>>> HBase issue >> >>>>>>>>>> >> to do >> >>>>>>>>>> >> > this. >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > And will report any problems on INFRA_22760 while >> migrating. >> >>>>>>>>>> When the >> >>>>>>>>>> >> > migration is half done, you can move for example, 5 hbase >> >>>>>>>>>> nodes from >> >>>>>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is finally >> >>>>>>>>>> done, you can >> >>>>>>>>>> >> move >> >>>>>>>>>> >> > all the remaining hbase nodes from ci-hadoop to ci-hbase. >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > WDYT? >> >>>>>>>>>> >> >> >>>>>>>>>> >> However you want to do it is fine with me :) >> >>>>>>>>>> >> >> >>>>>>>>>> >> You have 10 nodes on there right now to get you started. >> >>>>>>>>>> >> >> >>>>>>>>>> >> Let me know how you get on >> >>>>>>>>>> >> >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > Thanks. >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 >> 16:16写道： >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > > Hi All, >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > I created an Infra ticket for tracking. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > For now, I left your current nodes hbase1 to hbase10 in >> >>>>>>>>>> ci-hadoop.a.o >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > I have created for you 10 new nodes so far out of our >> >>>>>>>>>> Hetzner DC >> >>>>>>>>>> >> > > of a similar specifications to the hbase1-10. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Those are hbase11-20 and are available right now for >> >>>>>>>>>> testing on >> >>>>>>>>>> >> > > ci-hbase.apache.org >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > I do not know your workflow well enough to decide the >> next >> >>>>>>>>>> steps for >> >>>>>>>>>> >> you >> >>>>>>>>>> >> > > for testing. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Options include: >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto ci-hbase >> >>>>>>>>>> >> > > 2. I can 'move' one or a number of jobs out of ci-hadoop >> >>>>>>>>>> onto ci-hbase >> >>>>>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of jobs over >> >>>>>>>>>> for testing. >> >>>>>>>>>> >> > > - I like copy, however quite a few projects jobs >> would >> >>>>>>>>>> not be >> >>>>>>>>>> >> > > appropriate >> >>>>>>>>>> >> > > to have the same job running twice, including >> those >> >>>>>>>>>> that >> >>>>>>>>>> >> deploy, or >> >>>>>>>>>> >> > > those that >> >>>>>>>>>> >> > > build from PRs and continue a workflow to report >> >>>>>>>>>> back, or >> >>>>>>>>>> >> approve >> >>>>>>>>>> >> > > for merge. >> >>>>>>>>>> >> > > So for a copy, would need to be a job that just >> >>>>>>>>>> 'builds and >> >>>>>>>>>> >> tests' >> >>>>>>>>>> >> > > with no outflow. >> >>>>>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org to >> >>>>>>>>>> test the nodes >> >>>>>>>>>> >> > > setup and the >> >>>>>>>>>> >> > > main controller setup, without affecting current >> jobs. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Obviously, you may to tweak version controlled script >> >>>>>>>>>> files such as a >> >>>>>>>>>> >> > > JenkinsFile, which makes the copy options harder unless >> >>>>>>>>>> testing from a >> >>>>>>>>>> >> > > branch created for the purpose. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > I may be overthinking things, and option 1 , the >> simplest >> >>>>>>>>>> and most >> >>>>>>>>>> >> > > straight forward from my end would also suit you guys >> and >> >>>>>>>>>> we deal >> >>>>>>>>>> >> with any >> >>>>>>>>>> >> > > breakage as it happens (I'm thinking of integrations >> with >> >>>>>>>>>> other >> >>>>>>>>>> >> services, >> >>>>>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed that >> are >> >>>>>>>>>> needed etc.) >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Let me know please ASAP how you want to proceed so we >> can >> >>>>>>>>>> get using >> >>>>>>>>>> >> these >> >>>>>>>>>> >> > > nodes, after which I can then proceed to move hbase1-10 >> >>>>>>>>>> nodes over >> >>>>>>>>>> >> also. >> >>>>>>>>>> >> > > And then start adding more into the mix. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It is >> >>>>>>>>>> available on >> >>>>>>>>>> >> > > ci-hbase.a.o as the nodes with that label are shared and >> >>>>>>>>>> leased out to >> >>>>>>>>>> >> > > controllers as needed upon request of a job. >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Thanks >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > Gav... >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >> >>>>>>>>>> >> > > > Thanks you Gavin. >> >>>>>>>>>> >> > > > >> >>>>>>>>>> >> > > > You can do it in your convenience, just let me know >> so I >> >>>>>>>>>> could >> >>>>>>>>>> >> change the >> >>>>>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at least >> >>>>>>>>>> our pre >> >>>>>>>>>> >> commit job >> >>>>>>>>>> >> > > > still uses the Hadoop label. >> >>>>>>>>>> >> > > > >> >>>>>>>>>> >> > > > And another thing is about the hbase_generate_website >> >>>>>>>>>> job, it needs >> >>>>>>>>>> >> to be >> >>>>>>>>>> >> > > > executed on the nodes under the git-websites label. >> I'm >> >>>>>>>>>> not sure if >> >>>>>>>>>> >> the >> >>>>>>>>>> >> > > > label is still available after the migration. >> >>>>>>>>>> >> > > > >> >>>>>>>>>> >> > > > Gavin McDonald <[email protected]> 于2021年12月8日周三 >> >>>>>>>>>> 17:28写道： >> >>>>>>>>>> >> > > > >> >>>>>>>>>> >> > > > > Hi HBase devs. >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > This is to let you know that Infra has created a >> >>>>>>>>>> dedicated Jenkins >> >>>>>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > You have new incoming donated agents which will get >> >>>>>>>>>> connected to >> >>>>>>>>>> >> this >> >>>>>>>>>> >> > > new >> >>>>>>>>>> >> > > > > controller by Infra. >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > You also have 10 or so agents connected to >> >>>>>>>>>> >> > > https://ci-hadoop.apache.org - >> >>>>>>>>>> >> > > > > I >> >>>>>>>>>> >> > > > > would like to move these agents - and all of your >> jobs >> >>>>>>>>>> - over to >> >>>>>>>>>> >> the >> >>>>>>>>>> >> > > new >> >>>>>>>>>> >> > > > > controller as soon as possible. >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > You may or may not need to reconfigure your jobs >> and or >> >>>>>>>>>> >> JenkinsFile or >> >>>>>>>>>> >> > > > > other. >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > Can I get the go ahead to move these at my >> convenience >> >>>>>>>>>> or suggest >> >>>>>>>>>> >> > > please a >> >>>>>>>>>> >> > > > > timeline for Infra to move these agents. >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > Thanks! >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > -- >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > > *Gavin McDonald* >> >>>>>>>>>> >> > > > > Systems Administrator >> >>>>>>>>>> >> > > > > ASF Infrastructure Team >> >>>>>>>>>> >> > > > > >> >>>>>>>>>> >> > > > >> >>>>>>>>>> >> > > >> >>>>>>>>>> >> > >> >>>>>>>>>> >> >> >>>>>>>>>> > >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> >> >>>>>>>>>> *Gavin McDonald* >> >>>>>>>>>> Systems Administrator >> >>>>>>>>>> ASF Infrastructure Team >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> >> >>>>>>>> *Gavin McDonald* >> >>>>>>>> Systems Administrator >> >>>>>>>> ASF Infrastructure Team >> >>>>>>>> >> >>>>>>> >> >>> >> >>> -- >> >>> >> >>> *Gavin McDonald* >> >>> Systems Administrator >> >>> ASF Infrastructure Team >> >>> >> >> >> >> -- >> >> *Gavin McDonald* >> Systems Administrator >> ASF Infrastructure Team >> > -- *Gavin McDonald* Systems Administrator ASF Infrastructure Team
