Most of the work is done. There is still an arm jenkins job on ci-hadoop, as we do not have arm nodes on ci-hbase.
Another problem is that it seems the github web hook will not notify ci-hbase, so we need to wait for the pre-commit job to scan the github repo for triggering a build. Will keep an eye on it and work together with infra to solve it if possible. Thanks. 张铎(Duo Zhang) <[email protected]> 于2022年2月10日周四 11:16写道： > Some updates. > > For nightly build we are still suffering from the python problem... > Anyway, using python2 directly can solve the problem. > > I'm still working on it. > > Thanks. > > 张铎(Duo Zhang) <[email protected]> 于2022年2月9日周三 09:34写道： > >> The new Jenkins jobs are still unavailable due to lack of python support. >> >> If this still can not be resolved tonight, I will rollback to ci-hadoop >> and try migration next time. >> >> Thanks. >> >> 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 21:35写道： >> >>> Hi Gavin, the ssh transfer plugin is fine, but now we miss python on >>> hbase11-20. >>> >>> I've filed INFRA-22855, PTAL. >>> >>> Thanks. >>> >>> 张铎(Duo Zhang) <[email protected]> 于2022年2月8日周二 20:19写道： >>> >>>> Thank you Gavin. WIll take a look soon! >>>> >>>> Gavin McDonald <[email protected]> 于2022年2月8日周二 18:31写道： >>>> >>>>> Give it a try now please >>>>> >>>>> On Sat, Feb 5, 2022 at 8:55 AM 张铎(Duo Zhang) <[email protected]> >>>>> wrote: >>>>> >>>>> > Hi, Gavin, any luck on the ssh transfer plugin? >>>>> > >>>>> > Thanks. >>>>> > >>>>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月29日周六 14:53写道： >>>>> > >>>>> >> So any updates here Gavin? >>>>> >> >>>>> >> Is the ssh transfer plugin available now? >>>>> >> >>>>> >> Thanks. >>>>> >> >>>>> >> Sean Busbey <[email protected]> 于2022年1月23日周日 09:04写道： >>>>> >> >>>>> >>> It looks like I currently do have access to manage the ci-hbase >>>>> >>> controller. >>>>> >>> >>>>> >>> >>>>> >>> Could you check that the git-websites label handling where a common >>>>> >>> machine gets leased to the ci-hbase controller is set up properly? >>>>> or give >>>>> >>> me a pointer on how I would check this myself? >>>>> >>> >>>>> >>> I started trying to migrate our website generation job again and >>>>> it's >>>>> >>> been waiting for a node with the git-websites label to show up for >>>>> ~8 hours >>>>> >>> at this point. >>>>> >>> >>>>> >>> https://ci-hbase.apache.org/job/hbase_generate_website/3/ >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald < >>>>> [email protected]> >>>>> >>> wrote: >>>>> >>> >>>>> >>>> Hi, >>>>> >>>> Timestamper plugin installed. >>>>> >>>> >>>>> >>>> By the way I thought I had given the PMC members permissions to >>>>> manage >>>>> >>>> Jenkins >>>>> >>>> on ci-hbase including installing plugins etc, can someone check? >>>>> >>>> >>>>> >>>> >>>>> >>>> On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) < >>>>> [email protected]> >>>>> >>>> wrote: >>>>> >>>> >>>>> >>>> > We faced a strange issue.. >>>>> >>>> > >>>>> >>>> > rg.codehaus.groovy.control.MultipleCompilationErrorsException: >>>>> >>>> startup failed: >>>>> >>>> > WorkflowScript: 29: Invalid option type "timestamps". Valid >>>>> option >>>>> >>>> types: [buildDiscarder, catchError, checkoutToSubdirectory, >>>>> >>>> disableConcurrentBuilds, disableResume, durabilityHint, >>>>> >>>> githubProjectProperty, newContainerPerStage, >>>>> overrideIndexTriggers, >>>>> >>>> parallelsAlwaysFailFast, preserveStashes, quietPeriod, >>>>> rateLimitBuilds, >>>>> >>>> retry, script, skipDefaultCheckout, skipStagesAfterUnstable, >>>>> timeout, >>>>> >>>> waitUntil, warnError, withChecks, withContext, withCredentials, >>>>> withEnv, >>>>> >>>> wrap, ws] @ line 29, column 5. >>>>> >>>> > timestamps() >>>>> >>>> > ^ >>>>> >>>> > >>>>> >>>> > 1 error >>>>> >>>> > >>>>> >>>> > at >>>>> >>>> >>>>> org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) >>>>> >>>> > at >>>>> >>>> >>>>> org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) >>>>> >>>> > at >>>>> >>>> >>>>> org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) >>>>> >>>> > at >>>>> >>>> >>>>> org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) >>>>> >>>> > at >>>>> >>>> >>>>> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) >>>>> >>>> > at >>>>> >>>> >>>>> groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) >>>>> >>>> > at >>>>> >>>> >>>>> groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) >>>>> >>>> > at >>>>> groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) >>>>> >>>> > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) >>>>> >>>> > at >>>>> >>>> >>>>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) >>>>> >>>> > at >>>>> >>>> >>>>> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) >>>>> >>>> > at >>>>> >>>> >>>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571) >>>>> >>>> > at >>>>> >>>> >>>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523) >>>>> >>>> > at >>>>> >>>> >>>>> org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334) >>>>> >>>> > at >>>>> >>>> >>>>> hudson.model.ResourceController.execute(ResourceController.java:97) >>>>> >>>> > at hudson.model.Executor.run(Executor.java:429) >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > After googling, it seems a restart of jenkins can fix it, or we >>>>> do not >>>>> >>>> > have timestamper plugin installed? Mind taking a look Gavin? >>>>> >>>> > >>>>> >>>> > Thanks. >>>>> >>>> > >>>>> >>>> > 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:40写道： >>>>> >>>> > >>>>> >>>> >> Oh, just saw that we could do this by ourselves. >>>>> >>>> >> >>>>> >>>> >> Let me have a try. >>>>> >>>> >> >>>>> >>>> >> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:18写道： >>>>> >>>> >> >>>>> >>>> >>> OK, so while copying you could keep the old jenkins builds? >>>>> >>>> >>> That's awesome. >>>>> >>>> >>> >>>>> >>>> >>> Then please help 'copying' HBase-Find-Flaky-Tests and >>>>> >>>> HBase-Flaky-Tests >>>>> >>>> >>> to ci-hbase too? We need the history builds to calucate the >>>>> flaky >>>>> >>>> test list >>>>> >>>> >>> so if all the old build data can be kept then it will help us >>>>> a lot. >>>>> >>>> >>> >>>>> >>>> >>> Thanks~ >>>>> >>>> >>> >>>>> >>>> >>> 张铎(Duo Zhang) <[email protected]> 于2022年1月22日周六 11:03写道： >>>>> >>>> >>> >>>>> >>>> >>>> Thank you Gavin. >>>>> >>>> >>>> >>>>> >>>> >>>> Let me first move some light weighted job first to see if >>>>> >>>> everything is >>>>> >>>> >>>> OK. >>>>> >>>> >>>> >>>>> >>>> >>>> HBase Nightly, some flaky related jobs, and also the main pre >>>>> >>>> commit >>>>> >>>> >>>> job for HBase are tied together. We need to move them at >>>>> once. >>>>> >>>> >>>> >>>>> >>>> >>>> Gavin McDonald <[email protected]> 于2022年1月22日周六 02:44写道： >>>>> >>>> >>>> >>>>> >>>> >>>>> Thanks for your patience >>>>> >>>> >>>>> >>>>> >>>> >>>>> I have 'copied' over a job - >>>>> >>>> >>>>> https://ci-hbase.apache.org/job/HBase%20Nightly/ >>>>> >>>> >>>>> >>>>> >>>> >>>>> which of course made me aware of what plugins I needed to >>>>> install. >>>>> >>>> >>>>> >>>>> >>>> >>>>> Feel free to delete that, or use it and delete the old, and >>>>> let me >>>>> >>>> >>>>> know if have any issues >>>>> >>>> >>>>> with any other missing plugins. >>>>> >>>> >>>>> >>>>> >>>> >>>>> >>>>> >>>> >>>>> >>>>> >>>> >>>>> On Thu, Jan 20, 2022 at 1:11 PM 张铎(Duo Zhang) < >>>>> >>>> [email protected]> >>>>> >>>> >>>>> wrote: >>>>> >>>> >>>>> >>>>> >>>> >>>>>> I’ve also replied on the jira issue. >>>>> >>>> >>>>>> >>>>> >>>> >>>>>> You can try to copy a job to the new Jenkins. >>>>> >>>> >>>>>> >>>>> >>>> >>>>>> Thanks. >>>>> >>>> >>>>>> >>>>> >>>> >>>>>> Gavin McDonald <[email protected]>于2022年1月20日 >>>>> 周四18:45写道： >>>>> >>>> >>>>>> >>>>> >>>> >>>>>>> Im taking a look ... >>>>> >>>> >>>>>>> >>>>> >>>> >>>>>>> On Thu, Jan 20, 2022 at 6:54 AM Sean Busbey < >>>>> [email protected]> >>>>> >>>> >>>>>>> wrote: >>>>> >>>> >>>>>>> >>>>> >>>> >>>>>>> > Hi Gavin >>>>> >>>> >>>>>>> > >>>>> >>>> >>>>>>> > I went to start migrating jobs to the ci-hbase >>>>> controller, >>>>> >>>> but the >>>>> >>>> >>>>>>> new job >>>>> >>>> >>>>>>> > workflow doesn't include pipelines or multibranch >>>>> pipelines. >>>>> >>>> We >>>>> >>>> >>>>>>> use both of >>>>> >>>> >>>>>>> > these job types for nearly all of our work. Can they be >>>>> >>>> enabled? >>>>> >>>> >>>>>>> > >>>>> >>>> >>>>>>> > On Tue, Jan 18, 2022 at 5:04 AM Gavin McDonald < >>>>> >>>> >>>>>>> [email protected]> >>>>> >>>> >>>>>>> > wrote: >>>>> >>>> >>>>>>> > >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >> On 2022/01/18 09:20:24 "张铎(Duo Zhang)" wrote: >>>>> >>>> >>>>>>> >> > Thank you Gavin. >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > We have some urls in our jenkins files which >>>>> references the >>>>> >>>> >>>>>>> artifacts of >>>>> >>>> >>>>>>> >> > the jenkins jobs, so a simple copy or move can not >>>>> work. >>>>> >>>> For >>>>> >>>> >>>>>>> example, >>>>> >>>> >>>>>>> >> here: >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >>>>> >>>> >>>>> https://github.com/apache/hbase/blob/c9bcd87b34a15d200a55ec7fdc2b1d86e3367a8c/dev-support/Jenkinsfile_GitHub#L47 >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > So I prefer we migrate the jobs by ourselves. Will >>>>> file a >>>>> >>>> HBase >>>>> >>>> >>>>>>> issue >>>>> >>>> >>>>>>> >> to do >>>>> >>>> >>>>>>> >> > this. >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > And will report any problems on INFRA_22760 while >>>>> >>>> migrating. >>>>> >>>> >>>>>>> When the >>>>> >>>> >>>>>>> >> > migration is half done, you can move for example, 5 >>>>> hbase >>>>> >>>> nodes >>>>> >>>> >>>>>>> from >>>>> >>>> >>>>>>> >> > ci-hadoop to ci-hbase, and once the migration is >>>>> finally >>>>> >>>> done, >>>>> >>>> >>>>>>> you can >>>>> >>>> >>>>>>> >> move >>>>> >>>> >>>>>>> >> > all the remaining hbase nodes from ci-hadoop to >>>>> ci-hbase. >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > WDYT? >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >> However you want to do it is fine with me :) >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >> You have 10 nodes on there right now to get you >>>>> started. >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >> Let me know how you get on >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > Thanks. >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > Gavin McDonald <[email protected]> 于2022年1月18日周二 >>>>> >>>> 16:16写道： >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> > > Hi All, >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > I created an Infra ticket for tracking. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > https://issues.apache.org/jira/browse/INFRA-22760 >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > For now, I left your current nodes hbase1 to >>>>> hbase10 in >>>>> >>>> >>>>>>> ci-hadoop.a.o >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > I have created for you 10 new nodes so far out of >>>>> our >>>>> >>>> Hetzner >>>>> >>>> >>>>>>> DC >>>>> >>>> >>>>>>> >> > > of a similar specifications to the hbase1-10. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Those are hbase11-20 and are available right now >>>>> for >>>>> >>>> testing >>>>> >>>> >>>>>>> on >>>>> >>>> >>>>>>> >> > > ci-hbase.apache.org >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > I do not know your workflow well enough to decide >>>>> the >>>>> >>>> next >>>>> >>>> >>>>>>> steps for >>>>> >>>> >>>>>>> >> you >>>>> >>>> >>>>>>> >> > > for testing. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Options include: >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > 1. I can 'move' all jobs out of ci-hadoop onto >>>>> ci-hbase >>>>> >>>> >>>>>>> >> > > 2. I can 'move' one or a number of jobs out of >>>>> ci-hadoop >>>>> >>>> onto >>>>> >>>> >>>>>>> ci-hbase >>>>> >>>> >>>>>>> >> > > 3. Similar to 1 or 2, but 'copy' any number of >>>>> jobs over >>>>> >>>> for >>>>> >>>> >>>>>>> testing. >>>>> >>>> >>>>>>> >> > > - I like copy, however quite a few projects >>>>> jobs >>>>> >>>> would >>>>> >>>> >>>>>>> not be >>>>> >>>> >>>>>>> >> > > appropriate >>>>> >>>> >>>>>>> >> > > to have the same job running twice, including >>>>> >>>> those that >>>>> >>>> >>>>>>> >> deploy, or >>>>> >>>> >>>>>>> >> > > those that >>>>> >>>> >>>>>>> >> > > build from PRs and continue a workflow to >>>>> report >>>>> >>>> back, >>>>> >>>> >>>>>>> or >>>>> >>>> >>>>>>> >> approve >>>>> >>>> >>>>>>> >> > > for merge. >>>>> >>>> >>>>>>> >> > > So for a copy, would need to be a job that >>>>> just >>>>> >>>> 'builds >>>>> >>>> >>>>>>> and >>>>> >>>> >>>>>>> >> tests' >>>>> >>>> >>>>>>> >> > > with no outflow. >>>>> >>>> >>>>>>> >> > > 4. You guys create new jobs on ci-hbase.apache.org >>>>> to >>>>> >>>> test >>>>> >>>> >>>>>>> the nodes >>>>> >>>> >>>>>>> >> > > setup and the >>>>> >>>> >>>>>>> >> > > main controller setup, without affecting >>>>> current >>>>> >>>> jobs. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Obviously, you may to tweak version controlled >>>>> script >>>>> >>>> files >>>>> >>>> >>>>>>> such as a >>>>> >>>> >>>>>>> >> > > JenkinsFile, which makes the copy options harder >>>>> unless >>>>> >>>> >>>>>>> testing from a >>>>> >>>> >>>>>>> >> > > branch created for the purpose. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > I may be overthinking things, and option 1 , the >>>>> >>>> simplest and >>>>> >>>> >>>>>>> most >>>>> >>>> >>>>>>> >> > > straight forward from my end would also suit you >>>>> guys >>>>> >>>> and we >>>>> >>>> >>>>>>> deal >>>>> >>>> >>>>>>> >> with any >>>>> >>>> >>>>>>> >> > > breakage as it happens (I'm thinking of >>>>> integrations with >>>>> >>>> >>>>>>> other >>>>> >>>> >>>>>>> >> services, >>>>> >>>> >>>>>>> >> > > Credentials/Secrets, Plugins I haven't installed >>>>> that are >>>>> >>>> >>>>>>> needed etc.) >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Let me know please ASAP how you want to proceed so >>>>> we >>>>> >>>> can get >>>>> >>>> >>>>>>> using >>>>> >>>> >>>>>>> >> these >>>>> >>>> >>>>>>> >> > > nodes, after which I can then proceed to move >>>>> hbase1-10 >>>>> >>>> nodes >>>>> >>>> >>>>>>> over >>>>> >>>> >>>>>>> >> also. >>>>> >>>> >>>>>>> >> > > And then start adding more into the mix. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Oh. The 'git-websites' label should be a no-op. It >>>>> is >>>>> >>>> >>>>>>> available on >>>>> >>>> >>>>>>> >> > > ci-hbase.a.o as the nodes with that label are >>>>> shared and >>>>> >>>> >>>>>>> leased out to >>>>> >>>> >>>>>>> >> > > controllers as needed upon request of a job. >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Thanks >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > Gav... >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > > On 2021/12/09 15:07:26 "张铎(Duo Zhang)" wrote: >>>>> >>>> >>>>>>> >> > > > Thanks you Gavin. >>>>> >>>> >>>>>>> >> > > > >>>>> >>>> >>>>>>> >> > > > You can do it in your convenience, just let me >>>>> know so >>>>> >>>> I >>>>> >>>> >>>>>>> could >>>>> >>>> >>>>>>> >> change the >>>>> >>>> >>>>>>> >> > > > label in the jenkinsfile of our jenkins jobs, at >>>>> least >>>>> >>>> our >>>>> >>>> >>>>>>> pre >>>>> >>>> >>>>>>> >> commit job >>>>> >>>> >>>>>>> >> > > > still uses the Hadoop label. >>>>> >>>> >>>>>>> >> > > > >>>>> >>>> >>>>>>> >> > > > And another thing is about the >>>>> hbase_generate_website >>>>> >>>> job, >>>>> >>>> >>>>>>> it needs >>>>> >>>> >>>>>>> >> to be >>>>> >>>> >>>>>>> >> > > > executed on the nodes under the git-websites >>>>> label. >>>>> >>>> I'm not >>>>> >>>> >>>>>>> sure if >>>>> >>>> >>>>>>> >> the >>>>> >>>> >>>>>>> >> > > > label is still available after the migration. >>>>> >>>> >>>>>>> >> > > > >>>>> >>>> >>>>>>> >> > > > Gavin McDonald <[email protected]> >>>>> 于2021年12月8日周三 >>>>> >>>> >>>>>>> 17:28写道： >>>>> >>>> >>>>>>> >> > > > >>>>> >>>> >>>>>>> >> > > > > Hi HBase devs. >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > This is to let you know that Infra has created >>>>> a >>>>> >>>> >>>>>>> dedicated Jenkins >>>>> >>>> >>>>>>> >> > > > > Controller at https://ci-hbase.apache.org . >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > You have new incoming donated agents which >>>>> will get >>>>> >>>> >>>>>>> connected to >>>>> >>>> >>>>>>> >> this >>>>> >>>> >>>>>>> >> > > new >>>>> >>>> >>>>>>> >> > > > > controller by Infra. >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > You also have 10 or so agents connected to >>>>> >>>> >>>>>>> >> > > https://ci-hadoop.apache.org - >>>>> >>>> >>>>>>> >> > > > > I >>>>> >>>> >>>>>>> >> > > > > would like to move these agents - and all of >>>>> your >>>>> >>>> jobs - >>>>> >>>> >>>>>>> over to >>>>> >>>> >>>>>>> >> the >>>>> >>>> >>>>>>> >> > > new >>>>> >>>> >>>>>>> >> > > > > controller as soon as possible. >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > You may or may not need to reconfigure your >>>>> jobs and >>>>> >>>> or >>>>> >>>> >>>>>>> >> JenkinsFile or >>>>> >>>> >>>>>>> >> > > > > other. >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > Can I get the go ahead to move these at my >>>>> >>>> convenience or >>>>> >>>> >>>>>>> suggest >>>>> >>>> >>>>>>> >> > > please a >>>>> >>>> >>>>>>> >> > > > > timeline for Infra to move these agents. >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > Thanks! >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > -- >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > > *Gavin McDonald* >>>>> >>>> >>>>>>> >> > > > > Systems Administrator >>>>> >>>> >>>>>>> >> > > > > ASF Infrastructure Team >>>>> >>>> >>>>>>> >> > > > > >>>>> >>>> >>>>>>> >> > > > >>>>> >>>> >>>>>>> >> > > >>>>> >>>> >>>>>>> >> > >>>>> >>>> >>>>>>> >> >>>>> >>>> >>>>>>> > >>>>> >>>> >>>>>>> >>>>> >>>> >>>>>>> -- >>>>> >>>> >>>>>>> >>>>> >>>> >>>>>>> *Gavin McDonald* >>>>> >>>> >>>>>>> Systems Administrator >>>>> >>>> >>>>>>> ASF Infrastructure Team >>>>> >>>> >>>>>>> >>>>> >>>> >>>>>> >>>>> >>>> >>>>> >>>>> >>>> >>>>> -- >>>>> >>>> >>>>> >>>>> >>>> >>>>> *Gavin McDonald* >>>>> >>>> >>>>> Systems Administrator >>>>> >>>> >>>>> ASF Infrastructure Team >>>>> >>>> >>>>> >>>>> >>>> >>>> >>>>> >>>> >>>>> >>>> -- >>>>> >>>> >>>>> >>>> *Gavin McDonald* >>>>> >>>> Systems Administrator >>>>> >>>> ASF Infrastructure Team >>>>> >>>> >>>>> >>> >>>>> >>>>> -- >>>>> >>>>> *Gavin McDonald* >>>>> Systems Administrator >>>>> ASF Infrastructure Team >>>>> >>>>
