Hi devs, There’s some movement from Apache on Jenkins migration. Github account is now available on the new instance which will be useful for PreCommit builds. I started the tests.
Deadline (15 Aug) is approaching. Is there anything else which has fallen through the cracks and has to be migrated to the new instance? Andor > On 2020. Jul 24., at 19:15, Patrick Hunt <ph...@apache.org> wrote: > > On Fri, Jul 24, 2020 at 3:05 AM Andor Molnar <an...@apache.org> wrote: > >> Hi, >> >> Jenkinsfile is now committed to master, 3.5 and 3.6 branches. Build is >> running here: >> >> https://ci-hadoop.apache.org/view/ZooKeeper/job/zookeeper-master-maven-multipipeline/ >> >> I use standard Git interface to access the repo, because using GitHub >> means that branch discovery has to run through the github API. The problem >> with this is that I haven’t specified credentials and anonymous access is >> limited by some quota, meaning the discovery process takes ages (1-2 hours). >> >> Builds are running fine except I see a lot of concurrent builds on master >> without any meaningful SCM change. Odd. I will change pollSCM trigger to >> run @hourly and will see how it goes. >> >> What’s outstanding? >> >> - JDKs: currently only two: 8, 11 (LTS versions). What other JDKs would >> you like to run against? >> >> - GitHub Pull Requests precommit job: I have no idea how to do this, but I >> suspect we have to use the GitHub Api for this to work. >> >> > Not sure if this is helpful but I saw it recently: > >> we need ability to trigger builds on pull request creation/updates which > >> requires the plug-in below or similar: > >> https://plugins.jenkins.io/ghprb/ > > > > That plugin is deprecated and not recommended for use . > > > I have installed the *branch source plugin instead*, hopefully that works > for > > you. > > > If not, let's investigate further what else we can do > > > >> - Windows build: there’s no Windows agent available currently in the new >> instance. >> >> - Ant builds: do we need this? >> >> Please share your thoughts. >> >> Regards, >> Andor >> >> >> >> >>> On 2020. Jul 23., at 22:08, Andor Molnar <an...@apache.org> wrote: >>> >>> Hi folks, >>> >>> PR is available: https://github.com/apache/zookeeper/pull/1409 >>> >>> Andor >>> >>> >>> >>> On Thu, 2020-07-23 at 11:32 +0200, Andor Molnar wrote: >>>> Created a Jira for the task: >>>> >>>> https://issues.apache.org/jira/browse/ZOOKEEPER-3896 >>>> >>>> Andor >>>> >>>> >>>> >>>>> On 2020. Jul 21., at 12:10, Andor Molnar <an...@apache.org> wrote: >>>>> >>>>> Where’s that example…? >>>>> >>>>> >>>>> >>>>>> On 2020. Jul 20., at 20:02, Enrico Olivelli <eolive...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Il Lun 20 Lug 2020, 19:40 Andor Molnar <an...@apache.org> ha >>>>>> scritto: >>>>>> >>>>>>> Hi Enrico, >>>>>>> >>>>>>> No worries, I only created a few jobs to make some progress, >>>>>>> but feel free >>>>>>> to ignore that and do it in a better way. The “View” or >>>>>>> “Folder” that I was >>>>>>> adding jobs is >>>>>>> >>>>>>> https://ci-hadoop.apache.org/view/ZooKeeper/ >>>>>>> >>>>>>> >>>>>>> Andor >>>>>>> >>>>>> >>>>>> This is an example from Apache Maven project. It is very complex >>>>>> because >>>>>> tests are in another repo and for lots if other reasons. We just >>>>>> have to >>>>>> create a simpler file. >>>>>> >>>>>> If nobody volunteers I can try to spend some time but I won't >>>>>> have a fast >>>>>> pace these weeks >>>>>> >>>>>> >>>>>> Enrico >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>>> On 2020. Jul 20., at 19:34, Enrico Olivelli < >>>>>>>> eolive...@gmail.com> wrote: >>>>>>>> >>>>>>>> Il Lun 20 Lug 2020, 19:02 Patrick Hunt <ph...@apache.org> ha >>>>>>>> scritto: >>>>>>>> >>>>>>>>> On Mon, Jul 20, 2020 at 9:47 AM Enrico Olivelli < >>>>>>>>> eolive...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Il Lun 20 Lug 2020, 18:41 Patrick Hunt <ph...@apache.org> >>>>>>>>>> ha scritto: >>>>>>>>>> >>>>>>>>>>> On Sat, Jul 18, 2020 at 12:20 PM Andor Molnar < >>>>>>>>>>> an...@apache.org> >>>>>>>>> wrote: >>>>>>>>>>>> Hi Pat, >>>>>>>>>>>> >>>>>>>>>>>> I have admin rights in the new system too and >>>>>>>>>>>> probably can work on >>>>>>>>> this >>>>>>>>>>> on >>>>>>>>>>>> Monday. >>>>>>>>>>>> What’s “matrix” config? Shouldn’t we just replicate >>>>>>>>>>>> the same jobs on >>>>>>>>>> the >>>>>>>>>>>> new instance? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Not sure on the exact name/feature but "matrix" is >>>>>>>>>>> basically the >>>>>>>>> ability >>>>>>>>>> to >>>>>>>>>>> say "run this build/test across a set of JDK versions" >>>>>>>>>>> rather than a >>>>>>>>>> single >>>>>>>>>>> version. As a result, instead of 3 jobs for zk3.6.0, >>>>>>>>>>> jdk 1/2/3 you end >>>>>>>>> up >>>>>>>>>>> with a single job which runs three times, one for each >>>>>>>>>>> jdk type and >>>>>>>>>>> summarizes the results. I've seen this before, I assume >>>>>>>>>>> it's a feature >>>>>>>>> of >>>>>>>>>>> jenkins itself? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We should use Jenkins files and this configuration will >>>>>>>>>> be easy and >>>>>>>>>> committed to git >>>>>>>>>> >>>>>>>>> >>>>>>>>> The ability to do "gitops" would be amazing! >>>>>>>>> >>>>>>>> >>>>>>>> Yes. >>>>>>>> But I saw on Slack that Andor already started to create new >>>>>>>> jobs. >>>>>>>> We should do only one way in order not to waste time. >>>>>>>> >>>>>>>> Andor can you please share your work? >>>>>>>> Thanks for doing it >>>>>>>> >>>>>>>> Enrico >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Patrick >>>>>>>>> >>>>>>>>> >>>>>>>>>> Enrico >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Patrick >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Andor >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 2020. Jul 17., at 2:51, Patrick Hunt < >>>>>>>>>>>>> ph...@apache.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I updated the job I linked earlier based on what's >>>>>>>>>>>>> the latest on >>>>>>>>> the >>>>>>>>>>>> legacy >>>>>>>>>>>>> jenkins. It ran successfully >>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/215/ >>>>>>>>>>>>> >>>>>>>>>>>>> I didn't replicate every config setting - main gap >>>>>>>>>>>>> is the spotbugs >>>>>>>>>>>>> post-build, which seems to be missing from the new >>>>>>>>>>>>> jenkins plugins. >>>>>>>>>>>>> >>>>>>>>>>>>> That's just maven master though. Not sure about the >>>>>>>>>>>>> rest. Can we do >>>>>>>>>>> more >>>>>>>>>>>> of >>>>>>>>>>>>> a "matrix" config in the new system vs cloning all >>>>>>>>>>>>> the time? >>>>>>>>>>>>> >>>>>>>>>>>>> Patrick >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:31 PM Patrick Hunt < >>>>>>>>>>>>> ph...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:22 PM Patrick Hunt < >>>>>>>>>>>>>> ph...@apache.org> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Jul 16, 2020 at 12:56 PM Enrico >>>>>>>>>>>>>>> Olivelli < >>>>>>>>>>> eolive...@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> FYI >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Fun. I do notice it says "Hadoop and related >>>>>>>>>>>>>>> projects have their >>>>>>>>>> own >>>>>>>>>>>>>>> migration path to follow" - any insight on >>>>>>>>>>>>>>> that? We are or are >>>>>>>>> not >>>>>>>>>>>> lumped >>>>>>>>>>>>>>> in? I would assume we are? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This (eventual migration) came up a while back >>>>>>>>>>>>>>> on the Hadoop PMC >>>>>>>>>> and >>>>>>>>>>> I >>>>>>>>>>>>>>> volunteered to try for us (ZK). I was never >>>>>>>>>>>>>>> able to get it to >>>>>>>>>> work, I >>>>>>>>>>>>>>> provided feedback to infra but they never got >>>>>>>>>>>>>>> back, as such we >>>>>>>>>> have >>>>>>>>>>> a >>>>>>>>>>>>>>> project here that's not working with some basic >>>>>>>>>>>>>>> dependencies >>>>>>>>>> missing: >>>>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That said, we can try again. Can we verify >>>>>>>>>>>>>>> where ZK is supposed >>>>>>>>> to >>>>>>>>>>>> land? >>>>>>>>>>>>>>> Perhaps we can try to delete and recreate the >>>>>>>>>>>>>>> POC job I created >>>>>>>>> at >>>>>>>>>>> that >>>>>>>>>>>>>>> link to see if we can get it working? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> I see another email thread on the list saying >>>>>>>>>>>>>> that we are part of >>>>>>>>>> said >>>>>>>>>>>>>> "related projects". We are expected to move to >>>>>>>>>>>>>> http://ci-hadoop.apache.org/ within 4 weeks. >>>>>>>>>>>>>> Seems nodes are >>>>>>>>>> already >>>>>>>>>>>>>> being removed/migrated from the "H#" pool. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> There are over 400 plugins on the current >>>>>>>>>>>>>> builds.apache.org - >>>>>>>>> most >>>>>>>>>> of >>>>>>>>>>>> which >>>>>>>>>>>>>> we don't need any more, or are replaced with >>>>>>>>>>>>>> different plugins on >>>>>>>>>> the >>>>>>>>>>>> new >>>>>>>>>>>>>> system. I expect there may be some plugins we >>>>>>>>>>>>>> still need to >>>>>>>>> install >>>>>>>>>> to >>>>>>>>>>>> get >>>>>>>>>>>>>> you going again, which is why it is vitally >>>>>>>>>>>>>> important that you >>>>>>>>> start >>>>>>>>>>>>>> testing and migrating your jobs over *now*. You >>>>>>>>>>>>>> should all have >>>>>>>>>> auth. >>>>>>>>>>>>>> Any questions, feel free to email the >>>>>>>>> hadoop-migrati...@apache.org >>>>>>>>>>>> list if >>>>>>>>>>>>>> you are one of the projects listed below. The >>>>>>>>>>>>>> rest of you, not >>>>>>>>>>> listed, a >>>>>>>>>>>>>> similar email to this one will be posted for you >>>>>>>>>>>>>> shortly on >>>>>>>>>>> builds@a.o. >>>>>>>>>>>>>> >>>>>>>>>>>>>> full details >>>>>>>>>>>>>> >>>>>>> >> https://lists.apache.org/thread.html/r21c9d40cdbf5461143dd7eb4ff48a200c2fd20c50e946884f61318fd%40%3Cbuilds.apache.org%3E >>>>>>>>>>>>>> Patrick >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Patrick >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ---------- Forwarded message --------- >>>>>>>>>>>>>>>> Da: Gavin McDonald <gmcdon...@apache.org> >>>>>>>>>>>>>>>> Date: Gio 16 Lug 2020, 18:33 >>>>>>>>>>>>>>>> Subject: [IMPORTANT] - Migration to ci- >>>>>>>>>>>>>>>> builds.a.o >>>>>>>>>>>>>>>> To: builds <bui...@apache.org> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This NOTICE is for everyone on >>>>>>>>>>>>>>>> builds.apache.org. We are >>>>>>>>>> migrating >>>>>>>>>>>> to a >>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>> Cloudbees based Client Master called >>>>>>>>> https://ci-builds.apache.org >>>>>>>>>> . >>>>>>>>>>>> The >>>>>>>>>>>>>>>> migrations of all jobs needs to be done >>>>>>>>>>>>>>>> before the switch off >>>>>>>>> date >>>>>>>>>>> of >>>>>>>>>>>>>>>> 15th >>>>>>>>>>>>>>>> August 2020, so you have a maximum of 4 >>>>>>>>>>>>>>>> weeks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is no tool or automated way of >>>>>>>>>>>>>>>> migrating your jobs, the >>>>>>>>>>>>>>>> differences in the platforms, the plugins and >>>>>>>>>>>>>>>> the setup makes it >>>>>>>>>>>>>>>> impossible >>>>>>>>>>>>>>>> to do in a safe way. So, you all need to >>>>>>>>>>>>>>>> start creating new jobs >>>>>>>>>> on >>>>>>>>>>>>>>>> ci-infra.a.o and then , when you are happy, >>>>>>>>>>>>>>>> turn off your old >>>>>>>>>> builds >>>>>>>>>>>> on >>>>>>>>>>>>>>>> builds.a.o. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There are currently 4 agents over there ready >>>>>>>>>>>>>>>> to take jobs, >>>>>>>>> plus a >>>>>>>>>>>>>>>> floating >>>>>>>>>>>>>>>> agent which is shared amongst many masters >>>>>>>>>>>>>>>> (more to come). I >>>>>>>>> will >>>>>>>>>>>> migrate >>>>>>>>>>>>>>>> away 2 more agents from builds.a.o to ci- >>>>>>>>>>>>>>>> builds.a.o every few >>>>>>>>>> days, >>>>>>>>>>>> and >>>>>>>>>>>>>>>> will keep an eye of load across both and >>>>>>>>>>>>>>>> adjust accordingly. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If needed, create a ticket on INFRA jira for >>>>>>>>>>>>>>>> any issues that >>>>>>>>> crop >>>>>>>>>>> up, >>>>>>>>>>>> or >>>>>>>>>>>>>>>> email here on builds@a.o. there may be one or >>>>>>>>>>>>>>>> two plugins we >>>>>>>>> need >>>>>>>>>>> to >>>>>>>>>>>>>>>> install/tweak etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We will be not using 'Views' at the top >>>>>>>>>>>>>>>> level, but rather we >>>>>>>>> will >>>>>>>>>>> take >>>>>>>>>>>>>>>> advantage of 'Folders Plus' - each project >>>>>>>>>>>>>>>> will get its own >>>>>>>>> Folder >>>>>>>>>>> and >>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>> authorisation access to create/edit jobs etc >>>>>>>>>>>>>>>> within that >>>>>>>>> folder. I >>>>>>>>>>>> will >>>>>>>>>>>>>>>> create these folders as you ask for them to >>>>>>>>>>>>>>>> start with. This >>>>>>>>> setup >>>>>>>>>>>> allows >>>>>>>>>>>>>>>> for credentials isolation amongst other >>>>>>>>>>>>>>>> benefits, including but >>>>>>>>>> not >>>>>>>>>>>>>>>> limited >>>>>>>>>>>>>>>> to exclusive agents (Controlled Agents) for >>>>>>>>>>>>>>>> your own use , >>>>>>>>> should >>>>>>>>>>> you >>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>> any project targeted donations of agents. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As with other aspects of the ASF, projects >>>>>>>>>>>>>>>> can choose to just >>>>>>>>>> enable >>>>>>>>>>>> all >>>>>>>>>>>>>>>> committers access to their folder, just ask. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We will re-use builds.apache.org as a CNAME >>>>>>>>>>>>>>>> to ci-builds.a.o >>>>>>>>> but >>>>>>>>>>> will >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>> be setting up any forwarding rules or >>>>>>>>>>>>>>>> anything like that. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So, please, get started *now *on this so you >>>>>>>>>>>>>>>> can be sure we are >>>>>>>>>> all >>>>>>>>>>>>>>>> completed before the final cutoff date of >>>>>>>>>>>>>>>> 15th August 2020. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Any questions - I expect a few (dozen :) ) - >>>>>>>>>>>>>>>> ask away and/or >>>>>>>>> file >>>>>>>>>>>> INFRA >>>>>>>>>>>>>>>> tickets. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hadoop and related projects have their own >>>>>>>>>>>>>>>> migration path to >>>>>>>>>> follow, >>>>>>>>>>>> same >>>>>>>>>>>>>>>> cut off date, Cassandra, Beam, CouchDB have >>>>>>>>>>>>>>>> already migrated and >>>>>>>>>> are >>>>>>>>>>>>>>>> doing >>>>>>>>>>>>>>>> very well in their new homes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Lets get going ... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Gavin McDonald* >>>>>>>>>>>>>>>> Systems Administrator >>>>>>>>>>>>>>>> ASF Infrastructure Team >>> >> >>
