On Fri, Jul 24, 2020 at 3:05 AM Andor Molnar <[email protected]> wrote:
> Hi, > > Jenkinsfile is now committed to master, 3.5 and 3.6 branches. Build is > running here: > > https://ci-hadoop.apache.org/view/ZooKeeper/job/zookeeper-master-maven-multipipeline/ > > I use standard Git interface to access the repo, because using GitHub > means that branch discovery has to run through the github API. The problem > with this is that I haven’t specified credentials and anonymous access is > limited by some quota, meaning the discovery process takes ages (1-2 hours). > > Builds are running fine except I see a lot of concurrent builds on master > without any meaningful SCM change. Odd. I will change pollSCM trigger to > run @hourly and will see how it goes. > > What’s outstanding? > > - JDKs: currently only two: 8, 11 (LTS versions). What other JDKs would > you like to run against? > > - GitHub Pull Requests precommit job: I have no idea how to do this, but I > suspect we have to use the GitHub Api for this to work. > > Not sure if this is helpful but I saw it recently: > we need ability to trigger builds on pull request creation/updates which > requires the plug-in below or similar: > https://plugins.jenkins.io/ghprb/ That plugin is deprecated and not recommended for use . I have installed the *branch source plugin instead*, hopefully that works for you. If not, let's investigate further what else we can do > - Windows build: there’s no Windows agent available currently in the new > instance. > > - Ant builds: do we need this? > > Please share your thoughts. > > Regards, > Andor > > > > > > On 2020. Jul 23., at 22:08, Andor Molnar <[email protected]> wrote: > > > > Hi folks, > > > > PR is available: https://github.com/apache/zookeeper/pull/1409 > > > > Andor > > > > > > > > On Thu, 2020-07-23 at 11:32 +0200, Andor Molnar wrote: > >> Created a Jira for the task: > >> > >> https://issues.apache.org/jira/browse/ZOOKEEPER-3896 > >> > >> Andor > >> > >> > >> > >>> On 2020. Jul 21., at 12:10, Andor Molnar <[email protected]> wrote: > >>> > >>> Where’s that example…? > >>> > >>> > >>> > >>>> On 2020. Jul 20., at 20:02, Enrico Olivelli <[email protected]> > >>>> wrote: > >>>> > >>>> Il Lun 20 Lug 2020, 19:40 Andor Molnar <[email protected]> ha > >>>> scritto: > >>>> > >>>>> Hi Enrico, > >>>>> > >>>>> No worries, I only created a few jobs to make some progress, > >>>>> but feel free > >>>>> to ignore that and do it in a better way. The “View” or > >>>>> “Folder” that I was > >>>>> adding jobs is > >>>>> > >>>>> https://ci-hadoop.apache.org/view/ZooKeeper/ > >>>>> > >>>>> > >>>>> Andor > >>>>> > >>>> > >>>> This is an example from Apache Maven project. It is very complex > >>>> because > >>>> tests are in another repo and for lots if other reasons. We just > >>>> have to > >>>> create a simpler file. > >>>> > >>>> If nobody volunteers I can try to spend some time but I won't > >>>> have a fast > >>>> pace these weeks > >>>> > >>>> > >>>> Enrico > >>>> > >>>> > >>>> > >>>>> > >>>>>> On 2020. Jul 20., at 19:34, Enrico Olivelli < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>> Il Lun 20 Lug 2020, 19:02 Patrick Hunt <[email protected]> ha > >>>>>> scritto: > >>>>>> > >>>>>>> On Mon, Jul 20, 2020 at 9:47 AM Enrico Olivelli < > >>>>>>> [email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Il Lun 20 Lug 2020, 18:41 Patrick Hunt <[email protected]> > >>>>>>>> ha scritto: > >>>>>>>> > >>>>>>>>> On Sat, Jul 18, 2020 at 12:20 PM Andor Molnar < > >>>>>>>>> [email protected]> > >>>>>>> wrote: > >>>>>>>>>> Hi Pat, > >>>>>>>>>> > >>>>>>>>>> I have admin rights in the new system too and > >>>>>>>>>> probably can work on > >>>>>>> this > >>>>>>>>> on > >>>>>>>>>> Monday. > >>>>>>>>>> What’s “matrix” config? Shouldn’t we just replicate > >>>>>>>>>> the same jobs on > >>>>>>>> the > >>>>>>>>>> new instance? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> Not sure on the exact name/feature but "matrix" is > >>>>>>>>> basically the > >>>>>>> ability > >>>>>>>> to > >>>>>>>>> say "run this build/test across a set of JDK versions" > >>>>>>>>> rather than a > >>>>>>>> single > >>>>>>>>> version. As a result, instead of 3 jobs for zk3.6.0, > >>>>>>>>> jdk 1/2/3 you end > >>>>>>> up > >>>>>>>>> with a single job which runs three times, one for each > >>>>>>>>> jdk type and > >>>>>>>>> summarizes the results. I've seen this before, I assume > >>>>>>>>> it's a feature > >>>>>>> of > >>>>>>>>> jenkins itself? > >>>>>>>>> > >>>>>>>> > >>>>>>>> We should use Jenkins files and this configuration will > >>>>>>>> be easy and > >>>>>>>> committed to git > >>>>>>>> > >>>>>>> > >>>>>>> The ability to do "gitops" would be amazing! > >>>>>>> > >>>>>> > >>>>>> Yes. > >>>>>> But I saw on Slack that Andor already started to create new > >>>>>> jobs. > >>>>>> We should do only one way in order not to waste time. > >>>>>> > >>>>>> Andor can you please share your work? > >>>>>> Thanks for doing it > >>>>>> > >>>>>> Enrico > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Patrick > >>>>>>> > >>>>>>> > >>>>>>>> Enrico > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> Patrick > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Andor > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On 2020. Jul 17., at 2:51, Patrick Hunt < > >>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I updated the job I linked earlier based on what's > >>>>>>>>>>> the latest on > >>>>>>> the > >>>>>>>>>> legacy > >>>>>>>>>>> jenkins. It ran successfully > >>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/215/ > >>>>>>>>>>> > >>>>>>>>>>> I didn't replicate every config setting - main gap > >>>>>>>>>>> is the spotbugs > >>>>>>>>>>> post-build, which seems to be missing from the new > >>>>>>>>>>> jenkins plugins. > >>>>>>>>>>> > >>>>>>>>>>> That's just maven master though. Not sure about the > >>>>>>>>>>> rest. Can we do > >>>>>>>>> more > >>>>>>>>>> of > >>>>>>>>>>> a "matrix" config in the new system vs cloning all > >>>>>>>>>>> the time? > >>>>>>>>>>> > >>>>>>>>>>> Patrick > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Jul 16, 2020 at 3:31 PM Patrick Hunt < > >>>>>>>>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>>>>>> On Thu, Jul 16, 2020 at 3:22 PM Patrick Hunt < > >>>>>>>>>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Jul 16, 2020 at 12:56 PM Enrico > >>>>>>>>>>>>> Olivelli < > >>>>>>>>> [email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> FYI > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Fun. I do notice it says "Hadoop and related > >>>>>>>>>>>>> projects have their > >>>>>>>> own > >>>>>>>>>>>>> migration path to follow" - any insight on > >>>>>>>>>>>>> that? We are or are > >>>>>>> not > >>>>>>>>>> lumped > >>>>>>>>>>>>> in? I would assume we are? > >>>>>>>>>>>>> > >>>>>>>>>>>>> This (eventual migration) came up a while back > >>>>>>>>>>>>> on the Hadoop PMC > >>>>>>>> and > >>>>>>>>> I > >>>>>>>>>>>>> volunteered to try for us (ZK). I was never > >>>>>>>>>>>>> able to get it to > >>>>>>>> work, I > >>>>>>>>>>>>> provided feedback to infra but they never got > >>>>>>>>>>>>> back, as such we > >>>>>>>> have > >>>>>>>>> a > >>>>>>>>>>>>> project here that's not working with some basic > >>>>>>>>>>>>> dependencies > >>>>>>>> missing: > >>>>>>>>>>>>> https://ci-hadoop.apache.org/job/zookeeper-master-maven/ > >>>>>>>>>>>>> > >>>>>>>>>>>>> That said, we can try again. Can we verify > >>>>>>>>>>>>> where ZK is supposed > >>>>>>> to > >>>>>>>>>> land? > >>>>>>>>>>>>> Perhaps we can try to delete and recreate the > >>>>>>>>>>>>> POC job I created > >>>>>>> at > >>>>>>>>> that > >>>>>>>>>>>>> link to see if we can get it working? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> I see another email thread on the list saying > >>>>>>>>>>>> that we are part of > >>>>>>>> said > >>>>>>>>>>>> "related projects". We are expected to move to > >>>>>>>>>>>> http://ci-hadoop.apache.org/ within 4 weeks. > >>>>>>>>>>>> Seems nodes are > >>>>>>>> already > >>>>>>>>>>>> being removed/migrated from the "H#" pool. > >>>>>>>>>>>> > >>>>>>>>>>>> Also this: > >>>>>>>>>>>> > >>>>>>>>>>>> There are over 400 plugins on the current > >>>>>>>>>>>> builds.apache.org - > >>>>>>> most > >>>>>>>> of > >>>>>>>>>> which > >>>>>>>>>>>> we don't need any more, or are replaced with > >>>>>>>>>>>> different plugins on > >>>>>>>> the > >>>>>>>>>> new > >>>>>>>>>>>> system. I expect there may be some plugins we > >>>>>>>>>>>> still need to > >>>>>>> install > >>>>>>>> to > >>>>>>>>>> get > >>>>>>>>>>>> you going again, which is why it is vitally > >>>>>>>>>>>> important that you > >>>>>>> start > >>>>>>>>>>>> testing and migrating your jobs over *now*. You > >>>>>>>>>>>> should all have > >>>>>>>> auth. > >>>>>>>>>>>> Any questions, feel free to email the > >>>>>>> [email protected] > >>>>>>>>>> list if > >>>>>>>>>>>> you are one of the projects listed below. The > >>>>>>>>>>>> rest of you, not > >>>>>>>>> listed, a > >>>>>>>>>>>> similar email to this one will be posted for you > >>>>>>>>>>>> shortly on > >>>>>>>>> [email protected]. > >>>>>>>>>>>> > >>>>>>>>>>>> full details > >>>>>>>>>>>> > >>>>> > https://lists.apache.org/thread.html/r21c9d40cdbf5461143dd7eb4ff48a200c2fd20c50e946884f61318fd%40%3Cbuilds.apache.org%3E > >>>>>>>>>>>> Patrick > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> Patrick > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> ---------- Forwarded message --------- > >>>>>>>>>>>>>> Da: Gavin McDonald <[email protected]> > >>>>>>>>>>>>>> Date: Gio 16 Lug 2020, 18:33 > >>>>>>>>>>>>>> Subject: [IMPORTANT] - Migration to ci- > >>>>>>>>>>>>>> builds.a.o > >>>>>>>>>>>>>> To: builds <[email protected]> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This NOTICE is for everyone on > >>>>>>>>>>>>>> builds.apache.org. We are > >>>>>>>> migrating > >>>>>>>>>> to a > >>>>>>>>>>>>>> new > >>>>>>>>>>>>>> Cloudbees based Client Master called > >>>>>>> https://ci-builds.apache.org > >>>>>>>> . > >>>>>>>>>> The > >>>>>>>>>>>>>> migrations of all jobs needs to be done > >>>>>>>>>>>>>> before the switch off > >>>>>>> date > >>>>>>>>> of > >>>>>>>>>>>>>> 15th > >>>>>>>>>>>>>> August 2020, so you have a maximum of 4 > >>>>>>>>>>>>>> weeks. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> There is no tool or automated way of > >>>>>>>>>>>>>> migrating your jobs, the > >>>>>>>>>>>>>> differences in the platforms, the plugins and > >>>>>>>>>>>>>> the setup makes it > >>>>>>>>>>>>>> impossible > >>>>>>>>>>>>>> to do in a safe way. So, you all need to > >>>>>>>>>>>>>> start creating new jobs > >>>>>>>> on > >>>>>>>>>>>>>> ci-infra.a.o and then , when you are happy, > >>>>>>>>>>>>>> turn off your old > >>>>>>>> builds > >>>>>>>>>> on > >>>>>>>>>>>>>> builds.a.o. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> There are currently 4 agents over there ready > >>>>>>>>>>>>>> to take jobs, > >>>>>>> plus a > >>>>>>>>>>>>>> floating > >>>>>>>>>>>>>> agent which is shared amongst many masters > >>>>>>>>>>>>>> (more to come). I > >>>>>>> will > >>>>>>>>>> migrate > >>>>>>>>>>>>>> away 2 more agents from builds.a.o to ci- > >>>>>>>>>>>>>> builds.a.o every few > >>>>>>>> days, > >>>>>>>>>> and > >>>>>>>>>>>>>> will keep an eye of load across both and > >>>>>>>>>>>>>> adjust accordingly. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If needed, create a ticket on INFRA jira for > >>>>>>>>>>>>>> any issues that > >>>>>>> crop > >>>>>>>>> up, > >>>>>>>>>> or > >>>>>>>>>>>>>> email here on [email protected]. there may be one or > >>>>>>>>>>>>>> two plugins we > >>>>>>> need > >>>>>>>>> to > >>>>>>>>>>>>>> install/tweak etc. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We will be not using 'Views' at the top > >>>>>>>>>>>>>> level, but rather we > >>>>>>> will > >>>>>>>>> take > >>>>>>>>>>>>>> advantage of 'Folders Plus' - each project > >>>>>>>>>>>>>> will get its own > >>>>>>> Folder > >>>>>>>>> and > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>> authorisation access to create/edit jobs etc > >>>>>>>>>>>>>> within that > >>>>>>> folder. I > >>>>>>>>>> will > >>>>>>>>>>>>>> create these folders as you ask for them to > >>>>>>>>>>>>>> start with. This > >>>>>>> setup > >>>>>>>>>> allows > >>>>>>>>>>>>>> for credentials isolation amongst other > >>>>>>>>>>>>>> benefits, including but > >>>>>>>> not > >>>>>>>>>>>>>> limited > >>>>>>>>>>>>>> to exclusive agents (Controlled Agents) for > >>>>>>>>>>>>>> your own use , > >>>>>>> should > >>>>>>>>> you > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>> any project targeted donations of agents. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As with other aspects of the ASF, projects > >>>>>>>>>>>>>> can choose to just > >>>>>>>> enable > >>>>>>>>>> all > >>>>>>>>>>>>>> committers access to their folder, just ask. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We will re-use builds.apache.org as a CNAME > >>>>>>>>>>>>>> to ci-builds.a.o > >>>>>>> but > >>>>>>>>> will > >>>>>>>>>>>>>> not > >>>>>>>>>>>>>> be setting up any forwarding rules or > >>>>>>>>>>>>>> anything like that. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> So, please, get started *now *on this so you > >>>>>>>>>>>>>> can be sure we are > >>>>>>>> all > >>>>>>>>>>>>>> completed before the final cutoff date of > >>>>>>>>>>>>>> 15th August 2020. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Any questions - I expect a few (dozen :) ) - > >>>>>>>>>>>>>> ask away and/or > >>>>>>> file > >>>>>>>>>> INFRA > >>>>>>>>>>>>>> tickets. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hadoop and related projects have their own > >>>>>>>>>>>>>> migration path to > >>>>>>>> follow, > >>>>>>>>>> same > >>>>>>>>>>>>>> cut off date, Cassandra, Beam, CouchDB have > >>>>>>>>>>>>>> already migrated and > >>>>>>>> are > >>>>>>>>>>>>>> doing > >>>>>>>>>>>>>> very well in their new homes. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Lets get going ... > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> *Gavin McDonald* > >>>>>>>>>>>>>> Systems Administrator > >>>>>>>>>>>>>> ASF Infrastructure Team > > > >
