I've add a new build test to run the ITs under sunny day conditions using minicluster for both java 7 and java 8 on the 1.2 line.
https://builds.apache.org/job/HBase-1.2-IT/ I haven't turned on notifications yet, because BulkIngest and TestIngest are read. details on HBASE-13750 On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <[email protected]> wrote: > sigh. that should have been "is now a matrix build". > > On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <[email protected]> wrote: > >> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8 in >> parallel. >> >> I saved the old job for now and left it disabled[2]. >> >> >> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/ >> [2]: >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/ >> >> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <[email protected]> wrote: >> >>> Sorry for the noise. I also updated the checkstyle step on HBase-TRUNK >>> >>> from >>> mvn checkstyle:checkstyle >>> to >>> mvn -DskipTests package checkstyle:checkstyle >>> >>> to deal with the same issue. Looks all clear now. >>> >>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <[email protected]> >>> wrote: >>> >>>> Updated the following builds: >>>> >>>> * HBase-TRUNK >>>> >>>> moved from >>>> >>>> mvn clean compile findbugs:findbugs >>>> >>>> to >>>> >>>> mvn clean -DskipTests package findbugs:findbugs >>>> >>>> To work around the known issue where we can't do a bootstrap compile >>>> without previous install or remote SNAPSHOT artifacts. (recently triggered >>>> by the addition of the procedure module on master) >>>> >>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <[email protected]> wrote: >>>> >>>>> Make findbugs and checkstyle plugins always run. >>>>> The default behavior only publishes result for stable builds, but >>>>> master is >>>>> not always stable and sometimes we introduce new warnings in unstable >>>>> builds(see builds 6310-6314). >>>>> And findbugs and checkstyle will not fail unless we abort the building >>>>> process I think, so it is safe to turn it on every time. >>>>> >>>>> Thanks. >>>>> >>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <[email protected]>: >>>>> >>>>> > +1 on letting HBase-TRUNK jenkins show coverage report. >>>>> > >>>>> > Cheers >>>>> > >>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <[email protected]> wrote: >>>>> > >>>>> > > Going to change the config of HBase-TRUNK jenkins to show findbugs, >>>>> > > checkstyle and jacoco coverage report. >>>>> > > The config has been tested on >>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly 30 >>>>> times. >>>>> > Not >>>>> > > much different from HBase-TRUNK unless it runs ~30% slower(the >>>>> overhead >>>>> > of >>>>> > > collecting information for code coverage). >>>>> > > Thanks. >>>>> > > >>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <[email protected]>: >>>>> > > >>>>> > > > +1, thanks a lot for improving our build hygiene. >>>>> > > > >>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar < >>>>> [email protected]> >>>>> > > wrote: >>>>> > > > >>>>> > > > > Thanks Sean. This is great. >>>>> > > > > >>>>> > > > > Enis >>>>> > > > > >>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey < >>>>> [email protected]> >>>>> > > > wrote: >>>>> > > > > >>>>> > > > > > FYI I just finished chasing down the breakage for mvn site >>>>> on all >>>>> > > patch >>>>> > > > > > builds. >>>>> > > > > > >>>>> > > > > > HBASE-13191 consolidates the few places in the test-patch >>>>> code >>>>> > where >>>>> > > we >>>>> > > > > > hard coded MAVEN_OPTS. >>>>> > > > > > >>>>> > > > > > If you look at the PreCommit job now, we use the "set >>>>> environment >>>>> > > > > > variables" option to set MAVEN_OPTS and then everything else >>>>> > respects >>>>> > > > > that >>>>> > > > > > setting. >>>>> > > > > > >>>>> > > > > > I set the initial value to be a combination of the memory >>>>> > limitations >>>>> > > > > we've >>>>> > > > > > been actually running with (the ~6G was getting ignored) and >>>>> the >>>>> > > > permgen >>>>> > > > > > needed for site. >>>>> > > > > > >>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData -XX:MaxPermSize=256m >>>>> > > > > > >>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack <[email protected]> >>>>> wrote: >>>>> > > > > > >>>>> > > > > > > I just made TRUNK and branch-1 builds use same jvm as >>>>> patch-build >>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS to >>>>> be the >>>>> > > same >>>>> > > > as >>>>> > > > > > > those of trunk build too, setting >>>>> MAVEN_OPTS="-Xmx6100m"... it >>>>> > had >>>>> > > > been >>>>> > > > > > > 3000. >>>>> > > > > > > >>>>> > > > > > > Yours, >>>>> > > > > > > St.Ack >>>>> > > > > > > >>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack <[email protected]> >>>>> wrote: >>>>> > > > > > > >>>>> > > > > > > > I upped hadoopqa retention to keep last 100 builds and >>>>> or last >>>>> > 7 >>>>> > > > > days, >>>>> > > > > > > > whichever comes first. >>>>> > > > > > > > St.Ack >>>>> > > > > > > > >>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack <[email protected]> >>>>> > wrote: >>>>> > > > > > > > >>>>> > > > > > > >> Branch-1 and master have stabilized and now run mostly >>>>> blue >>>>> > > (give >>>>> > > > or >>>>> > > > > > > take >>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue branch-1 >>>>> has >>>>> > > helped >>>>> > > > us >>>>> > > > > > > >> identify at least one destabilizing commit in the last >>>>> few >>>>> > days, >>>>> > > > > maybe >>>>> > > > > > > two; >>>>> > > > > > > >> this is as it should be (smile). >>>>> > > > > > > >> >>>>> > > > > > > >> Lets keep our builds blue. If you commit a patch, make >>>>> sure >>>>> > > > > subsequent >>>>> > > > > > > >> builds stay blue. You can subscribe to >>>>> > [email protected] >>>>> > > to >>>>> > > > > get >>>>> > > > > > > >> notice of failures if not already subscribed. >>>>> > > > > > > >> >>>>> > > > > > > >> Thanks, >>>>> > > > > > > >> St.Ack >>>>> > > > > > > >> >>>>> > > > > > > >> 1. >>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/ >>>>> > > > > > > >> 2. >>>>> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/ >>>>> > > > > > > >> >>>>> > > > > > > >> >>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack < >>>>> [email protected]> >>>>> > > wrote: >>>>> > > > > > > >> >>>>> > > > > > > >>> A few notes on testing. >>>>> > > > > > > >>> >>>>> > > > > > > >>> Too long to read, infra is more capable now and after >>>>> some >>>>> > > work, >>>>> > > > we >>>>> > > > > > are >>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue. Lets >>>>> try and >>>>> > > keep >>>>> > > > it >>>>> > > > > > > this >>>>> > > > > > > >>> way going forward. >>>>> > > > > > > >>> >>>>> > > > > > > >>> Apache Infra has new, more capable hardware. >>>>> > > > > > > >>> >>>>> > > > > > > >>> A recent spurt of test fixing combined with more >>>>> capable >>>>> > > hardware >>>>> > > > > > seems >>>>> > > > > > > >>> to have gotten us to a new place; tests are mostly >>>>> passing >>>>> > now >>>>> > > on >>>>> > > > > > > branch-1 >>>>> > > > > > > >>> and master. Lets try and keep it this way and start >>>>> to trust >>>>> > > our >>>>> > > > > > test >>>>> > > > > > > runs >>>>> > > > > > > >>> again. Just a few flakies remain. Lets try and nail >>>>> them. >>>>> > > > > > > >>> >>>>> > > > > > > >>> Our tests now run in parallel with other test suites >>>>> where >>>>> > > > previous >>>>> > > > > > we >>>>> > > > > > > >>> ran alone. You can see this sometimes when our zombie >>>>> > detector >>>>> > > > > > reports >>>>> > > > > > > >>> tests from another project altogether as lingerers (To >>>>> be >>>>> > > fixed). >>>>> > > > > > > Some of >>>>> > > > > > > >>> our tests are failing because a concurrent hbase run is >>>>> > undoing >>>>> > > > > > > classes and >>>>> > > > > > > >>> data from under it. Also, lets fix. >>>>> > > > > > > >>> >>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for them to >>>>> > complete. >>>>> > > > > Many >>>>> > > > > > > >>> are heavy-duty integration tests starting up multiple >>>>> > clusters >>>>> > > > and >>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle they >>>>> pass at >>>>> > all. >>>>> > > > > > > Usually >>>>> > > > > > > >>> integration tests have been cast as unit tests because >>>>> there >>>>> > > was >>>>> > > > no >>>>> > > > > > > where >>>>> > > > > > > >>> else for them to get an airing. We have the hbase-it >>>>> suite >>>>> > now >>>>> > > > > which >>>>> > > > > > > would >>>>> > > > > > > >>> be a more apt place but until these are run on a >>>>> regular >>>>> > basis >>>>> > > in >>>>> > > > > > > public >>>>> > > > > > > >>> for all to see, the fat integration tests disguised as >>>>> unit >>>>> > > tests >>>>> > > > > > will >>>>> > > > > > > >>> remain. A review of our current unit tests weeding >>>>> the old >>>>> > > cruft >>>>> > > > > and >>>>> > > > > > > the >>>>> > > > > > > >>> no longer relevant or duplicates would be a nice >>>>> undertaking >>>>> > if >>>>> > > > > > > someone is >>>>> > > > > > > >>> looking to contribute. >>>>> > > > > > > >>> >>>>> > > > > > > >>> Alex Newman has been working on making our tests work >>>>> up on >>>>> > > > travis >>>>> > > > > > and >>>>> > > > > > > >>> circle-ci. That'll be sweet when it goes end-to-end. >>>>> He >>>>> > also >>>>> > > > > added >>>>> > > > > > in >>>>> > > > > > > >>> some "type" categorizations -- client, filter, >>>>> mapreduce -- >>>>> > > > > alongside >>>>> > > > > > > our >>>>> > > > > > > >>> old "sizing" categorizations of small/medium/large. >>>>> His >>>>> > > thinking >>>>> > > > > is >>>>> > > > > > > that >>>>> > > > > > > >>> we can run these categorizations in parallel so we >>>>> could run >>>>> > > the >>>>> > > > > > total >>>>> > > > > > > >>> suite in about the time of the longest test, say >>>>> > 20-30minutes? >>>>> > > > We >>>>> > > > > > > could >>>>> > > > > > > >>> even change Apache to run them this way. >>>>> > > > > > > >>> >>>>> > > > > > > >>> FYI, >>>>> > > > > > > >>> St.Ack >>>>> > > > > > > >>> >>>>> > > > > > > >>> >>>>> > > > > > > >>> >>>>> > > > > > > >>> >>>>> > > > > > > >>> >>>>> > > > > > > >>> >>>>> > > > > > > >>> >>>>> > > > > > > >> >>>>> > > > > > > > >>>>> > > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > -- >>>>> > > > > > Sean >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > -- >>>>> > > > Best regards, >>>>> > > > >>>>> > > > - Andy >>>>> > > > >>>>> > > > Problems worthy of attack prove their worth by hitting back. - >>>>> Piet >>>>> > Hein >>>>> > > > (via Tom White) >>>>> > > > >>>>> > > >>>>> > >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean >>>> >>> >>> >>> >>> -- >>> Sean >>> >> >> >> >> -- >> Sean >> > > > > -- > Sean > -- Sean
