I thought HBase_TRUNK was already disabled. if I missed it, disabling should be sufficient. There are a few older non-matrix (0.98 I think?) that we can delete once we're confident in the matrix version
On Tue, Nov 3, 2015 at 8:44 AM, Stack <st...@duboce.net> wrote: > Stuff seems to be basically working. I just committed a fix for this > message so it should be gone now: > > grep: unknown devices method > grep: write error: Broken pipe > > > Hopefully we've seen last of the errant surefire kills. > > We have an HBASE_TRUNK build and we have an HBASE_TRUNK_MATRIX. We should > get rid of the former now? > > St.Ack > > On Mon, Nov 2, 2015 at 11:03 PM, Stack <st...@duboce.net> wrote: > >> I just did an edit of all of our jenkins configuration. I changed the >> post-build zombie section. The surefire-killer was hanging out here it >> seems (For detail, see HBASE-14589 Looking for the surefire-killer; builds >> being killed...). I may have messed up builds. Will fix in morning if so >> (currently builds seem a bit backed up so won't wait around). >> >> St.Ack >> >> On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey <bus...@cloudera.com> wrote: >> >>> Hi Folks, >>> >>> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS] >>> reducing our load on builds.a.o' on reasoning or if you have concerns. >>> >>> Our project job view now has a brief summary of what's going on: >>> >>> https://builds.apache.org/view/H-L/view/HBase/ >>> >>> * Where we had matrix builds, I confirmed they're doing the same thing >>> (in at least one test-configuration) as the non-matrix version >>> * I disabled jobs that were duplicating work done in matrix jobs, or >>> that only worked on a tag >>> * Made sure all non-disabled 0.98+ jobs are set to notify jira + >>> builds mailing list >>> * I updated pre-1.1 jobs to look for changes 1/day >>> * I updated 1.1 jobs to look for changes every 8 hours >>> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours >>> >>> There's still some non-pressing clean up work to do, so I'll file a >>> jira later today just to make sure it's all recorded somewhere. >>> >>> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <bus...@cloudera.com> >>> wrote: >>> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2 >>> builds to >>> > use branch-1.2. (ref HBASE-13912) >>> > >>> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <bus...@cloudera.com> >>> wrote: >>> >> >>> >> I've add a new build test to run the ITs under sunny day conditions >>> using >>> >> minicluster for both java 7 and java 8 on the 1.2 line. >>> >> >>> >> https://builds.apache.org/job/HBase-1.2-IT/ >>> >> >>> >> I haven't turned on notifications yet, because BulkIngest and >>> TestIngest >>> >> are read. details on HBASE-13750 >>> >> >>> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <bus...@cloudera.com> >>> wrote: >>> >>> >>> >>> sigh. that should have been "is now a matrix build". >>> >>> >>> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <bus...@cloudera.com> >>> wrote: >>> >>>> >>> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8 >>> in >>> >>>> parallel. >>> >>>> >>> >>>> I saved the old job for now and left it disabled[2]. >>> >>>> >>> >>>> >>> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/ >>> >>>> [2]: >>> >>>> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/ >>> >>>> >>> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <bus...@cloudera.com> >>> >>>> wrote: >>> >>>>> >>> >>>>> Sorry for the noise. I also updated the checkstyle step on >>> HBase-TRUNK >>> >>>>> >>> >>>>> from >>> >>>>> mvn checkstyle:checkstyle >>> >>>>> to >>> >>>>> mvn -DskipTests package checkstyle:checkstyle >>> >>>>> >>> >>>>> to deal with the same issue. Looks all clear now. >>> >>>>> >>> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <bus...@cloudera.com> >>> >>>>> wrote: >>> >>>>>> >>> >>>>>> Updated the following builds: >>> >>>>>> >>> >>>>>> * HBase-TRUNK >>> >>>>>> >>> >>>>>> moved from >>> >>>>>> >>> >>>>>> mvn clean compile findbugs:findbugs >>> >>>>>> >>> >>>>>> to >>> >>>>>> >>> >>>>>> mvn clean -DskipTests package findbugs:findbugs >>> >>>>>> >>> >>>>>> To work around the known issue where we can't do a bootstrap >>> compile >>> >>>>>> without previous install or remote SNAPSHOT artifacts. (recently >>> triggered >>> >>>>>> by the addition of the procedure module on master) >>> >>>>>> >>> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino...@gmail.com> wrote: >>> >>>>>>> >>> >>>>>>> Make findbugs and checkstyle plugins always run. >>> >>>>>>> The default behavior only publishes result for stable builds, but >>> >>>>>>> master is >>> >>>>>>> not always stable and sometimes we introduce new warnings in >>> unstable >>> >>>>>>> builds(see builds 6310-6314). >>> >>>>>>> And findbugs and checkstyle will not fail unless we abort the >>> >>>>>>> building >>> >>>>>>> process I think, so it is safe to turn it on every time. >>> >>>>>>> >>> >>>>>>> Thanks. >>> >>>>>>> >>> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: >>> >>>>>>> >>> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report. >>> >>>>>>> > >>> >>>>>>> > Cheers >>> >>>>>>> > >>> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino...@gmail.com> >>> wrote: >>> >>>>>>> > >>> >>>>>>> > > Going to change the config of HBase-TRUNK jenkins to show >>> >>>>>>> > > findbugs, >>> >>>>>>> > > checkstyle and jacoco coverage report. >>> >>>>>>> > > The config has been tested on >>> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly >>> 30 >>> >>>>>>> > > times. >>> >>>>>>> > Not >>> >>>>>>> > > much different from HBase-TRUNK unless it runs ~30% slower(the >>> >>>>>>> > > overhead >>> >>>>>>> > of >>> >>>>>>> > > collecting information for code coverage). >>> >>>>>>> > > Thanks. >>> >>>>>>> > > >>> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <apurt...@apache.org >>> >: >>> >>>>>>> > > >>> >>>>>>> > > > +1, thanks a lot for improving our build hygiene. >>> >>>>>>> > > > >>> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar >>> >>>>>>> > > > <enis....@gmail.com> >>> >>>>>>> > > wrote: >>> >>>>>>> > > > >>> >>>>>>> > > > > Thanks Sean. This is great. >>> >>>>>>> > > > > >>> >>>>>>> > > > > Enis >>> >>>>>>> > > > > >>> >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey >>> >>>>>>> > > > > <bus...@cloudera.com> >>> >>>>>>> > > > wrote: >>> >>>>>>> > > > > >>> >>>>>>> > > > > > FYI I just finished chasing down the breakage for mvn >>> site >>> >>>>>>> > > > > > on all >>> >>>>>>> > > patch >>> >>>>>>> > > > > > builds. >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > HBASE-13191 consolidates the few places in the >>> test-patch >>> >>>>>>> > > > > > code >>> >>>>>>> > where >>> >>>>>>> > > we >>> >>>>>>> > > > > > hard coded MAVEN_OPTS. >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > If you look at the PreCommit job now, we use the "set >>> >>>>>>> > > > > > environment >>> >>>>>>> > > > > > variables" option to set MAVEN_OPTS and then everything >>> >>>>>>> > > > > > else >>> >>>>>>> > respects >>> >>>>>>> > > > > that >>> >>>>>>> > > > > > setting. >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > I set the initial value to be a combination of the >>> memory >>> >>>>>>> > limitations >>> >>>>>>> > > > > we've >>> >>>>>>> > > > > > been actually running with (the ~6G was getting ignored) >>> >>>>>>> > > > > > and the >>> >>>>>>> > > > permgen >>> >>>>>>> > > > > > needed for site. >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData >>> -XX:MaxPermSize=256m >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack < >>> st...@duboce.net> >>> >>>>>>> > > > > > wrote: >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > > I just made TRUNK and branch-1 builds use same jvm as >>> >>>>>>> > > > > > > patch-build >>> >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS >>> to >>> >>>>>>> > > > > > > be the >>> >>>>>>> > > same >>> >>>>>>> > > > as >>> >>>>>>> > > > > > > those of trunk build too, setting >>> >>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"... it >>> >>>>>>> > had >>> >>>>>>> > > > been >>> >>>>>>> > > > > > > 3000. >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > Yours, >>> >>>>>>> > > > > > > St.Ack >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack < >>> st...@duboce.net> >>> >>>>>>> > > > > > > wrote: >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > > I upped hadoopqa retention to keep last 100 builds >>> and >>> >>>>>>> > > > > > > > or last >>> >>>>>>> > 7 >>> >>>>>>> > > > > days, >>> >>>>>>> > > > > > > > whichever comes first. >>> >>>>>>> > > > > > > > St.Ack >>> >>>>>>> > > > > > > > >>> >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack >>> >>>>>>> > > > > > > > <st...@duboce.net> >>> >>>>>>> > wrote: >>> >>>>>>> > > > > > > > >>> >>>>>>> > > > > > > >> Branch-1 and master have stabilized and now run >>> mostly >>> >>>>>>> > > > > > > >> blue >>> >>>>>>> > > (give >>> >>>>>>> > > > or >>> >>>>>>> > > > > > > take >>> >>>>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue >>> branch-1 >>> >>>>>>> > > > > > > >> has >>> >>>>>>> > > helped >>> >>>>>>> > > > us >>> >>>>>>> > > > > > > >> identify at least one destabilizing commit in the >>> last >>> >>>>>>> > > > > > > >> few >>> >>>>>>> > days, >>> >>>>>>> > > > > maybe >>> >>>>>>> > > > > > > two; >>> >>>>>>> > > > > > > >> this is as it should be (smile). >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > >> Lets keep our builds blue. If you commit a patch, >>> make >>> >>>>>>> > > > > > > >> sure >>> >>>>>>> > > > > subsequent >>> >>>>>>> > > > > > > >> builds stay blue. You can subscribe to >>> >>>>>>> > bui...@hbase.apache.org >>> >>>>>>> > > to >>> >>>>>>> > > > > get >>> >>>>>>> > > > > > > >> notice of failures if not already subscribed. >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > >> Thanks, >>> >>>>>>> > > > > > > >> St.Ack >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > >> 1. >>> >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/ >>> >>>>>>> > > > > > > >> 2. >>> >>>>>>> > > >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/ >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack >>> >>>>>>> > > > > > > >> <st...@duboce.net> >>> >>>>>>> > > wrote: >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > >>> A few notes on testing. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> Too long to read, infra is more capable now and >>> after >>> >>>>>>> > > > > > > >>> some >>> >>>>>>> > > work, >>> >>>>>>> > > > we >>> >>>>>>> > > > > > are >>> >>>>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue. >>> Lets >>> >>>>>>> > > > > > > >>> try and >>> >>>>>>> > > keep >>> >>>>>>> > > > it >>> >>>>>>> > > > > > > this >>> >>>>>>> > > > > > > >>> way going forward. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> Apache Infra has new, more capable hardware. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> A recent spurt of test fixing combined with more >>> >>>>>>> > > > > > > >>> capable >>> >>>>>>> > > hardware >>> >>>>>>> > > > > > seems >>> >>>>>>> > > > > > > >>> to have gotten us to a new place; tests are mostly >>> >>>>>>> > > > > > > >>> passing >>> >>>>>>> > now >>> >>>>>>> > > on >>> >>>>>>> > > > > > > branch-1 >>> >>>>>>> > > > > > > >>> and master. Lets try and keep it this way and >>> start >>> >>>>>>> > > > > > > >>> to trust >>> >>>>>>> > > our >>> >>>>>>> > > > > > test >>> >>>>>>> > > > > > > runs >>> >>>>>>> > > > > > > >>> again. Just a few flakies remain. Lets try and >>> nail >>> >>>>>>> > > > > > > >>> them. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> Our tests now run in parallel with other test >>> suites >>> >>>>>>> > > > > > > >>> where >>> >>>>>>> > > > previous >>> >>>>>>> > > > > > we >>> >>>>>>> > > > > > > >>> ran alone. You can see this sometimes when our >>> zombie >>> >>>>>>> > detector >>> >>>>>>> > > > > > reports >>> >>>>>>> > > > > > > >>> tests from another project altogether as lingerers >>> >>>>>>> > > > > > > >>> (To be >>> >>>>>>> > > fixed). >>> >>>>>>> > > > > > > Some of >>> >>>>>>> > > > > > > >>> our tests are failing because a concurrent hbase >>> run >>> >>>>>>> > > > > > > >>> is >>> >>>>>>> > undoing >>> >>>>>>> > > > > > > classes and >>> >>>>>>> > > > > > > >>> data from under it. Also, lets fix. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for >>> them to >>> >>>>>>> > complete. >>> >>>>>>> > > > > Many >>> >>>>>>> > > > > > > >>> are heavy-duty integration tests starting up >>> multiple >>> >>>>>>> > clusters >>> >>>>>>> > > > and >>> >>>>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle they >>> >>>>>>> > > > > > > >>> pass at >>> >>>>>>> > all. >>> >>>>>>> > > > > > > Usually >>> >>>>>>> > > > > > > >>> integration tests have been cast as unit tests >>> >>>>>>> > > > > > > >>> because there >>> >>>>>>> > > was >>> >>>>>>> > > > no >>> >>>>>>> > > > > > > where >>> >>>>>>> > > > > > > >>> else for them to get an airing. We have the >>> hbase-it >>> >>>>>>> > > > > > > >>> suite >>> >>>>>>> > now >>> >>>>>>> > > > > which >>> >>>>>>> > > > > > > would >>> >>>>>>> > > > > > > >>> be a more apt place but until these are run on a >>> >>>>>>> > > > > > > >>> regular >>> >>>>>>> > basis >>> >>>>>>> > > in >>> >>>>>>> > > > > > > public >>> >>>>>>> > > > > > > >>> for all to see, the fat integration tests >>> disguised >>> >>>>>>> > > > > > > >>> as unit >>> >>>>>>> > > tests >>> >>>>>>> > > > > > will >>> >>>>>>> > > > > > > >>> remain. A review of our current unit tests >>> weeding >>> >>>>>>> > > > > > > >>> the old >>> >>>>>>> > > cruft >>> >>>>>>> > > > > and >>> >>>>>>> > > > > > > the >>> >>>>>>> > > > > > > >>> no longer relevant or duplicates would be a nice >>> >>>>>>> > > > > > > >>> undertaking >>> >>>>>>> > if >>> >>>>>>> > > > > > > someone is >>> >>>>>>> > > > > > > >>> looking to contribute. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> Alex Newman has been working on making our tests >>> work >>> >>>>>>> > > > > > > >>> up on >>> >>>>>>> > > > travis >>> >>>>>>> > > > > > and >>> >>>>>>> > > > > > > >>> circle-ci. That'll be sweet when it goes >>> end-to-end. >>> >>>>>>> > > > > > > >>> He >>> >>>>>>> > also >>> >>>>>>> > > > > added >>> >>>>>>> > > > > > in >>> >>>>>>> > > > > > > >>> some "type" categorizations -- client, filter, >>> >>>>>>> > > > > > > >>> mapreduce -- >>> >>>>>>> > > > > alongside >>> >>>>>>> > > > > > > our >>> >>>>>>> > > > > > > >>> old "sizing" categorizations of >>> small/medium/large. >>> >>>>>>> > > > > > > >>> His >>> >>>>>>> > > thinking >>> >>>>>>> > > > > is >>> >>>>>>> > > > > > > that >>> >>>>>>> > > > > > > >>> we can run these categorizations in parallel so we >>> >>>>>>> > > > > > > >>> could run >>> >>>>>>> > > the >>> >>>>>>> > > > > > total >>> >>>>>>> > > > > > > >>> suite in about the time of the longest test, say >>> >>>>>>> > 20-30minutes? >>> >>>>>>> > > > We >>> >>>>>>> > > > > > > could >>> >>>>>>> > > > > > > >>> even change Apache to run them this way. >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> FYI, >>> >>>>>>> > > > > > > >>> St.Ack >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >>> >>> >>>>>>> > > > > > > >> >>> >>>>>>> > > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > >>> >>>>>>> > > > > > -- >>> >>>>>>> > > > > > Sean >>> >>>>>>> > > > > > >>> >>>>>>> > > > > >>> >>>>>>> > > > >>> >>>>>>> > > > >>> >>>>>>> > > > >>> >>>>>>> > > > -- >>> >>>>>>> > > > Best regards, >>> >>>>>>> > > > >>> >>>>>>> > > > - Andy >>> >>>>>>> > > > >>> >>>>>>> > > > Problems worthy of attack prove their worth by hitting >>> back. - >>> >>>>>>> > > > Piet >>> >>>>>>> > Hein >>> >>>>>>> > > > (via Tom White) >>> >>>>>>> > > > >>> >>>>>>> > > >>> >>>>>>> > >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> -- >>> >>>>>> Sean >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> Sean >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Sean >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Sean >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> Sean >>> > >>> > >>> > >>> > >>> > -- >>> > Sean >>> >>> >>> >>> -- >>> Sean >>> >> >> -- Sean
