I just ran the tests in hbase-spark module using 'mvn verify'. All passed.
I am testing a patch locally where hbase-spark tests are run in test phase. If the tests pass, I will log a JIRA. Thanks > On Oct 14, 2016, at 11:41 AM, Andrew Purtell <apurt...@apache.org> wrote: > > The hbase-spark integration tests run (and fail) for me locally whenever I > build master with 'mvn clean install -DskipITs' . > > HBaseConnectionCacheSuite: > - all test cases *** FAILED *** > 2 did not equal 1 (HBaseConnectionCacheSuite.scala:92) > > Saw it but had to ignore/triage to get something else done. > > We have a weird situation where integration tests run when they shouldn't > locally yet no tests run at all for patch process? > > I would like to see Spark behave like the other modules. I remember filing > a JIRA asking that hbase-spark honor -DskipITs. It still doesn't. > Meanwhile, it does its own thing with '-DskipSparkTests', which is not > appropriate given that none of the other modules have their own distinct > control parameters. There also doesn't seem to be a distinction between > unit tests and integration tests. The 'test' target does nothing. > Everything happens during the 'integration-test' phase. Is this a Spark > limitation? > > >> On Fri, Oct 14, 2016 at 11:27 AM, Sean Busbey <bus...@cloudera.com> wrote: >> >> Do the HBase Spark tests only run during the maven verify command? >> We'll need to update our personality to say that that command should >> be used for unit tests when in the hbase spark module. ugh. >> >> On Thu, Oct 13, 2016 at 7:42 PM, Apekshit Sharma <a...@cloudera.com> >> wrote: >>> Our patch process isn't running hbase-spark tests. See this for example: >>> >>> https://builds.apache.org/job/PreCommit-HBASE-Build/3842/ >>> https://builds.apache.org/job/PreCommit-HBASE-Build/3842/ >> artifact/patchprocess/patch-unit-hbase-spark.txt/*view*/ >>> >>> Found it when trying to debug cause of trunk failures. Part of the cause >> is >>> hbase-spark's HBaseConnectionCacheSuite test failure ( >>> https://builds.apache.org/view/All/job/HBase-Trunk_ >> matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1776/consoleFull >> <https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20%28latest%29,label=yahoo-not-h2/1776/consoleFull> >> ) >>> which was added in HBASE-16638. However, to be fair, QA was green and >>> reported passing hbase-spark tests for that jira. >>> >>>> On Mon, Sep 19, 2016 at 12:57 PM, Stack <st...@duboce.net> wrote: >>>> >>>> childCustomWorkspace seems to be just the ticket. Nice find Appy. >>>> St.Ack >>>> >>>> On Mon, Sep 19, 2016 at 10:03 AM, Sean Busbey <bus...@cloudera.com> >> wrote: >>>> >>>>> Option 2c looks to be working really well. Thanks for tackling this >> Appy! >>>>> >>>>> We still have some failures on the master build, but it looks like >>>>> actual problems (or perhaps a flakey). There are several passing >>>>> builds. >>>>> >>>>> This should be pretty easy to replicate on the other jobs. I don't see >>>>> a downside. Anyone else have concerns? >>>>> >>>>> >>>>> On Fri, Sep 16, 2016 at 6:15 PM, Apekshit Sharma <a...@cloudera.com> >>>>> wrote: >>>>>> So this all started with spaces-in-path issue, right? I think it >> has >>>>>> gobbled up a lot of time of a lot of people. >>>>>> Let's discuss our options and try to fix it for good. Here are what >> i >>>> can >>>>>> think of, and my opinion about them. >>>>>> >>>>>> 1. Not use matrix build >>>>>> Temporary fix. Not preferred since not applicable to other >>>>>> branches' builds. >>>>>> >>>>>> 2. Use matrix build >>>>>> >>>>>> a. Use tool environment trick >>>>>> I applied this few days ago. Seemed to work until we >>>>> discovered >>>>>> scalatest issue. While the solution looks legitimate, we can't trust >>>> that >>>>>> all tools will use JAVA_HOME instead of directly using java command. >>>>>> >>>>>> b. Use JDK axix >>>>>> Doesn't work right now. I don't have much idea of what's >> the >>>>> cost >>>>>> for fixing it. >>>>>> >>>>>> c. Use JDK axis with custom child workspace >>>>>> >>>>>> https://github.com/jenkinsci/matrix-project-plugin/blob/ >>>>> master/src/main/resources/hudson/matrix/MatrixProject/ >>>>> help-childCustomWorkspace.html >>>>>> Just found this one, and it might solve things for good. I >>>> have >>>>>> updated the job to use this. Let's see how it works. >>>>>> >>>>>> What do others think? >>>>>> >>>>>>> On Fri, Sep 16, 2016 at 3:31 PM, Stack <st...@duboce.net> wrote: >>>>>>> >>>>>>> The profile (or define) skipSparkTests looks like it will skip >> spark >>>>> tests. >>>>>>> Setting skipIntegrationTests to true will skip it. >>>>>>> S >>>>>>> >>>>>>> On Fri, Sep 16, 2016 at 1:40 PM, Dima Spivak < >> dimaspi...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Doesn't seem we need a matrix project for master anymore since >> we're >>>>> just >>>>>>>> doing JDK 8 now, right? Also, it looks like the hbase-spark >>>>>>>> integration-test phase is what's tripping up the build. Why not >> just >>>>>>>> temporarily disable that to unblock testing? >>>>>>>> >>>>>>>> On Friday, September 16, 2016, Apekshit Sharma < >> a...@cloudera.com> >>>>>>> wrote: >>>>>>>> >>>>>>>>> So the issue is, we set JAVA_HOME to jdk8 based on matrix param >>>> and >>>>>>> using >>>>>>>>> tool environment. Since mvn uses the env variable, it compiles >>>> with >>>>> jdk >>>>>>>> 8. >>>>>>>>> But i suspect that scalatest isn't using the env variable, >> instead >>>>> it >>>>>>>> might >>>>>>>>> be directly using 'java' cmd, which can be jdk 7 or 8, and can >>>> vary >>>>> by >>>>>>>>> machine. >>>>>>>>> Build succeed if 'java' points to jdk 8, otherwise fails. >>>>>>>>> Note that we didn't have this issue earlier since we were using >>>>> jenkins >>>>>>>>> 'JDK' axis which would set the 'java' to the appropriate >> version. >>>>> But >>>>>>>> that >>>>>>>>> methods had spaces-in-path issue, so i had to change it. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 16, 2016 at 3:46 AM, aman poonia < >>>>> aman.poonia...@gmail.com >>>>>>>>> <javascript:;>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I am not sure if this will help. But it looks like it is >> because >>>>> of >>>>>>>>> version >>>>>>>>>> mismatch, that is, it is expecting JDK1.7 and we are >> compiling >>>>> with >>>>>>>>> jdk1.8. >>>>>>>>>> That means there is some library which has to be compiled >> with >>>>> jdk8 >>>>>>> or >>>>>>>>>> needs to be updated to a jdk8 compatible version. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *With Regards:-* >>>>>>>>>> *Aman Poonia* >>>>>>>>>> >>>>>>>>>> On Fri, Sep 16, 2016 at 2:40 AM, Apekshit Sharma < >>>>> a...@cloudera.com >>>>>>>>> <javascript:;>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> And....everything is back to red. >>>>>>>>>>> Because something is plaguing our builds again. :( >>>>>>>>>>> >>>>>>>>>>> If anyone knows what's problem in this case, please reply >> on >>>>> this >>>>>>>>> thread, >>>>>>>>>>> otherwise i'll try to fix it later sometime today. >>>>>>>>>>> >>>>>>>>>>> [INFO] *--- scalatest-maven-plugin:1.0:test >>>> (integration-test) >>>>> @ >>>>>>>>>>> hbase-spark --- >>>>>>>>>>> * [36mDiscovery starting. [0m >>>>>>>>>>> [31m*** RUN ABORTED *** [0m >>>>>>>>>>> [31m java.lang.UnsupportedClassVersionError: >>>>>>>>>>> org/apache/hadoop/hbase/spark/example/hbasecontext/ >>>>>>>>>>> JavaHBaseDistributedScan >>>>>>>>>>> : Unsupported major.minor version 52.0 [0m >>>>>>>>>>> [31m at java.lang.ClassLoader.defineClass1(Native >> Method) >>>> [0m >>>>>>>>>>> [31m at java.lang.ClassLoader. >> defineClass(ClassLoader.java: >>>>> 803) >>>>>>>> [0m >>>>>>>>>>> [31m at java.security.SecureClassLoader.defineClass( >>>>>>>>>>> SecureClassLoader.java:142) >>>>>>>>>>> [0m >>>>>>>>>>> [31m at java.net.URLClassLoader. >> defineClass(URLClassLoader. >>>>>>>> java:449) >>>>>>>>>> [0m >>>>>>>>>>> [31m at java.net.URLClassLoader. >> access$100(URLClassLoader. >>>>>>> java:71) >>>>>>>>> [0m >>>>>>>>>>> [31m at java.net.URLClassLoader$1.run( >>>>> URLClassLoader.java:361) >>>>>>> [0m >>>>>>>>>>> [31m at java.net.URLClassLoader$1.run( >>>>> URLClassLoader.java:355) >>>>>>> [0m >>>>>>>>>>> [31m at java.security.AccessController.doPrivileged( >> Native >>>>>>> Method) >>>>>>>>> [0m >>>>>>>>>>> [31m at java.net.URLClassLoader. >>>>> findClass(URLClassLoader.java: >>>>>>> 354) >>>>>>>>> [0m >>>>>>>>>>> [31m at java.lang.ClassLoader. >> loadClass(ClassLoader.java: >>>> 425) >>>>>>> [0m >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Sep 12, 2016 at 5:01 PM, Mikhail Antonov < >>>>>>>> olorinb...@gmail.com >>>>>>>>> <javascript:;>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Great work indeed! >>>>>>>>>>>> >>>>>>>>>>>> Agreed, occasional failed runs may not be that bad, but >>>> fairly >>>>>>>>> regular >>>>>>>>>>>> failed runs ruin the idea of CI. Especially for released >> or >>>>>>>> otherwise >>>>>>>>>>>> supposedly stable branches. >>>>>>>>>>>> >>>>>>>>>>>> -Mikhail >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 12, 2016 at 4:53 PM, Sean Busbey < >>>>>>> bus...@cloudera.com >>>>>>>>> <javascript:;>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> awesome work Appy! >>>>>>>>>>>>> >>>>>>>>>>>>> That's certainly good news to hear. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Sep 12, 2016 at 2:14 PM, Apekshit Sharma < >>>>>>>>> a...@cloudera.com <javascript:;>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> On a separate note: >>>>>>>>>>>>>> Trunk had 8 green runs in last 3 days! ( >>>>>>>>>>>>>> https://builds.apache.org/job/HBase-Trunk_matrix/) >>>>>>>>>>>>>> This was due to fixing just the mass failures on >> trunk >>>>> and no >>>>>>>>>> change >>>>>>>>>>> in >>>>>>>>>>>>>> flaky infra. Which made me to conclude two things: >>>>>>>>>>>>>> 1. Flaky infra works. >>>>>>>>>>>>>> 2. It relies heavily on the post-commit build's >>>> stability >>>>>>>> (which >>>>>>>>>>> every >>>>>>>>>>>>>> project should anyways strive for). If the build >> fails >>>>>>>>>>> catastrophically >>>>>>>>>>>>>> once in a while, we can just exclude that one run >> using >>>> a >>>>>>> flag >>>>>>>>> and >>>>>>>>>>>>>> everything will work, but if it happens frequently, >> then >>>>> it >>>>>>>> won't >>>>>>>>>>> work >>>>>>>>>>>>>> right. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have re-enabled Flaky tests job ( >>>>>>>>>>>>>> https://builds.apache.org/ >>>> view/H-L/view/HBase/job/HBASE- >>>>>>>>>> Flaky-Tests/ >>>>>>>>>>> ) >>>>>>>>>>>>> which >>>>>>>>>>>>>> was disabled for almost a month due to trunk being on >>>>> fire. >>>>>>>>>>>>>> I will keep an eye on how things are going. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Sep 12, 2016 at 2:02 PM, Apekshit Sharma < >>>>>>>>>> a...@cloudera.com <javascript:;>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> @Sean, Mikhail: I found the alternate solution. >> Using >>>>> user >>>>>>>>> defined >>>>>>>>>>>> axis, >>>>>>>>>>>>>>> tool environment and env variable injection. >>>>>>>>>>>>>>> See latest diff to https://builds.apache.org/job/ >>>>>>>>>>> HBase-Trunk_matrix/ >>>>>>>>>>>>> job >>>>>>>>>>>>>>> for reference. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Aug 30, 2016 at 7:39 PM, Mikhail Antonov < >>>>>>>>>>>> olorinb...@gmail.com <javascript:;>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> FYI, I did the same for branch-1.3 builds. I've >>>>> disabled >>>>>>>>>> hbase-1.3 >>>>>>>>>>>> and >>>>>>>>>>>>>>>> hbase-1.3-IT jobs and instead created >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://builds.apache.org/job/HBase-1.3-JDK8 and >>>>>>>>>>>>>>>> https://builds.apache.org/job/HBase-1.3-JDK7 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This should work for now until we figure out how to >>>> move >>>>>>>>> forward. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Mikhail >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Aug 17, 2016 at 1:41 PM, Sean Busbey < >>>>>>>>>> bus...@cloudera.com <javascript:;>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> /me smacks forehead >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> these replacement jobs, of course, also have >> special >>>>>>>>> characters >>>>>>>>>>> in >>>>>>>>>>>>>>>>> their names which then show up in the working >> path. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> renaming them to skip spaces and parens. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Aug 17, 2016 at 1:34 PM, Sean Busbey < >>>>>>>>>>>> sean.bus...@gmail.com <javascript:;>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> FYI, it looks like essentially our entire CI >> suite >>>>> is >>>>>>>> red, >>>>>>>>>>>> probably >>>>>>>>>>>>>>>> due >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> parts of our codebase not tolerating spaces or >>>> other >>>>>>>>> special >>>>>>>>>>>>>>>> characters >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the working directory. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I've made a stop-gap non-multi-configuration >> set >>>> of >>>>>>> jobs >>>>>>>>> for >>>>>>>>>>>>> running >>>>>>>>>>>>>>>> unit >>>>>>>>>>>>>>>>>> tests for the 1.2 branch against JDK 7 and JDK >> 8: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/ >>>>>>> view/H-L/view/HBase/job/HBase% >>>>>>>>>>>>>>>>> 201.2%20(JDK%201.7)/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/ >>>>>>> view/H-L/view/HBase/job/HBase% >>>>>>>>>>>>>>>>> 201.2%20(JDK%201.8)/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Due to the lack of response from infra@ I >> suspect >>>>> our >>>>>>>> only >>>>>>>>>>>> options >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> continuing on ASF infra is to fix whatever >> part of >>>>> our >>>>>>>>> build >>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>> tolerate the new paths, or stop using >>>>>>> multiconfiguration >>>>>>>>>>>>> deployments. >>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>> obviously less than thrilled at the idea of >> having >>>>>>>> several >>>>>>>>>>>>> multiples >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> current jobs. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Aug 10, 2016 at 6:28 PM, Sean Busbey < >>>>>>>>>>>> bus...@cloudera.com <javascript:;>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Ugh. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I sent a reply to Gav on builds@ about maybe >>>>> getting >>>>>>>>> names >>>>>>>>>>> that >>>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>>>> have spaces in them: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread.html/ >>>>>>>>>>>>> 8ac03dc62f9d6862d4f3d5eb37119c >>>>>>>>>>>>>>>>>>> 9c73b4059aaa3ebba52fc63bb6@%3C >> builds.apache.org >>>> %3E >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> In the mean time, is this an issue we need >> file >>>>> with >>>>>>>>> Hadoop >>>>>>>>>> or >>>>>>>>>>>>>>>>>>> something we need to fix in our own code? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, Aug 10, 2016 at 6:04 PM, Matteo >> Bertozzi >>>>>>>>>>>>>>>>>>> <theo.berto...@gmail.com <javascript:;>> >> wrote: >>>>>>>>>>>>>>>>>>>> There are a bunch of builds that have most >> of >>>> the >>>>>>> test >>>>>>>>>>>> failing. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Example: >>>>>>>>>>>>>>>>>>>> https://builds.apache.org/job/ >>>>>>>>>> HBase-Trunk_matrix/1392/jdk= >>>>>>>>>>>>>>>>>>> JDK%201.7%20(latest),label= >>>>>>>> yahoo-not-h2/testReport/junit/ >>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase/ >> TestLocalHBaseCluster/ >>>>>>>>>>>>> testLocalHBaseCluster/ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> from the stack trace looks like the problem >> is >>>>> with >>>>>>>> the >>>>>>>>>> jdk >>>>>>>>>>>> name >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>> spaces: >>>>>>>>>>>>>>>>>>>> the hadoop FsVolumeImpl calls >>>> setNameFormat(... + >>>>>>>>>>>>>>>> fileName.toString() >>>>>>>>>>>>>>>>> + >>>>>>>>>>>>>>>>>>> ...) >>>>>>>>>>>>>>>>>>>> and this seems to not be escaped >>>>>>>>>>>>>>>>>>>> so we end up with JDK%25201.7%2520(latest) >> in >>>> the >>>>>>>> string >>>>>>>>>>>> format >>>>>>>>>>>>>>>> and we >>>>>>>>>>>>>>>>>>> get >>>>>>>>>>>>>>>>>>>> a IllegalFormatPrecisionException: 7 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 2016-08-10 22:07:46,108 WARN [DataNode: >>>>>>>>>>>>>>>>>>>> [[[DISK]file:/home/jenkins/ >>>>>>>>> jenkins-slave/workspace/HBase- >>>>>>>>>>>>>>>>>>> Trunk_matrix/jdk/JDK%25201.7% >>>>>>>>> 2520(latest)/label/yahoo-not- >>>>>>>>>>>>>>>>>>> h2/hbase-server/target/test- >>>>>>>> data/e7099624-ecfa-4674-87de- >>>>>>>>>>>>>>>>>>> a8733d13b582/dfscluster_ >> 10fdcfc3-cd1b-45be-9b5a- >>>>>>>>>>>>>>>>>>> 9c88f385e6f1/dfs/data/data1/, >>>>>>>>>>>>>>>>>>>> [DISK]file:/home/jenkins/ >>>>>>>> jenkins-slave/workspace/HBase- >>>>>>>>>>>>>>>>>>> Trunk_matrix/jdk/JDK%25201.7% >>>>>>>>> 2520(latest)/label/yahoo-not- >>>>>>>>>>>>>>>>>>> h2/hbase-server/target/test- >>>>>>>> data/e7099624-ecfa-4674-87de- >>>>>>>>>>>>>>>>>>> a8733d13b582/dfscluster_ >> 10fdcfc3-cd1b-45be-9b5a- >>>>>>>>>>>>>>>>>>> 9c88f385e6f1/dfs/data/data2/]] >>>>>>>>>>>>>>>>>>>> heartbeating to localhost/127.0.0.1:34629] >>>>>>>>>>>>>>>>>>>> datanode.BPServiceActor(831): Unexpected >>>>> exception >>>>>>> in >>>>>>>>>> block >>>>>>>>>>>> pool >>>>>>>>>>>>>>>> Block >>>>>>>>>>>>>>>>>>>> pool <registering> (Datanode Uuid >> unassigned) >>>>>>> service >>>>>>>> to >>>>>>>>>>>>>>>>>>>> localhost/127.0.0.1:34629 >>>>>>>>>>>>>>>>>>>> java.util.IllegalFormatPrecisionException: >> 7 >>>>>>>>>>>>>>>>>>>> at java.util.Formatter$ >>>>>>>>> FormatSpecifier.checkText( >>>>>>>>>>>>>>>>>>> Formatter.java:2984) >>>>>>>>>>>>>>>>>>>> at java.util.Formatter$ >>>>>>>> FormatSpecifier.<init>( >>>>>>>>>>>>>>>>>>> Formatter.java:2688) >>>>>>>>>>>>>>>>>>>> at java.util.Formatter.parse( >>>>>>>>> Formatter.java:2528) >>>>>>>>>>>>>>>>>>>> at java.util.Formatter.format( >>>>>>>>>> Formatter.java:2469) >>>>>>>>>>>>>>>>>>>> at java.util.Formatter.format( >>>>>>>>>> Formatter.java:2423) >>>>>>>>>>>>>>>>>>>> at java.lang.String.format( >>>>>>> String.java:2792) >>>>>>>>>>>>>>>>>>>> at com.google.common.util. >> concurrent. >>>>>>>>>>>>> ThreadFactoryBuilder. >>>>>>>>>>>>>>>>>>> setNameFormat(ThreadFactoryBuilder.java:68) >>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.hdfs.server. >>>>>>>>>>>>> datanode.fsdataset.impl. >>>>>>>>>>>>>>>>>>> FsVolumeImpl.initializeCacheExecutor( >>>>>>>>> FsVolumeImpl.java:140) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Matteo >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, Aug 9, 2016 at 9:55 AM, Stack < >>>>>>>> st...@duboce.net >>>>>>>>> <javascript:;>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Good on you Sean. >>>>>>>>>>>>>>>>>>>>> S >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Mon, Aug 8, 2016 at 9:43 PM, Sean >> Busbey < >>>>>>>>>>>> bus...@apache.org <javascript:;> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I updated all of our jobs to use the >> updated >>>>> JDK >>>>>>>>>> versions >>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> infra. >>>>>>>>>>>>>>>>>>>>>> These have spaces in the names, and those >>>>> names >>>>>>> end >>>>>>>>> up >>>>>>>>>> in >>>>>>>>>>>> our >>>>>>>>>>>>>>>>>>>>>> workspace path, so try to keep an eye >> out. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, Aug 8, 2016 at 10:42 AM, Sean >>>> Busbey < >>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> running in docker is the default now. >>>>> relying >>>>>>> on >>>>>>>>> the >>>>>>>>>>>>> default >>>>>>>>>>>>>>>>> docker >>>>>>>>>>>>>>>>>>>>>>> image that comes with Yetus means that >> our >>>>>>> protoc >>>>>>>>>>> checks >>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>> failing[1]. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [1]: https://issues.apache.org/ >>>>>>>>>> jira/browse/HBASE-16373 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Sat, Aug 6, 2016 at 5:03 PM, Sean >>>> Busbey >>>>> < >>>>>>>>>>>>>>>> bus...@apache.org <javascript:;>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> Hi folks! >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> this morning I merged the patch that >>>>> updates >>>>>>> us >>>>>>>> to >>>>>>>>>>> Yetus >>>>>>>>>>>>>>>>> 0.3.0[1] >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> updated the precommit job appropriately. >> I >>>>> also >>>>>>>>> changed >>>>>>>>>>> it >>>>>>>>>>>> to >>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>> the Java versions post the puppet >> changes to >>>>> asf >>>>>>>>> build. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> The last three builds look normal >> (#2975 >>>> - >>>>>>>> #2977). >>>>>>>>>> I'm >>>>>>>>>>>>> gonna >>>>>>>>>>>>>>>> try >>>>>>>>>>>>>>>>>>>>>> running things in docker next. I'll email >>>>> again >>>>>>>> when >>>>>>>>> I >>>>>>>>>>> make >>>>>>>>>>>>> it >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> default. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [1]: https://issues.apache.org/ >>>>>>>>>>> jira/browse/HBASE-15882 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 2016-06-16 10:43 (-0500), Sean >> Busbey >>>> < >>>>>>>>>>>>> bus...@apache.org <javascript:;>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> FYI, today our precommit jobs started >>>>> failing >>>>>>>>>> because >>>>>>>>>>>> our >>>>>>>>>>>>>>>>> chosen >>>>>>>>>>>>>>>>>>> jdk >>>>>>>>>>>>>>>>>>>>>>>>> (1.7.0.79) disappeared (mentioned on >>>>>>>>> HBASE-16032). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Initially we were doing something >> wrong, >>>>>>> namely >>>>>>>>>>>> directly >>>>>>>>>>>>>>>>>>> referencing >>>>>>>>>>>>>>>>>>>>>>>>> the jenkins build tools area without >>>>> telling >>>>>>>>>> jenkins >>>>>>>>>>> to >>>>>>>>>>>>> give >>>>>>>>>>>>>>>>> us an >>>>>>>>>>>>>>>>>>>>> env >>>>>>>>>>>>>>>>>>>>>>>>> variable that stated where the jdk is >>>>>>> located. >>>>>>>>>>> However, >>>>>>>>>>>>>>>> after >>>>>>>>>>>>>>>>>>>>>>>>> attempting to switch to the >> appropriate >>>>>>> tooling >>>>>>>>>>>> variable >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> jdk >>>>>>>>>>>>>>>>>>>>>>>>> 1.7.0.79, I found that it didn't >> point >>>> to >>>>> a >>>>>>>> place >>>>>>>>>>> that >>>>>>>>>>>>>>>> worked. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I've now updated the job to rely on >> the >>>>>>> latest >>>>>>>>> 1.7 >>>>>>>>>>> jdk, >>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> currently 1.7.0.80. I don't know how >>>> often >>>>>>>>> "latest" >>>>>>>>>>>>> updates. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Personally, I think this is a sign >> that >>>> we >>>>>>> need >>>>>>>>> to >>>>>>>>>>>>>>>> prioritize >>>>>>>>>>>>>>>>>>>>>>>>> HBASE-15882 so that we can switch >> back >>>> to >>>>>>> using >>>>>>>>>>>> Docker. I >>>>>>>>>>>>>>>> won't >>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>> time this week, so if anyone else >> does >>>>> please >>>>>>>>> pick >>>>>>>>>> up >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> ticket. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Mar 17, 2016 at 5:19 PM, >> Stack < >>>>>>>>>>>> st...@duboce.net <javascript:;> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Sean. >>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 12:04 PM, >> Sean >>>>>>>> Busbey < >>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I updated the precommit job >>>> today >>>>> to >>>>>>>>>> specify >>>>>>>>>>>> that >>>>>>>>>>>>>>>> only >>>>>>>>>>>>>>>>>>>>> compile >>>>>>>>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>>>>>>>>>>>> checks should be done against jdks >>>>> other >>>>>>>> than >>>>>>>>>> the >>>>>>>>>>>>> primary >>>>>>>>>>>>>>>>> jdk7 >>>>>>>>>>>>>>>>>>>>>> instance. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 7, 2016 at 8:43 PM, >> Sean >>>>>>> Busbey >>>>>>>> < >>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I tested things out, and while >>>>>>>> YETUS-297[1] >>>>>>>>> is >>>>>>>>>>>>> present >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> default runs >>>>>>>>>>>>>>>>>>>>>>>>>>>> all plugins that can do multiple >>>> jdks >>>>>>>>> against >>>>>>>>>>>> those >>>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>>>>>>>> (jdk7 and >>>>>>>>>>>>>>>>>>>>>>>>>>>> jdk8 in our case). >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> We can configure things to only >> do >>>> a >>>>>>>> single >>>>>>>>>> run >>>>>>>>>>> of >>>>>>>>>>>>> unit >>>>>>>>>>>>>>>>>>> tests. >>>>>>>>>>>>>>>>>>>>>> They'll be >>>>>>>>>>>>>>>>>>>>>>>>>>>> against jdk7, since that is our >>>>> default >>>>>>>> jdk. >>>>>>>>>>> That >>>>>>>>>>>>> fine >>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>> everyone? It'll >>>>>>>>>>>>>>>>>>>>>>>>>>>> save ~1.5 hours on any build >> that >>>>> hits >>>>>>>>>>>> hbase-server. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 7, 2016 at 1:22 PM, >>>>> Stack < >>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hurray! >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks like YETUS-96 is in >> there >>>>> and >>>>>>> we >>>>>>>>> are >>>>>>>>>>>> only >>>>>>>>>>>>>>>>> running >>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>> jdk build >>>>>>>>>>>>>>>>>>>>>>>>>>>>> now, the default (but testing >>>>> compile >>>>>>>>> against >>>>>>>>>>>>>>>> both).... >>>>>>>>>>>>>>>>> Will >>>>>>>>>>>>>>>>>>>>>> keep an >>>>>>>>>>>>>>>>>>>>>>>>>>> eye. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 7, 2016 at 10:27 >> AM, >>>>> Sean >>>>>>>>> Busbey >>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I've just updated our >>>>> precommit >>>>>>>> jobs >>>>>>>>>> to >>>>>>>>>>>> use >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> 0.2.0 >>>>>>>>>>>>>>>>>>>>>> release of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yetus >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that came out today. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After keeping an eye out for >>>>>>>> strangeness >>>>>>>>>>> today >>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>> turn >>>>>>>>>>>>>>>>>>>>>> docker mode >>>>>>>>>>>>>>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on by default tonight. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jan 13, 2016 at 10:14 >>>> AM, >>>>>>> Sean >>>>>>>>>>> Busbey < >>>>>>>>>>>>>>>>>>>>>> bus...@apache.org <javascript:;>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I added a new >> parameter >>>> to >>>>> the >>>>>>>>>>> precommit >>>>>>>>>>>>> job: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * USE_YETUS_PRERELEASE - >>>> causes >>>>> us >>>>>>> to >>>>>>>>> use >>>>>>>>>>> the >>>>>>>>>>>>>>>> HEAD of >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> apache/yetus >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repo rather than our chosen >>>>> release >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It defaults to inactive, >> but >>>>> can be >>>>>>>>> used >>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>> manually-triggered runs >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> test a solution to a >> problem >>>> in >>>>> the >>>>>>>>> yetus >>>>>>>>>>>>>>>> library. At >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> moment, >>>>>>>>>>>>>>>>>>>>>>>>>>> I'm >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using it to test a >> solution to >>>>>>>> default >>>>>>>>>>> module >>>>>>>>>>>>>>>>> ordering >>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>> seen in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HBASE-15075. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 8, 2016 at 7:58 >>>> AM, >>>>>>> Sean >>>>>>>>>>> Busbey < >>>>>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I just pushed >>>> HBASE-13525 >>>>>>>>> (switch >>>>>>>>>> to >>>>>>>>>>>>> Apache >>>>>>>>>>>>>>>>> Yetus >>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>> precommit >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and updated our jenkins >>>>> precommit >>>>>>>>> build >>>>>>>>>>> to >>>>>>>>>>>>> use >>>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jenkins job has some >>>>> explanation: >>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/ >>>>>>>>>>>>> view/PreCommit%20Builds/job/ >>>>>>>>>>>>>>>>>>>>>> PreCommit-HBASE-Build/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Release note from >>>> HBASE-13525 >>>>>>> does >>>>>>>> as >>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The old job will stick >>>> around >>>>>>> here >>>>>>>>> for >>>>>>>>>> a >>>>>>>>>>>>> couple >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> weeks, >>>>>>>>>>>>>>>>>>>>>> in case >>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to refer back to it: >>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/ >>>>>>>>>>>>> view/PreCommit%20Builds/job/ >>>>>>>>>>>>>>>>>>>>>> PreCommit-HBASE-Build-deprecated/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If something looks awry, >>>>> please >>>>>>>> drop >>>>>>>>> a >>>>>>>>>>> note >>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>> HBASE-13525 >>>>>>>>>>>>>>>>>>>>>> while >>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> remains >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open (and make a new >> issue >>>>>>> after). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Dec 2, 2015 at >> 3:22 >>>>> PM, >>>>>>>>> Stack < >>>>>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As part of my continuing >>>>>>> advocacy >>>>>>>> of >>>>>>>>>>>>>>>>>>> builds.apache.org >>>>>>>>>>>>>>>>>>>>>> and that >>>>>>>>>>>>>>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> results are now worthy >> of >>>> our >>>>>>>> trust >>>>>>>>>> and >>>>>>>>>>>>>>>> nurture, >>>>>>>>>>>>>>>>> here >>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> highlights >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the last few days >> of >>>>>>> builds: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + hadoopqa is now >> finding >>>>>>> zombies >>>>>>>>>> before >>>>>>>>>>>> the >>>>>>>>>>>>>>>>> patch is >>>>>>>>>>>>>>>>>>>>>> committed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HBASE-14888 showed "-1 >> core >>>>>>> tests. >>>>>>>>> The >>>>>>>>>>>> patch >>>>>>>>>>>>>>>>> failed >>>>>>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>>>>> unit >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests:" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't have any failed >>>> tests >>>>>>>> listed >>>>>>>>>> (I'm >>>>>>>>>>>>>>>> trying to >>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>> I can >>>>>>>>>>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> anything >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about this...). Running >> our >>>>>>> little >>>>>>>>>>>>>>>>>>>>>>>>>>> ./dev-tools/findHangingTests.py >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> against >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the consoleText, it >> showed >>>> a >>>>>>>> hanging >>>>>>>>>>> test. >>>>>>>>>>>>>>>> Running >>>>>>>>>>>>>>>>>>>>>> locally, I see >>>>>>>>>>>>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hang. This is before the >>>>> patch >>>>>>>>> landed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + Our branch runs are >> now >>>>> near >>>>>>>>> totally >>>>>>>>>>>>> zombie >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> flakey >>>>>>>>>>>>>>>>>>>>>> free -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work to do -- but a >> recent >>>>> patch >>>>>>>>> that >>>>>>>>>>>> seemed >>>>>>>>>>>>>>>>> harmless >>>>>>>>>>>>>>>>>>>>> was >>>>>>>>>>>>>>>>>>>>>>>>>>> causing a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reliable flake fail in >> the >>>>>>>> backport >>>>>>>>> to >>>>>>>>>>>>>>>> branch-1* >>>>>>>>>>>>>>>>>>>>>> confirmed by >>>>>>>>>>>>>>>>>>>>>>>>>>> local >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The flakeyness was >> plain to >>>>> see >>>>>>> up >>>>>>>>> in >>>>>>>>>>>>>>>>>>> builds.apache.org >>>>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + In the last few days >> I've >>>>>>>>> committed >>>>>>>>>> a >>>>>>>>>>>>> patch >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>> included >>>>>>>>>>>>>>>>>>>>>>>>>>> javadoc >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warnings even though >>>> hadoopqa >>>>>>> said >>>>>>>>> the >>>>>>>>>>>> patch >>>>>>>>>>>>>>>>>>> introduced >>>>>>>>>>>>>>>>>>>>>> javadoc >>>>>>>>>>>>>>>>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (I >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> missed it). This messed >> up >>>>> life >>>>>>>> for >>>>>>>>>>> folks >>>>>>>>>>>>>>>>>>> subsequently >>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patches >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reported javadoc >> issues.... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In short, I suggest that >>>>>>>>>>>> builds.apache.org >>>>>>>>>>>>> is >>>>>>>>>>>>>>>>> worth >>>>>>>>>>>>>>>>>>>>>> keeping an >>>>>>>>>>>>>>>>>>>>>>>>>>> eye >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sure you get a clean >> build >>>>> out >>>>>>> of >>>>>>>>>>> hadoopqa >>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>>>>> committing >>>>>>>>>>>>>>>>>>>>>>>>>>>>> anything, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lets all work together >> to >>>> try >>>>>>> and >>>>>>>>> keep >>>>>>>>>>> our >>>>>>>>>>>>>>>> builds >>>>>>>>>>>>>>>>>>> blue: >>>>>>>>>>>>>>>>>>>>>> it'll >>>>>>>>>>>>>>>>>>>>>>>>>>> save >>>>>>>>>>>>>>>>>>>>>>>>>>>>> us >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work in the long run. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Nov 4, 2014 at >> 9:38 >>>>> AM, >>>>>>>>> Stack >>>>>>>>>> < >>>>>>>>>>>>>>>>>>> st...@duboce.net <javascript:;> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Branch-1 and master >> have >>>>>>>>> stabilized >>>>>>>>>>> and >>>>>>>>>>>>> now >>>>>>>>>>>>>>>> run >>>>>>>>>>>>>>>>>>> mostly >>>>>>>>>>>>>>>>>>>>>> blue >>>>>>>>>>>>>>>>>>>>>>>>>>>>> (give or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the odd failure) >> [1][2]. >>>>>>> Having >>>>>>>> a >>>>>>>>>>> mostly >>>>>>>>>>>>> blue >>>>>>>>>>>>>>>>>>> branch-1 >>>>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>>>>>> helped us >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> identify at least one >>>>>>>>> destabilizing >>>>>>>>>>>>> commit in >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> last >>>>>>>>>>>>>>>>>>>>>> few >>>>>>>>>>>>>>>>>>>>>>>>>>> days, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two; >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is as it should >> be >>>>>>> (smile). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lets keep our builds >>>> blue. >>>>> If >>>>>>>> you >>>>>>>>>>>> commit a >>>>>>>>>>>>>>>>> patch, >>>>>>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>>>>>>>> sure >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> subsequent >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> builds stay blue. You >> can >>>>>>>>> subscribe >>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> bui...@hbase.apache.org <javascript:;> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> notice of failures if >> not >>>>>>>> already >>>>>>>>>>>>> subscribed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. >>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/ >>>>>>>>>>>>> view/H-L/view/HBase/job/HBase- >>>>>>>>>>>>>>>>> 1.0/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/view >>>>>>>>>>>>>>>> /H-L/view/HBase/job/HBase- >>>>>>>>>>>>>>>>>>> TRUNK/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 13, 2014 >> at >>>>> 4:41 >>>>>>> PM, >>>>>>>>>>> Stack < >>>>>>>>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A few notes on >> testing. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Too long to read, >> infra >>>> is >>>>>>> more >>>>>>>>>>> capable >>>>>>>>>>>>> now >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> after >>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>> work, we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seeing branch-1 and >>>> trunk >>>>>>>> mostly >>>>>>>>>>>> running >>>>>>>>>>>>>>>> blue. >>>>>>>>>>>>>>>>>>> Lets >>>>>>>>>>>>>>>>>>>>>> try and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> keep it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> way going forward. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Infra has new, >>>> more >>>>>>>>> capable >>>>>>>>>>>>> hardware. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A recent spurt of >> test >>>>> fixing >>>>>>>>>>> combined >>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>> capable >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hardware >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have gotten us to >> a >>>> new >>>>>>>> place; >>>>>>>>>>> tests >>>>>>>>>>>>> are >>>>>>>>>>>>>>>>> mostly >>>>>>>>>>>>>>>>>>>>>> passing now >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch-1 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and master. Lets try >>>> and >>>>>>> keep >>>>>>>> it >>>>>>>>>>> this >>>>>>>>>>>>> way >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>>>>> to trust >>>>>>>>>>>>>>>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> again. Just a few >>>> flakies >>>>>>>>> remain. >>>>>>>>>>>> Lets >>>>>>>>>>>>> try >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> nail >>>>>>>>>>>>>>>>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Our tests now run in >>>>> parallel >>>>>>>>> with >>>>>>>>>>>> other >>>>>>>>>>>>>>>> test >>>>>>>>>>>>>>>>>>> suites >>>>>>>>>>>>>>>>>>>>>> where >>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ran alone. You can >> see >>>>> this >>>>>>>>>> sometimes >>>>>>>>>>>>> when >>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>>>> zombie >>>>>>>>>>>>>>>>>>>>>> detector >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reports >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests from another >>>> project >>>>>>>>>> altogether >>>>>>>>>>>> as >>>>>>>>>>>>>>>>> lingerers >>>>>>>>>>>>>>>>>>>>> (To >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixed). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our tests are failing >>>>>>> because a >>>>>>>>>>>>> concurrent >>>>>>>>>>>>>>>>> hbase >>>>>>>>>>>>>>>>>>> run >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>> undoing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> classes >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data from under it. >>>> Also, >>>>>>> lets >>>>>>>>> fix. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Our tests are >> brittle. >>>> It >>>>>>> takes >>>>>>>>>>>> 75minutes >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> them to >>>>>>>>>>>>>>>>>>>>>>>>>>> complete. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Many >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> heavy-duty >> integration >>>>> tests >>>>>>>>>> starting >>>>>>>>>>>> up >>>>>>>>>>>>>>>>> multiple >>>>>>>>>>>>>>>>>>>>>> clusters and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mapreduce >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all in the one JVM. >> It >>>> is >>>>> a >>>>>>>>> miracle >>>>>>>>>>>> they >>>>>>>>>>>>>>>> pass >>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> all. >>>>>>>>>>>>>>>>>>>>>> Usually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests have been cast >> as >>>>> unit >>>>>>>>> tests >>>>>>>>>>>>> because >>>>>>>>>>>>>>>>> there >>>>>>>>>>>>>>>>>>> was >>>>>>>>>>>>>>>>>>>>>> no where >>>>>>>>>>>>>>>>>>>>>>>>>>>>> else >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to get an airing. We >>>> have >>>>>>> the >>>>>>>>>>> hbase-it >>>>>>>>>>>>>>>> suite >>>>>>>>>>>>>>>>> now >>>>>>>>>>>>>>>>>>>>>> which would >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apt >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> place but until these >>>> are >>>>> run >>>>>>>> on >>>>>>>>> a >>>>>>>>>>>>> regular >>>>>>>>>>>>>>>>> basis >>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>> public for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the fat integration >>>> tests >>>>>>>>> disguised >>>>>>>>>>> as >>>>>>>>>>>>> unit >>>>>>>>>>>>>>>>> tests >>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>> remain. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our current unit >> tests >>>>>>> weeding >>>>>>>>> the >>>>>>>>>>> old >>>>>>>>>>>>> cruft >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> no longer >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> relevant >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> duplicates would be a >>>> nice >>>>>>>>>>> undertaking >>>>>>>>>>>> if >>>>>>>>>>>>>>>>> someone >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> looking >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> contribute. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alex Newman has been >>>>> working >>>>>>> on >>>>>>>>>>> making >>>>>>>>>>>>> our >>>>>>>>>>>>>>>>> tests >>>>>>>>>>>>>>>>>>> work >>>>>>>>>>>>>>>>>>>>>> up on >>>>>>>>>>>>>>>>>>>>>>>>>>>>> travis >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> circle-ci. That'll >> be >>>>> sweet >>>>>>>> when >>>>>>>>>> it >>>>>>>>>>>> goes >>>>>>>>>>>>>>>>>>> end-to-end. >>>>>>>>>>>>>>>>>>>>>> He also >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some "type" >>>>> categorizations >>>>>>> -- >>>>>>>>>>> client, >>>>>>>>>>>>>>>> filter, >>>>>>>>>>>>>>>>>>>>>> mapreduce -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> alongside >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> old "sizing" >>>>> categorizations >>>>>>> of >>>>>>>>>>>>>>>>>>> small/medium/large. >>>>>>>>>>>>>>>>>>>>>> His >>>>>>>>>>>>>>>>>>>>>>>>>>>>> thinking >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can run these >>>>>>>> categorizations >>>>>>>>> in >>>>>>>>>>>>> parallel >>>>>>>>>>>>>>>>> so we >>>>>>>>>>>>>>>>>>>>>> could run >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suite in about the >> time >>>> of >>>>>>> the >>>>>>>>>>> longest >>>>>>>>>>>>> test, >>>>>>>>>>>>>>>>> say >>>>>>>>>>>>>>>>>>>>>> 20-30minutes? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> even change Apache to >>>> run >>>>>>> them >>>>>>>>> this >>>>>>>>>>>> way. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sean >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> busbey >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> busbey >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>> busbey >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> busbey >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> busbey >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Sean >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> busbey >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Michael Antonov >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- Appy >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- Appy >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> busbey >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Michael Antonov >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> -- Appy >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> -- Appy >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -Dima >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> -- Appy >>>>> >>>>> >>>>> >>>>> -- >>>>> busbey >>> >>> >>> >>> -- >>> >>> -- Appy >> >> >> >> -- >> busbey > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White)
