No Joy (no time) yet. On Fri, Jan 6, 2012 at 6:59 AM, Grant Ingersoll <[email protected]> wrote:
> Any luck on this, as it is still failing for me on Ubuntu? > > We are slowing whittling down the 0.6 issues, but this one seems critical. > > > On Dec 31, 2011, at 8:07 PM, Ted Dunning wrote: > > > I have replicated this and applied this patch but the test still fails. > > > > I will take a longer look when I get a chance. > > > > On Sat, Dec 31, 2011 at 3:57 PM, Ted Dunning <[email protected]> > wrote: > > > >> I just took a look and found several files not closed issues as Dmitriy > >> mentioned. There might be a buffering difference on Linux versus make. > >> > >> I have a patch to fix this and will test right now on EC2 if I have > enough > >> time. > >> > >> > >> On Sat, Dec 31, 2011 at 2:25 PM, Grant Ingersoll <[email protected] > >wrote: > >> > >>> Any progress on this? Seems like it is a show stopper for code freeze. > >>> On Dec 29, 2011, at 3:58 AM, Dmitriy Lyubimov wrote: > >>> > >>>> note identical md5 for u200 and u400. > >>>> > >>>> On Thu, Dec 29, 2011 at 12:57 AM, Dmitriy Lyubimov <[email protected] > > > >>> wrote: > >>>>> MD5 sums > >>>>> > >>>>> b8217318a29ef69c58b921013eb019e5 > >>> /tmp/matrix8554072597307396201/A-000000000 > >>>>> 41db088ff74c5efd5b766dba253efc03 > >>> /tmp/matrix8554072597307396201/A-000000200 > >>>>> b8217318a29ef69c58b921013eb019e5 > >>> /tmp/matrix8554072597307396201/A-000000400 > >>>>> 41db088ff74c5efd5b766dba253efc03 > >>> /tmp/matrix8554072597307396201/A-000000600 > >>>>> c8dc2a7df82065b5c1e8284ff23aecc6 > >>> /tmp/matrix8554072597307396201/A-000000800 > >>>>> 83bccdd2fa191e01d34646e2030f0e77 > >>> /tmp/matrix8554072597307396201/B-000000000 > >>>>> 9d6878fb789d61d5453b994ea1a5c6db > >>> /tmp/matrix8554072597307396201/B-000000210 > >>>>> cbdf720b17ce25feb686effd1aa0ebef > >>> /tmp/matrix8554072597307396201/B-000000420 > >>>>> 2f71d6ba6891b242575b5cc6ba1c4358 > >>> /tmp/matrix8554072597307396201/B-000000630 > >>>>> f50b76bb48c8f6a791a6d8206d980492 > >>> /tmp/matrix8554072597307396201/B-000000840 > >>>>> 6bb29ca304889a6c8effff6e5c062dc8 /tmp/matrix8554072597307396201/U-0 > >>>>> 019c881c1d7c5748a1cacb3e0b3e5899 > /tmp/matrix8554072597307396201/U-200 > >>>>> 019c881c1d7c5748a1cacb3e0b3e5899 > /tmp/matrix8554072597307396201/U-400 > >>>>> b84e4b01ffb9d691c87b496f8b4d84ec > /tmp/matrix8554072597307396201/U-600 > >>>>> 6bb29ca304889a6c8effff6e5c062dc8 > /tmp/matrix8554072597307396201/U-770 > >>>>> > >>>>> > >>>>> On Thu, Dec 29, 2011 at 12:52 AM, Ted Dunning <[email protected] > > > >>> wrote: > >>>>>> Thanks. Good hints. > >>>>>> > >>>>>> I will take a look on a linux machine in the next few days. > >>>>>> > >>>>>> On Thu, Dec 29, 2011 at 12:42 AM, Dmitriy Lyubimov < > [email protected] > >>>> wrote: > >>>>>> > >>>>>>> yes. i would venture to say that U computation (or restoration) is > >>>>>>> somehow corrupted starting with 2nd block. at least it looks this > >>> way. > >>>>>>> > >>>>>>> On Thu, Dec 29, 2011 at 12:31 AM, Dmitriy Lyubimov < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>> A-reconstructed difference looks good up to row 399 but starting > at > >>>>>>>> row 400 differences do not add up to 0 anymore (although both > inputs > >>>>>>>> are not 0). > >>>>>>>> > >>>>>>>> So it doesn't look like trivial case of something is not > initialized > >>>>>>>> on top of it. It does seem something to do with blocking mechanism > >>>>>>>> though since apparently 400th row is a boundary of some blocking > >>>>>>>> somewhere, but it is hard for me to see where it fails at this > >>> point. > >>>>>>>> > >>>>>>>> On Thu, Dec 29, 2011 at 12:13 AM, Dmitriy Lyubimov < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>>> oh. it's because the synthetic input has only 4 singular values. > >>>>>>>>> > >>>>>>>>> On Wed, Dec 28, 2011 at 11:56 PM, Dmitriy Lyubimov < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>>>> But it is not a problem reading U or V files, that's indeed what > >>> U and > >>>>>>>>>> V contain. > >>>>>>>>>> > >>>>>>>>>> On Wed, Dec 28, 2011 at 11:49 PM, Dmitriy Lyubimov < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>>>>> U and V look suspect, degenerate (only 4 first columns are > >>> nonzero, > >>>>>>>>>>> the rest of matrices are zeros. > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Dec 28, 2011 at 11:44 PM, Dmitriy Lyubimov < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>>>>>> Yeah, fails for me on ubuntu without any special environment > >>> issues. > >>>>>>>>>>>> Which makes it easier, i can step thru. > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Dec 28, 2011 at 9:01 PM, Ted Dunning < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>>>>>>> What do checksums look like? > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, Dec 28, 2011 at 6:33 PM, Grant Ingersoll < > >>>>>>> [email protected]>wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I commented out the deletion of the dir in the tearDown. > Not > >>> sure > >>>>>>> if that > >>>>>>>>>>>>>> looks reasonable or not, but on the surface they look > >>> equivalent. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Here's the contents of the dir on Ubuntu: > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 > >>> A-000000000 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 > >>> A-000000200 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 > >>> A-000000400 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 > >>> A-000000600 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1387722 2011-12-28 21:17 > >>> A-000000800 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 > >>> B-000000000 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 > >>> B-000000210 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 > >>> B-000000420 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 > >>> B-000000630 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 144312 2011-12-28 21:17 > >>> B-000000840 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-0 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-200 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-400 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-600 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 136352 2011-12-28 21:17 U-800 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-0 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-1 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-2 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-3 > >>>>>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 144372 2011-12-28 21:17 V-4 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Here's what my Mac looks like: > >>>>>>>>>>>>>> total 20296 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000000 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000200 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000400 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000600 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.3M Dec 28 21:28 A-000000800 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000000 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000210 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000420 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000630 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 141K Dec 28 21:28 B-000000840 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-0 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-200 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-400 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-600 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 133K Dec 28 21:28 U-800 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-0 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-1 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-2 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-3 > >>>>>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 141K Dec 28 21:28 V-4 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Dec 28, 2011, at 7:15 PM, Ted Dunning wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Yeah.. but this is a difference from the correct answer. I > >>> am > >>>>>>> moderately > >>>>>>>>>>>>>>> sure that this is a problem writing to the temp directory. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, Dec 28, 2011 at 3:45 PM, Grant Ingersoll < > >>>>>>> [email protected] > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> It's expecting the answer to be 0, but it's some really > >>> large > >>>>>>> value. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>> > >>> > testSingularValues(org.apache.mahout.math.ssvd.SequentialOutOfCoreSvdTest): > >>>>>>>>>>>>>>>> expected:<0.0> but was:<4131200.0000000037> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Dec 28, 2011, at 6:30 PM, Ted Dunning wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I think that the answer is 0 because the model is not > being > >>>>>>> read and we > >>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>> swallowing an exception somewhere. This is what an > >>>>>>> uninitialized > >>>>>>>>>>>>>> matrix > >>>>>>>>>>>>>>>>> would give as a result. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Wed, Dec 28, 2011 at 3:21 PM, Grant Ingersoll < > >>>>>>> [email protected] > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I can reproduce outside of Jenkins. It really seems odd > >>> that > >>>>>>> the > >>>>>>>>>>>>>> answer > >>>>>>>>>>>>>>>>>> is off by so much. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Dec 28, 2011, at 2:15 AM, Dmitriy Lyubimov wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I vaguely remember Jenkins had problems with creating > >>> stuff > >>>>>>> in Java > >>>>>>>>>>>>>> tmp > >>>>>>>>>>>>>>>>>>> dir. E.g. I remember that was creating problems for Mr > >>> tasks > >>>>>>> in local > >>>>>>>>>>>>>>>> mr > >>>>>>>>>>>>>>>>>>> mode legitimately using boxed task temporary space. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> OK I'll try to scan for the problem tomorrow. > >>>>>>>>>>>>>>>>>>> On Dec 27, 2011 10:50 PM, "Ted Dunning" < > >>>>>>> [email protected]> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> So I am like everybody else. The test works for me. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> My suspicion is that there is something going on with > >>> the > >>>>>>> temporary > >>>>>>>>>>>>>>>>>>>> directory that I am trying to use and that the > >>> environment > >>>>>>> that > >>>>>>>>>>>>>>>> Jenkins > >>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>> using is somehow strange. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The only slightly surprising idiom I am using is to > >>> create a > >>>>>>>>>>>>>> temporary > >>>>>>>>>>>>>>>>>>>> file, delete it and recreate it as a directory. I > even > >>>>>>> check the > >>>>>>>>>>>>>>>> return > >>>>>>>>>>>>>>>>>>>> values from the delete and the mkdir. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I will keep looking. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Tue, Dec 27, 2011 at 10:37 PM, Ted Dunning < > >>>>>>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Indeed it does. Thanks for pointing that out. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> This error is very strange. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Tue, Dec 27, 2011 at 10:06 PM, Dmitriy Lyubimov < > >>>>>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Ted, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> do you have an idea why this test may be failing? I > >>> think > >>>>>>> this > >>>>>>>>>>>>>> test > >>>>>>>>>>>>>>>>>>>> comes > >>>>>>>>>>>>>>>>>>>>>> with M-792 commit. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I can take a look at it, I suspect something in the > >>>>>>> environment > >>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>> tripping it. > >>>>>>>>>>>>>>>>>>>>>> On Dec 27, 2011 8:54 PM, "Sean Owen" < > >>> [email protected]> > >>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> It's all errors in the Apache infrastructure, > rather > >>>>>>> than a real > >>>>>>>>>>>>>>>> test > >>>>>>>>>>>>>>>>>>>>>>> failure. At least, stuff passes for me locally, and > >>>>>>> that's what's > >>>>>>>>>>>>>>>>>>>>>>> important. > >>>>>>>>>>>>>>>>>>>>>>> So I'm ignoring these. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Dec 27, 2011 at 9:34 PM, Jeff Eastman > >>>>>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> I'm getting a lot of these emails yet all the > tests > >>> run > >>>>>>> locally > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> me. > >>>>>>>>>>>>>>>>>>>>>>> Does > >>>>>>>>>>>>>>>>>>>>>>>> anybody have an idea what the problem is? This > >>> close to > >>>>>>> a > >>>>>>>>>>>>>> release > >>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>> be really nice to have Jenkins on our side. > >>>>>>>>>>>>>>>>>>>>>>>> Jeff > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> -------------------------------------------- > >>>>>>>>>>>>>>>>>> Grant Ingersoll > >>>>>>>>>>>>>>>>>> http://www.lucidimagination.com > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -------------------------------------------- > >>>>>>>>>>>>>>>> Grant Ingersoll > >>>>>>>>>>>>>>>> http://www.lucidimagination.com > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -------------------------------------------- > >>>>>>>>>>>>>> Grant Ingersoll > >>>>>>>>>>>>>> http://www.lucidimagination.com > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>> > >>> > >>> -------------------------------------------- > >>> Grant Ingersoll > >>> http://www.lucidimagination.com > >>> > >>> > >>> > >>> > >> > > -------------------------- > Grant Ingersoll > http://www.lucidimagination.com > > > > > >
