I have replicated this and applied this patch but the test still fails. I will take a longer look when I get a chance.
On Sat, Dec 31, 2011 at 3:57 PM, Ted Dunning <[email protected]> wrote: > I just took a look and found several files not closed issues as Dmitriy > mentioned. There might be a buffering difference on Linux versus make. > > I have a patch to fix this and will test right now on EC2 if I have enough > time. > > > On Sat, Dec 31, 2011 at 2:25 PM, Grant Ingersoll <[email protected]>wrote: > >> Any progress on this? Seems like it is a show stopper for code freeze. >> On Dec 29, 2011, at 3:58 AM, Dmitriy Lyubimov wrote: >> >> > note identical md5 for u200 and u400. >> > >> > On Thu, Dec 29, 2011 at 12:57 AM, Dmitriy Lyubimov <[email protected]> >> wrote: >> >> MD5 sums >> >> >> >> b8217318a29ef69c58b921013eb019e5 >> /tmp/matrix8554072597307396201/A-000000000 >> >> 41db088ff74c5efd5b766dba253efc03 >> /tmp/matrix8554072597307396201/A-000000200 >> >> b8217318a29ef69c58b921013eb019e5 >> /tmp/matrix8554072597307396201/A-000000400 >> >> 41db088ff74c5efd5b766dba253efc03 >> /tmp/matrix8554072597307396201/A-000000600 >> >> c8dc2a7df82065b5c1e8284ff23aecc6 >> /tmp/matrix8554072597307396201/A-000000800 >> >> 83bccdd2fa191e01d34646e2030f0e77 >> /tmp/matrix8554072597307396201/B-000000000 >> >> 9d6878fb789d61d5453b994ea1a5c6db >> /tmp/matrix8554072597307396201/B-000000210 >> >> cbdf720b17ce25feb686effd1aa0ebef >> /tmp/matrix8554072597307396201/B-000000420 >> >> 2f71d6ba6891b242575b5cc6ba1c4358 >> /tmp/matrix8554072597307396201/B-000000630 >> >> f50b76bb48c8f6a791a6d8206d980492 >> /tmp/matrix8554072597307396201/B-000000840 >> >> 6bb29ca304889a6c8effff6e5c062dc8 /tmp/matrix8554072597307396201/U-0 >> >> 019c881c1d7c5748a1cacb3e0b3e5899 /tmp/matrix8554072597307396201/U-200 >> >> 019c881c1d7c5748a1cacb3e0b3e5899 /tmp/matrix8554072597307396201/U-400 >> >> b84e4b01ffb9d691c87b496f8b4d84ec /tmp/matrix8554072597307396201/U-600 >> >> 6bb29ca304889a6c8effff6e5c062dc8 /tmp/matrix8554072597307396201/U-770 >> >> >> >> >> >> On Thu, Dec 29, 2011 at 12:52 AM, Ted Dunning <[email protected]> >> wrote: >> >>> Thanks. Good hints. >> >>> >> >>> I will take a look on a linux machine in the next few days. >> >>> >> >>> On Thu, Dec 29, 2011 at 12:42 AM, Dmitriy Lyubimov <[email protected] >> >wrote: >> >>> >> >>>> yes. i would venture to say that U computation (or restoration) is >> >>>> somehow corrupted starting with 2nd block. at least it looks this >> way. >> >>>> >> >>>> On Thu, Dec 29, 2011 at 12:31 AM, Dmitriy Lyubimov < >> [email protected]> >> >>>> wrote: >> >>>>> A-reconstructed difference looks good up to row 399 but starting at >> >>>>> row 400 differences do not add up to 0 anymore (although both inputs >> >>>>> are not 0). >> >>>>> >> >>>>> So it doesn't look like trivial case of something is not initialized >> >>>>> on top of it. It does seem something to do with blocking mechanism >> >>>>> though since apparently 400th row is a boundary of some blocking >> >>>>> somewhere, but it is hard for me to see where it fails at this >> point. >> >>>>> >> >>>>> On Thu, Dec 29, 2011 at 12:13 AM, Dmitriy Lyubimov < >> [email protected]> >> >>>> wrote: >> >>>>>> oh. it's because the synthetic input has only 4 singular values. >> >>>>>> >> >>>>>> On Wed, Dec 28, 2011 at 11:56 PM, Dmitriy Lyubimov < >> [email protected]> >> >>>> wrote: >> >>>>>>> But it is not a problem reading U or V files, that's indeed what >> U and >> >>>>>>> V contain. >> >>>>>>> >> >>>>>>> On Wed, Dec 28, 2011 at 11:49 PM, Dmitriy Lyubimov < >> [email protected]> >> >>>> wrote: >> >>>>>>>> U and V look suspect, degenerate (only 4 first columns are >> nonzero, >> >>>>>>>> the rest of matrices are zeros. >> >>>>>>>> >> >>>>>>>> On Wed, Dec 28, 2011 at 11:44 PM, Dmitriy Lyubimov < >> [email protected]> >> >>>> wrote: >> >>>>>>>>> Yeah, fails for me on ubuntu without any special environment >> issues. >> >>>>>>>>> Which makes it easier, i can step thru. >> >>>>>>>>> >> >>>>>>>>> On Wed, Dec 28, 2011 at 9:01 PM, Ted Dunning < >> [email protected]> >> >>>> wrote: >> >>>>>>>>>> What do checksums look like? >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Dec 28, 2011 at 6:33 PM, Grant Ingersoll < >> >>>> [email protected]>wrote: >> >>>>>>>>>> >> >>>>>>>>>>> I commented out the deletion of the dir in the tearDown. Not >> sure >> >>>> if that >> >>>>>>>>>>> looks reasonable or not, but on the surface they look >> equivalent. >> >>>>>>>>>>> >> >>>>>>>>>>> Here's the contents of the dir on Ubuntu: >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 >> A-000000000 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 >> A-000000200 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 >> A-000000400 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1632612 2011-12-28 21:17 >> A-000000600 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 1387722 2011-12-28 21:17 >> A-000000800 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 >> B-000000000 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 >> B-000000210 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 >> B-000000420 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168312 2011-12-28 21:17 >> B-000000630 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 144312 2011-12-28 21:17 >> B-000000840 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-0 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-200 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-400 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 160412 2011-12-28 21:17 U-600 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 136352 2011-12-28 21:17 U-800 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-0 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-1 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-2 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 168432 2011-12-28 21:17 V-3 >> >>>>>>>>>>> -rw-rw-r-- 1 XXXXXX XXXXXX 144372 2011-12-28 21:17 V-4 >> >>>>>>>>>>> >> >>>>>>>>>>> Here's what my Mac looks like: >> >>>>>>>>>>> total 20296 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000000 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000200 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000400 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.6M Dec 28 21:28 A-000000600 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 1.3M Dec 28 21:28 A-000000800 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000000 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000210 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000420 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 B-000000630 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 141K Dec 28 21:28 B-000000840 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-0 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-200 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-400 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 157K Dec 28 21:28 U-600 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 133K Dec 28 21:28 U-800 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-0 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-1 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-2 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 164K Dec 28 21:28 V-3 >> >>>>>>>>>>> -rw-r--r-- 1 XXXXXX staff 141K Dec 28 21:28 V-4 >> >>>>>>>>>>> >> >>>>>>>>>>> On Dec 28, 2011, at 7:15 PM, Ted Dunning wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Yeah.. but this is a difference from the correct answer. I >> am >> >>>> moderately >> >>>>>>>>>>>> sure that this is a problem writing to the temp directory. >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Wed, Dec 28, 2011 at 3:45 PM, Grant Ingersoll < >> >>>> [email protected] >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> It's expecting the answer to be 0, but it's some really >> large >> >>>> value. >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>> >> testSingularValues(org.apache.mahout.math.ssvd.SequentialOutOfCoreSvdTest): >> >>>>>>>>>>>>> expected:<0.0> but was:<4131200.0000000037> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Dec 28, 2011, at 6:30 PM, Ted Dunning wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> I think that the answer is 0 because the model is not being >> >>>> read and we >> >>>>>>>>>>>>> are >> >>>>>>>>>>>>>> swallowing an exception somewhere. This is what an >> >>>> uninitialized >> >>>>>>>>>>> matrix >> >>>>>>>>>>>>>> would give as a result. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Wed, Dec 28, 2011 at 3:21 PM, Grant Ingersoll < >> >>>> [email protected] >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I can reproduce outside of Jenkins. It really seems odd >> that >> >>>> the >> >>>>>>>>>>> answer >> >>>>>>>>>>>>>>> is off by so much. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Dec 28, 2011, at 2:15 AM, Dmitriy Lyubimov wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> I vaguely remember Jenkins had problems with creating >> stuff >> >>>> in Java >> >>>>>>>>>>> tmp >> >>>>>>>>>>>>>>>> dir. E.g. I remember that was creating problems for Mr >> tasks >> >>>> in local >> >>>>>>>>>>>>> mr >> >>>>>>>>>>>>>>>> mode legitimately using boxed task temporary space. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> OK I'll try to scan for the problem tomorrow. >> >>>>>>>>>>>>>>>> On Dec 27, 2011 10:50 PM, "Ted Dunning" < >> >>>> [email protected]> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> So I am like everybody else. The test works for me. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> My suspicion is that there is something going on with >> the >> >>>> temporary >> >>>>>>>>>>>>>>>>> directory that I am trying to use and that the >> environment >> >>>> that >> >>>>>>>>>>>>> Jenkins >> >>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>> using is somehow strange. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> The only slightly surprising idiom I am using is to >> create a >> >>>>>>>>>>> temporary >> >>>>>>>>>>>>>>>>> file, delete it and recreate it as a directory. I even >> >>>> check the >> >>>>>>>>>>>>> return >> >>>>>>>>>>>>>>>>> values from the delete and the mkdir. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> I will keep looking. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On Tue, Dec 27, 2011 at 10:37 PM, Ted Dunning < >> >>>>>>>>>>> [email protected]> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> Indeed it does. Thanks for pointing that out. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> This error is very strange. >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> On Tue, Dec 27, 2011 at 10:06 PM, Dmitriy Lyubimov < >> >>>>>>>>>>>>> [email protected] >> >>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Ted, >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> do you have an idea why this test may be failing? I >> think >> >>>> this >> >>>>>>>>>>> test >> >>>>>>>>>>>>>>>>> comes >> >>>>>>>>>>>>>>>>>>> with M-792 commit. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I can take a look at it, I suspect something in the >> >>>> environment >> >>>>>>>>>>> can >> >>>>>>>>>>>>> be >> >>>>>>>>>>>>>>>>>>> tripping it. >> >>>>>>>>>>>>>>>>>>> On Dec 27, 2011 8:54 PM, "Sean Owen" < >> [email protected]> >> >>>> wrote: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> It's all errors in the Apache infrastructure, rather >> >>>> than a real >> >>>>>>>>>>>>> test >> >>>>>>>>>>>>>>>>>>>> failure. At least, stuff passes for me locally, and >> >>>> that's what's >> >>>>>>>>>>>>>>>>>>>> important. >> >>>>>>>>>>>>>>>>>>>> So I'm ignoring these. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> On Tue, Dec 27, 2011 at 9:34 PM, Jeff Eastman >> >>>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >> >>>>>>>>>>>>>>>>>>>>> I'm getting a lot of these emails yet all the tests >> run >> >>>> locally >> >>>>>>>>>>>>> for >> >>>>>>>>>>>>>>>>>>> me. >> >>>>>>>>>>>>>>>>>>>> Does >> >>>>>>>>>>>>>>>>>>>>> anybody have an idea what the problem is? This >> close to >> >>>> a >> >>>>>>>>>>> release >> >>>>>>>>>>>>> it >> >>>>>>>>>>>>>>>>>>>> would >> >>>>>>>>>>>>>>>>>>>>> be really nice to have Jenkins on our side. >> >>>>>>>>>>>>>>>>>>>>> Jeff >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> -------------------------------------------- >> >>>>>>>>>>>>>>> Grant Ingersoll >> >>>>>>>>>>>>>>> http://www.lucidimagination.com >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -------------------------------------------- >> >>>>>>>>>>>>> Grant Ingersoll >> >>>>>>>>>>>>> http://www.lucidimagination.com >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> -------------------------------------------- >> >>>>>>>>>>> Grant Ingersoll >> >>>>>>>>>>> http://www.lucidimagination.com >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>> >> >> -------------------------------------------- >> Grant Ingersoll >> http://www.lucidimagination.com >> >> >> >> >
