Hi Sean, On May 8, 2011, at 4:14pm, Sean Owen wrote:
> (The build error indicates you have some old class files somewhere -- > "clean" first) Thanks, my bad - our internal ant-based builds always do an implicit clean before an install. > Here, the lib/ directory definitely has the right dependencies and it > still doesn't work. Benson investigated and found out it's just how > Hadoop works in this case. The last thing I saw from Benson was: > Well, of course. We made ours work by carefully tuning our use of the > assembly plugin to make sure that all the mahout jobs ended up in the > 'unpacked' part of the circus, and Math and other dependencies ended > up jarred up in lib. This is the key point - that anything which is run as the main class for the job has to be on the regular classpath, not buried in an embedded jar. Mahout is interesting in that it's a mix of support code, map/reduce tasks, and jobs. If I had to solve this problem, I'd look at establishing a naming convention where all such job classes end with "Job". Then I could (easily, in ant - I'm sure Maven would require some black belt moves) have a build where these classes don't go into the regular mahout-xxx.jar, but rather a mahout-jobs.jar Now, if I am incorporating Mahout into one of my projects, for my custom job jar: * Add all of the regular mahout-xxx.jar files that I need as sub-jars in my job jar. * Add the mahout-jobs.jar as a regular dependency. -- Ken > On Mon, May 9, 2011 at 12:06 AM, Ken Krugler > <[email protected]> wrote: >> I haven't been actively running Mahout for a while, but I do watch plenty of >> Hadoop students run into the ClassNotFoundException problem. >> >> A standard Hadoop job jar has a lib subdir, which contains (as jars) all of >> the dependencies. >> >> Typically the missing class problem is caused by somebody building their own >> Hadoop job jar, where they don't include a dependent jar (such as >> mahout-math) in the lib subdir. >> >> Or somebody is trying to run a job locally, using the job jar directly, >> which then has to be unpacked as otherwise these embedded lib/*.jar classes >> aren't on the classpath. >> >> But neither of those seem to match what Jake was doing: >> >>> (just running things like "./bin/mahout svd -i <input> -o <output> etc... ") >> >> >> I was going to try this out from trunk, but an svn up on trunk and then "mvn >> install" failed to pass one of the tests: >> >>> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.025 sec >>> <<< FAILURE! >>> fullRankTall(org.apache.mahout.math.QRDecompositionTest) Time elapsed: >>> 0.014 sec <<< ERROR! >>> java.lang.NoSuchFieldError: MAX >>> at >>> org.apache.mahout.math.QRDecompositionTest.assertEquals(QRDecompositionTest.java:122) >>> at >>> org.apache.mahout.math.QRDecompositionTest.fullRankTall(QRDecompositionTest.java:38) >> >> >> -- Ken -------------------------- Ken Krugler +1 530-210-6378 http://bixolabs.com e l a s t i c w e b m i n i n g
