So, I'm running: mvn -o install -DskipTests=true at project root (in mahout)
Comment out or remove the maven-assembly-plugin definition in core/pom.xml -- it reduced my core build time from 26s to 6s -- I can submit a patch for this. Mahout math is still 17s here due to code generation. I'm wondering if there's a way to modify the generation plugin to that it doesn't re-generate if there are no changes to the templates. You can remove the plugin definition from math/pom.xml and it doesn't seem to break anything unless you're doing a clean. Brings down math compilation to 3s without it. Total compile time is 22s. re: the job, I'll have to look into that further later. On Fri, Feb 5, 2010 at 12:06 PM, Robin Anil <robin.a...@gmail.com> wrote: > Yes for editing i use eclipse in the same fashion. If i want to try out a > job and see how it performs on hadoop I need job compiled fast. > > On another note. I think there will be a lot of dead code in the job(with > all the jar files bundles) Is there an optimiser for that i.e to remove > classes which mahout never use indirectly > > I see loading jar takes 10-20 seconds when initializing mapper or reducer. > It doesnt affect long running jobs but 20 sec overhead for processing 64MB > chunk sucks > > On Fri, Feb 5, 2010 at 10:19 PM, Ted Dunning <ted.dunn...@gmail.com> wrote: > >> I usually do an initial compilation using mvn package. Then, during >> development I use IntelliJ's incremental compilation which generally only >> takes a few seconds. Since that compilation doesn't handle things like >> copying resources, I get caught out and surprised now and again, but this >> works almost all the time. >> >> On Fri, Feb 5, 2010 at 12:27 AM, Robin Anil <robin.a...@gmail.com> wrote: >> >> > When developing mahout core/util/examples we dont need to generate math >> > often and dont need to tar gzip bzip2 the jar files. We are mostly >> > concerned >> > with the job file/ jar file. >> > Cant there be another target like develop which does this. (waiting 2-3 >> > mins >> > for a 2 line change is frustrating) >> > >> > Robin >> > >> >> >> >> -- >> Ted Dunning, CTO >> DeepDyve >> >
