This is perhaps tangential, but pig 0.10+ does this automatically with option pig.exec.mapPartAgg = true:
http://pig.apache.org/docs/r0.10.0/perf.html, section "Hash-based Aggregation in Map Task" https://issues.apache.org/jira/browse/PIG-2228 https://cwiki.apache.org/PIG/pig-performance-optimization.html http://wiki.apache.org/pig/PigHashBasedAggInMap On Wed, Jun 12, 2013 at 8:59 AM, Jake Mannix <[email protected]> wrote: > In fact, I think we're doing exactly this "design pattern" in a few places > already. In particular, the CachingCV0Driver is effectively an in-memory > mapside cache of topic/term counts, and it only flushes them all out in the > cleanup phase of the mapper execution. > > I'd certainly like to see what sort of API this would look like, a > relatively general form of this could be quite useful, especially if the > LRU cache can be tuned and controlled (sometimes you might want to control > it's flushing, as there may be business/algorithm logic which needs to be > executed at flush time). > > > On Wed, Jun 12, 2013 at 8:45 AM, Sebastian Schelter <[email protected]> > wrote: > > > Regarding the in-memory combiner: It would be good if you showcase the > > benefits on one specific implementation in Mahout, by replacing its > > normal combiner with the in-memory one and benchmarking it. > > > > I'm curious to see the results. > > > > Best, > > Sebastian > > > > > > On 12.06.2013 17:06, Grant Ingersoll wrote: > > > Hi DB, > > > > > > This all sounds rather interesting. I see a number of places where we > > use combiners, so perhaps focus on those first? > > > > > > Also, any thoughts on when the scalable SVM would be ready? We are > > trying to get 1.0 out in the next few months and I personally think it > > would be good to have SVM in. > > > > > > -Grant > > > > > > On Jun 11, 2013, at 8:20 PM, DB Tsai <[email protected]> wrote: > > > > > >> Hi, > > >> > > >> Recently we started to use the in-mapper combiner design patterns in > > >> our hadoop based algorithms at Alpine Data Labs; those algorithms > > >> include variable selection using info gain, decision tree, naive bayes > > >> model and SVM, and we found that we can have 20~40% performance > > >> speedup without doing too much work. > > >> > > >> The whole idea is really simple, just use a in-mapper LRU cache to > > >> combine the result first instead of using combiner directly. If the > > >> cache is full, just emit the result to combiner or reducer. The detail > > >> is discussed in Data-Intensive Text Processing with MapReduce > > >> ( > http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf) > > >> by Jimmy Lin and Chris Dyer at University of Maryland, College Park. > > >> > > >> We would like to contribute the api to mahout, and work closer with > > >> open source community. I'm now working on random forest using > > >> information gain, and we have the plan to contribute to mahout > > >> community. We also have a scalable kernel SVM implementation which > > >> intends to contribute to mahout as well. We just presented a talk > > >> about our SVM in SF machine learning meetup with great feedback, see > > >> > > >> > > > http://www.meetup.com/sfmachinelearning/events/116497192/?_af_eid=116497192&a=uc1_te&_af=event > > >> > > >> The api is pretty simple, just change context.write to combiner.write, > > >> and remember to flush the cache in the clean up method. > > >> > > >> This is the example of implementing hadoop classical word count using > > >> in-mapper combiner, > > >> > > > https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerExampleTest.java > > >> > > >> , and all we need to do is just change from context.write to > > >> combiner.write. The test code for this example is in > > >> > > > https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java > > >> > > >> This is the actually implementation of in-mapper combiner using LRU > > cache, > > >> > > > https://github.com/dbtsai/mahout/blob/trunk/core/src/main/java/org/apache/mahout/common/mapreduce/InMapperCombiner.java > > >> > > >> and this implementation is well tested. > > >> > > > https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java > > >> > > >> I'm wondering what is the best candidate in mahout to use this kind of > > >> in-mapper combiner now to demonstrate this idea works, and I'll focus > > >> on that particular use case, and do benchmark. > > >> > > >> Thanks. > > >> > > >> Sincerely, > > >> > > >> DB Tsai > > >> ----------------------------------- > > >> Web: http://www.dbtsai.com > > >> Phone : +1-650-383-8392 > > > > > > -------------------------------------------- > > > Grant Ingersoll | @gsingers > > > http://www.lucidworks.com > > > > > > > > > > > > > > > > > > > > > > > > > -- > > -jake >
