Hi, Recently we started to use the in-mapper combiner design patterns in our hadoop based algorithms at Alpine Data Labs; those algorithms include variable selection using info gain, decision tree, naive bayes model and SVM, and we found that we can have 20~40% performance speedup without doing too much work.
The whole idea is really simple, just use a in-mapper LRU cache to combine the result first instead of using combiner directly. If the cache is full, just emit the result to combiner or reducer. The detail is discussed in Data-Intensive Text Processing with MapReduce (http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf) by Jimmy Lin and Chris Dyer at University of Maryland, College Park. We would like to contribute the api to mahout, and work closer with open source community. I'm now working on random forest using information gain, and we have the plan to contribute to mahout community. We also have a scalable kernel SVM implementation which intends to contribute to mahout as well. We just presented a talk about our SVM in SF machine learning meetup with great feedback, see http://www.meetup.com/sfmachinelearning/events/116497192/?_af_eid=116497192&a=uc1_te&_af=event The api is pretty simple, just change context.write to combiner.write, and remember to flush the cache in the clean up method. This is the example of implementing hadoop classical word count using in-mapper combiner, https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerExampleTest.java , and all we need to do is just change from context.write to combiner.write. The test code for this example is in https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java This is the actually implementation of in-mapper combiner using LRU cache, https://github.com/dbtsai/mahout/blob/trunk/core/src/main/java/org/apache/mahout/common/mapreduce/InMapperCombiner.java and this implementation is well tested. https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java I'm wondering what is the best candidate in mahout to use this kind of in-mapper combiner now to demonstrate this idea works, and I'll focus on that particular use case, and do benchmark. Thanks. Sincerely, DB Tsai ----------------------------------- Web: http://www.dbtsai.com Phone : +1-650-383-8392
