Hi DB, This all sounds rather interesting. I see a number of places where we use combiners, so perhaps focus on those first?
Also, any thoughts on when the scalable SVM would be ready? We are trying to get 1.0 out in the next few months and I personally think it would be good to have SVM in. -Grant On Jun 11, 2013, at 8:20 PM, DB Tsai <[email protected]> wrote: > Hi, > > Recently we started to use the in-mapper combiner design patterns in > our hadoop based algorithms at Alpine Data Labs; those algorithms > include variable selection using info gain, decision tree, naive bayes > model and SVM, and we found that we can have 20~40% performance > speedup without doing too much work. > > The whole idea is really simple, just use a in-mapper LRU cache to > combine the result first instead of using combiner directly. If the > cache is full, just emit the result to combiner or reducer. The detail > is discussed in Data-Intensive Text Processing with MapReduce > (http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf) > by Jimmy Lin and Chris Dyer at University of Maryland, College Park. > > We would like to contribute the api to mahout, and work closer with > open source community. I'm now working on random forest using > information gain, and we have the plan to contribute to mahout > community. We also have a scalable kernel SVM implementation which > intends to contribute to mahout as well. We just presented a talk > about our SVM in SF machine learning meetup with great feedback, see > > http://www.meetup.com/sfmachinelearning/events/116497192/?_af_eid=116497192&a=uc1_te&_af=event > > The api is pretty simple, just change context.write to combiner.write, > and remember to flush the cache in the clean up method. > > This is the example of implementing hadoop classical word count using > in-mapper combiner, > https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerExampleTest.java > > , and all we need to do is just change from context.write to > combiner.write. The test code for this example is in > https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java > > This is the actually implementation of in-mapper combiner using LRU cache, > https://github.com/dbtsai/mahout/blob/trunk/core/src/main/java/org/apache/mahout/common/mapreduce/InMapperCombiner.java > > and this implementation is well tested. > https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java > > I'm wondering what is the best candidate in mahout to use this kind of > in-mapper combiner now to demonstrate this idea works, and I'll focus > on that particular use case, and do benchmark. > > Thanks. > > Sincerely, > > DB Tsai > ----------------------------------- > Web: http://www.dbtsai.com > Phone : +1-650-383-8392 -------------------------------------------- Grant Ingersoll | @gsingers http://www.lucidworks.com
