Regarding the in-memory combiner: It would be good if you showcase the benefits on one specific implementation in Mahout, by replacing its normal combiner with the in-memory one and benchmarking it.
I'm curious to see the results. Best, Sebastian On 12.06.2013 17:06, Grant Ingersoll wrote: > Hi DB, > > This all sounds rather interesting. I see a number of places where we use > combiners, so perhaps focus on those first? > > Also, any thoughts on when the scalable SVM would be ready? We are trying to > get 1.0 out in the next few months and I personally think it would be good to > have SVM in. > > -Grant > > On Jun 11, 2013, at 8:20 PM, DB Tsai <[email protected]> wrote: > >> Hi, >> >> Recently we started to use the in-mapper combiner design patterns in >> our hadoop based algorithms at Alpine Data Labs; those algorithms >> include variable selection using info gain, decision tree, naive bayes >> model and SVM, and we found that we can have 20~40% performance >> speedup without doing too much work. >> >> The whole idea is really simple, just use a in-mapper LRU cache to >> combine the result first instead of using combiner directly. If the >> cache is full, just emit the result to combiner or reducer. The detail >> is discussed in Data-Intensive Text Processing with MapReduce >> (http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf) >> by Jimmy Lin and Chris Dyer at University of Maryland, College Park. >> >> We would like to contribute the api to mahout, and work closer with >> open source community. I'm now working on random forest using >> information gain, and we have the plan to contribute to mahout >> community. We also have a scalable kernel SVM implementation which >> intends to contribute to mahout as well. We just presented a talk >> about our SVM in SF machine learning meetup with great feedback, see >> >> http://www.meetup.com/sfmachinelearning/events/116497192/?_af_eid=116497192&a=uc1_te&_af=event >> >> The api is pretty simple, just change context.write to combiner.write, >> and remember to flush the cache in the clean up method. >> >> This is the example of implementing hadoop classical word count using >> in-mapper combiner, >> https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerExampleTest.java >> >> , and all we need to do is just change from context.write to >> combiner.write. The test code for this example is in >> https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java >> >> This is the actually implementation of in-mapper combiner using LRU cache, >> https://github.com/dbtsai/mahout/blob/trunk/core/src/main/java/org/apache/mahout/common/mapreduce/InMapperCombiner.java >> >> and this implementation is well tested. >> https://github.com/dbtsai/mahout/blob/trunk/core/src/test/java/org/apache/mahout/common/mapreduce/InMapperCombinerTest.java >> >> I'm wondering what is the best candidate in mahout to use this kind of >> in-mapper combiner now to demonstrate this idea works, and I'll focus >> on that particular use case, and do benchmark. >> >> Thanks. >> >> Sincerely, >> >> DB Tsai >> ----------------------------------- >> Web: http://www.dbtsai.com >> Phone : +1-650-383-8392 > > -------------------------------------------- > Grant Ingersoll | @gsingers > http://www.lucidworks.com > > > > > >
