0.11 targets 1.3+. I don't quite have anything on top of my head affecting A'B specifically, but i think there were some chanages affecting in-memory multiplication (which is of course used in distributed A'B).
I am not in particular familiar or remember details of row similarity on top of my head, i really wish the original contributor would comment on that. trying to see if i can come up with anything useful though. what behavior do you see in this job -- cpu-bound or i/o bound? there are a few pointers to look at: (1) I/O many times exceeds the input size, so spills are inevitable. So tuning memory sizes and look at spark spill locations to make sure disks are not slow there is critical. Also, i think in spark 1.6 spark added a lot of flexibility in managing task/cache/shuffle memory sizes, it may help in some unexpected way. (2) sufficient cache: many pipelines commit reused matrices into cache (MEMORY_ONLY) which is the default mahout algebra behavior, assuming there is enough cache memory there for only good things to happen. if it is not, however, it will cause recomputation of results that were evicted. (not saying it is a known case for row similarity in particular). make sure this is not the case. For cases of scatter type exchanges it is especially super bad. (3) A'B -- try to hack and play with implemetnation there in AtB (spark side) class. See if you can come up with a better arrangement. (4) in-memory computations (MMul class) if that's the bottleneck can be in practice quick-hacked with mutlithreaded multiplication and bridge to native solvers (netlib-java) at least for dense cases. this is found to improve performance of distributed multiplications a bit. Works best if you get 2 threads in the backend and all threads in the front end. There are other known things that can improve speed multiplication of the public mahout version, i hope mahout will improve on those in the future. -d On Wed, Apr 27, 2016 at 6:14 AM, Nikaash Puri <[email protected]> wrote: > Hi, > > I’ve been working with LLR in Mahout for a while now. Mostly using the > SimilarityAnalysis.cooccurenceIDss function. I recently upgraded the Mahout > libraries to 0.11, and subsequently also tried with 0.12 and the same > program is running orders of magnitude slower (at least 3x based on initial > analysis). > > Looking into the tasks more carefully, comparing 0.10 and 0.11 shows that > the amount of Shuffle being done in 0.11 is significantly higher, > especially in the AtB step. This could possibly be a reason for the > reduction in performance. > > Although, I am working on Spark 1.2.0. So, its possible that this could be > causing the problem. It works fine with Mahout 0.10. > > Any ideas why this might be happening? > > Thank you, > Nikaash Puri
