Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83045631 I updated with the @mengxr checkin that cleaned jblas dependencies from NNLS....the results updated above is with rank = 10 I think...A more realistic rank is 50... breeze nnls (including iterator overhead): 15/03/18 09:21:45 INFO ALS: solveTime 177.048 ms 15/03/18 09:21:45 INFO ALS: solveTime 189.642 ms 15/03/18 09:21:45 INFO ALS: solveTime 171.894 ms 15/03/18 09:21:45 INFO ALS: solveTime 203.503 ms 15/03/18 09:21:45 INFO ALS: solveTime 185.352 ms 15/03/18 09:21:45 INFO ALS: solveTime 190.051 ms 15/03/18 09:21:45 INFO ALS: solveTime 181.634 ms 15/03/18 09:21:45 INFO ALS: solveTime 148.325 ms 15/03/18 09:21:46 INFO ALS: solveTime 59.523 ms 15/03/18 09:21:46 INFO ALS: solveTime 78.501 ms 15/03/18 09:21:46 INFO ALS: solveTime 66.649 ms 15/03/18 09:21:46 INFO ALS: solveTime 66.078 ms 15/03/18 09:21:46 INFO ALS: solveTime 75.827 ms 15/03/18 09:21:46 INFO ALS: solveTime 50.72 ms 15/03/18 09:21:46 INFO ALS: solveTime 65.047 ms 15/03/18 09:21:46 INFO ALS: solveTime 59.184 ms 15/03/18 09:21:46 INFO ALS: solveTime 26.837 ms 15/03/18 09:21:46 INFO ALS: solveTime 26.061 ms 15/03/18 09:21:46 INFO ALS: solveTime 36.14 ms 15/03/18 09:21:46 INFO ALS: solveTime 33.341 ms 15/03/18 09:21:46 INFO ALS: solveTime 29.399 ms 15/03/18 09:21:46 INFO ALS: solveTime 28.61 ms 15/03/18 09:21:46 INFO ALS: solveTime 30.074 ms 15/03/18 09:21:46 INFO ALS: solveTime 32.351 ms 15/03/18 09:21:46 INFO ALS: solveTime 56.625 ms 15/03/18 09:21:47 INFO ALS: solveTime 54.29 ms 15/03/18 09:21:47 INFO ALS: solveTime 49.305 ms 15/03/18 09:21:47 INFO ALS: solveTime 63.742 ms 15/03/18 09:21:47 INFO ALS: solveTime 67.626 ms 15/03/18 09:21:47 INFO ALS: solveTime 55.348 ms 15/03/18 09:21:47 INFO ALS: solveTime 54.473 ms 15/03/18 09:21:47 INFO ALS: solveTime 66.704 ms 15/03/18 09:21:47 INFO ALS: solveTime 29.57 ms 15/03/18 09:21:47 INFO ALS: solveTime 33.871 ms 15/03/18 09:21:47 INFO ALS: solveTime 33.329 ms 15/03/18 09:21:47 INFO ALS: solveTime 40.322 ms 15/03/18 09:21:47 INFO ALS: solveTime 35.376 ms 15/03/18 09:21:47 INFO ALS: solveTime 35.826 ms 15/03/18 09:21:47 INFO ALS: solveTime 38.973 ms 15/03/18 09:21:47 INFO ALS: solveTime 34.908 ms breeze nnls inner solve (subtracting iterator time): 15/03/18 09:21:45 INFO ALS: innerTime 103.235 ms 15/03/18 09:21:45 INFO ALS: innerTime 115.842 ms 15/03/18 09:21:45 INFO ALS: innerTime 104.233 ms 15/03/18 09:21:45 INFO ALS: innerTime 112.84 ms 15/03/18 09:21:45 INFO ALS: innerTime 97.474 ms 15/03/18 09:21:45 INFO ALS: innerTime 91.188 ms 15/03/18 09:21:45 INFO ALS: innerTime 115.043 ms 15/03/18 09:21:45 INFO ALS: innerTime 96.443 ms 15/03/18 09:21:46 INFO ALS: innerTime 31.696 ms 15/03/18 09:21:46 INFO ALS: innerTime 40.446 ms 15/03/18 09:21:46 INFO ALS: innerTime 42.288 ms 15/03/18 09:21:46 INFO ALS: innerTime 34.147 ms 15/03/18 09:21:46 INFO ALS: innerTime 33.404 ms 15/03/18 09:21:46 INFO ALS: innerTime 28.02 ms 15/03/18 09:21:46 INFO ALS: innerTime 33.372 ms 15/03/18 09:21:46 INFO ALS: innerTime 38.531 ms 15/03/18 09:21:46 INFO ALS: innerTime 20.651 ms 15/03/18 09:21:46 INFO ALS: innerTime 19.721 ms 15/03/18 09:21:46 INFO ALS: innerTime 27.865 ms 15/03/18 09:21:46 INFO ALS: innerTime 21.655 ms 15/03/18 09:21:46 INFO ALS: innerTime 22.413 ms 15/03/18 09:21:46 INFO ALS: innerTime 19.793 ms 15/03/18 09:21:46 INFO ALS: innerTime 18.858 ms 15/03/18 09:21:46 INFO ALS: innerTime 24.089 ms 15/03/18 09:21:46 INFO ALS: innerTime 43.602 ms 15/03/18 09:21:47 INFO ALS: innerTime 44.16 ms 15/03/18 09:21:47 INFO ALS: innerTime 32.279 ms 15/03/18 09:21:47 INFO ALS: innerTime 53.446 ms 15/03/18 09:21:47 INFO ALS: innerTime 48.279 ms 15/03/18 09:21:47 INFO ALS: innerTime 44.157 ms 15/03/18 09:21:47 INFO ALS: innerTime 37.386 ms 15/03/18 09:21:47 INFO ALS: innerTime 56.11 ms 15/03/18 09:21:47 INFO ALS: innerTime 21.043 ms 15/03/18 09:21:47 INFO ALS: innerTime 25.038 ms 15/03/18 09:21:47 INFO ALS: innerTime 24.97 ms 15/03/18 09:21:47 INFO ALS: innerTime 27.335 ms 15/03/18 09:21:47 INFO ALS: innerTime 26.706 ms 15/03/18 09:21:47 INFO ALS: innerTime 27.418 ms 15/03/18 09:21:47 INFO ALS: innerTime 30.885 ms 15/03/18 09:21:47 INFO ALS: innerTime 19.971 ms mllib nnls: 15/03/18 09:23:14 INFO ALS: solveTime 89.822 ms 15/03/18 09:23:14 INFO ALS: solveTime 107.005 ms 15/03/18 09:23:14 INFO ALS: solveTime 80.083 ms 15/03/18 09:23:14 INFO ALS: solveTime 106.405 ms 15/03/18 09:23:14 INFO ALS: solveTime 94.193 ms 15/03/18 09:23:14 INFO ALS: solveTime 102.188 ms 15/03/18 09:23:14 INFO ALS: solveTime 100.847 ms 15/03/18 09:23:14 INFO ALS: solveTime 75.717 ms 15/03/18 09:23:14 INFO ALS: solveTime 45.384 ms 15/03/18 09:23:14 INFO ALS: solveTime 45.79 ms 15/03/18 09:23:14 INFO ALS: solveTime 67.02 ms 15/03/18 09:23:14 INFO ALS: solveTime 51.034 ms 15/03/18 09:23:14 INFO ALS: solveTime 45.844 ms 15/03/18 09:23:15 INFO ALS: solveTime 46.944 ms 15/03/18 09:23:15 INFO ALS: solveTime 45.024 ms 15/03/18 09:23:15 INFO ALS: solveTime 55.672 ms 15/03/18 09:23:15 INFO ALS: solveTime 26.168 ms 15/03/18 09:23:15 INFO ALS: solveTime 27.104 ms 15/03/18 09:23:15 INFO ALS: solveTime 33.295 ms 15/03/18 09:23:15 INFO ALS: solveTime 28.376 ms 15/03/18 09:23:15 INFO ALS: solveTime 27.73 ms 15/03/18 09:23:15 INFO ALS: solveTime 27.867 ms 15/03/18 09:23:15 INFO ALS: solveTime 27.287 ms 15/03/18 09:23:15 INFO ALS: solveTime 24.709 ms 15/03/18 09:23:15 INFO ALS: solveTime 45.617 ms 15/03/18 09:23:15 INFO ALS: solveTime 47.411 ms 15/03/18 09:23:15 INFO ALS: solveTime 60.453 ms 15/03/18 09:23:15 INFO ALS: solveTime 50.855 ms 15/03/18 09:23:15 INFO ALS: solveTime 44.043 ms 15/03/18 09:23:15 INFO ALS: solveTime 51.125 ms 15/03/18 09:23:15 INFO ALS: solveTime 58.812 ms 15/03/18 09:23:15 INFO ALS: solveTime 49.631 ms 15/03/18 09:23:16 INFO ALS: solveTime 35.489 ms 15/03/18 09:23:16 INFO ALS: solveTime 31.978 ms 15/03/18 09:23:16 INFO ALS: solveTime 29.305 ms 15/03/18 09:23:16 INFO ALS: solveTime 28.568 ms 15/03/18 09:23:16 INFO ALS: solveTime 40.257 ms 15/03/18 09:23:16 INFO ALS: solveTime 24.955 ms 15/03/18 09:23:16 INFO ALS: solveTime 28.446 ms 15/03/18 09:23:16 INFO ALS: solveTime 35.841 ms So at realistic ranks still there is an overhead from iterator but in overall flow I don't know how much difference does this overhead causes... @mengxr if you have pointers on how to optimize Breeze NNLS code further please let me know...
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org