[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/5005 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-168112609 I'm going to close this pull request. If this is still relevant and you are interested in pushing it forward, please open a new pull request. Thanks! --- If your project

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-147524211 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-147524207 [Test build #43582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43582/consoleFull) for PR 5005 at commit [`2e0603a`](https://git

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-147524210 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-147522544 [Test build #43582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43582/consoleFull) for PR 5005 at commit [`2e0603a`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-147519034 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-147519092 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-96807849 **[Test build #31054 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31054/consoleFull)** for PR 5005 at commit [`2e0603a`](https://git

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-96770888 [Test build #31054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31054/consoleFull) for PR 5005 at commit [`2e0603a`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user tmyklebu commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90963126 OK. I haven't made a serious attempt to write a solver for general L1-constrained least squares problems. I don't see anything wrong with implementing a generalisation

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90950074 if you look into breeze.optimize.proximal.Proximal, I added a library of projection/proximal operators...in my experiments looks like projection based algorithms (SPG

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90950364 Application is topic modeling using Sparsity constraints like L1 and probability simplex and supporting bounds in ALS --- If your project is set up for it, you can r

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user tmyklebu commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90947409 Not at home right now, so I don't have everything in front of me. If you have a "projection onto tangent cone" operator and you keep explicit track of the active set, y

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90942562 @tmyklebu do you have the original NNLS paper in english ? Breeze also has a linear CG...I am thinking if it is possible to merge simple projections like positivity an

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90753271 Sure...Let me do that and point you to the repo...most likely it will be a breeze based branch and I will copy the mllib implementation over thr... --- If your projec

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-90740759 We should do a micro-benchmark instead of comparing the running times in ALS. Could you create a repo, copy the implementation over, and put your benchmark code there. I c

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-89594722 @mengxr any insight on it ? the runtime issue is only in first iteration and I think you can point out if there is any obvious issue in the way I call the solver...loo

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-87291127 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-87291120 [Test build #29352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29352/consoleFull) for PR 5005 at commit [`2e0603a`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-87277070 [Test build #29352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29352/consoleFull) for PR 5005 at commit [`2e0603a`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-87276063 Updated the PR with breeze 0.11.2...Except first iteration, rest of them are at par: Breeze NNLS: TUSCA09LMLVT00C:spark-brznnls v606014$ grep solveTim

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-86949884 @mengxr any updates on it ? breeze 0.11.2 is now integrated with Spark --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-85348266 All the runtime enhancements are being added to Breeze in this PR: https://github.com/scalanlp/breeze/pull/386 Please let me know if there are additional feedbacks.

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84519127 failure testcase is due to changing the als seed to 0L and get repeatable results over multiple runs... --- If your project is set up for it, you can reply to this em

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84513987 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84513971 [Test build #28953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28953/consoleFull) for PR 5005 at commit [`c592d56`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84512558 I am confused why the mllib NNLS does not show it...we are allocating exactly same memory in both Breeze and mllib NNLS. In Breeze we call it State and in mllib NNLS i

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user dlwh commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84507139 It's probably just HotSpot warming up. I wouldn't worry about it. On Sat, Mar 21, 2015 at 7:34 PM, Debasish Das wrote: > Even after cleaning up iterator

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84499918 Even after cleaning up iterator, adding in-place gemv and create the state and re-use the memory, still the first iteration of Breeze NNLS is slower than mllib NNLS...

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84499544 [Test build #28953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28953/consoleFull) for PR 5005 at commit [`c592d56`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84199377 yeah will push it over the weekend...I am almost done with the changes.. --- If your project is set up for it, you can reply to this email and have your reply appear o

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user dlwh commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84184709 could you submit a PR for the changes soon. I want to get the fix out for the critical SparseVector bug ASAP. On Fri, Mar 20, 2015 at 10:51 AM, David Hall wr

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user dlwh commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84087409 sure On Fri, Mar 20, 2015 at 10:47 AM, Debasish Das wrote: > @dlwh @tmyklebu > consensus ?

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-84086432 @dlwh @tmyklebu consensus ? The change will be here: def iterations(q: DenseVector[Double], rho: Double,initialState: State) : State vs default: def it

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83752607 @tmyklebu these least squares problem need not be necessarily small but for mllib ALS they are... Think about TRON (breeze.optimize.TruncatedNewtonMinimizer) a

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user tmyklebu commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83743835 @dlwh: Intermediate states do not matter in ANNLS. In ANNLS, we allow ourselves to do a crappy job solving the least squares problems at each iteration because the resu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user dlwh commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83739506 I mean, the liblbfgs code does callbacks, not iterators. They are duals w.r.t. who has control over what gets executed when. (Push iterates vs pull iterates) In genera

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83720384 @mengxr it's interesting that the iterator overhead is high only at initial iterations and as JVM gets hotter the runtime gets at par with the while loop...The iterato

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83671037 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83671020 [Test build #28876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28876/consoleFull) for PR 5005 at commit [`b472c0c`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83641535 [Test build #28876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28876/consoleFull) for PR 5005 at commit [`b472c0c`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83639402 [Test build #28875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28875/consoleFull) for PR 5005 at commit [`78de96e`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83639407 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83639180 [Test build #28875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28875/consoleFull) for PR 5005 at commit [`78de96e`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83279886 [Test build #28853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28853/consoleFull) for PR 5005 at commit [`6ba77b1`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83279915 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83278868 [Test build #28853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28853/consoleFull) for PR 5005 at commit [`6ba77b1`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83278308 to run mllib nnls use "export solver=mllib" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83277484 @tmyklebu fixed it...thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user tmyklebu commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83239328 The console output on the link provided says, among other things, this: Could not find Apache license headers in the following files: !? /home/j

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83222414 any idea why the tests are failing ? they ran fine in my local testing --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83127440 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83127405 [Test build #28828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28828/consoleFull) for PR 5005 at commit [`b66cf7f`](https://githu

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83127436 [Test build #28828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28828/consoleFull) for PR 5005 at commit [`b66cf7f`](https://gith

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83126362 On the whole flow I don't see much difference...I merged with jblas and cleaned the old NNLS from my code...Also I compared on one thread now... Breeze NNLS:

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user debasish83 commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83058133 I will time the whole flow...most likely this does not even show up --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [ML] SPARK-2426: Integrate Breeze NNLS with ML...

Github user dlwh commented on the pull request: https://github.com/apache/spark/pull/5005#issuecomment-83054484 That's big enough that it's worth fixing. Are there any confounds? On Mar 18, 2015 9:27 AM, "Debasish Das" wrote: > I updated with the @mengxr