Dmitry, I have skimmed through the current samsara implementation and your input below and have some initial questions, for starters I would like to take advantage of the work you've already done and bring that into production state, given that, here are some thoughts/questions:
1) What work does the pull request below still need done, unit tests, integration tests , seems like the implementation is complete from reading the code but I'm coming into this new so not sure here? 2) It seems to be that your points 2 and 3 could be written as generic mahout modules that can be used by all algorithms as appropriate, what do you think? 3) On the feature extraction per R like formula can you elaborate more here, are you talking about feature extraction using R like dataframes and operators? More later as I read through the papers. ________________________________ From: Dmitriy Lyubimov <dlie...@gmail.com> Sent: Friday, February 17, 2017 1:45 PM To: dev@mahout.apache.org Subject: Re: Contributing an algorithm for samsara in particular, this is the samsara implementation of double-weighed als : https://github.com/apache/mahout/pull/14/files#diff-0fbeb8b848ed0c5e3f782c72569cf626 MAHOUT-1365 Implicit feedback ALS-WR by dlyubimov · Pull Request #14 · apache/mahout · GitHub<https://github.com/apache/mahout/pull/14/files#diff-0fbeb8b848ed0c5e3f782c72569cf626> github.com mahout - Mirror of Apache Mahout On Fri, Feb 17, 2017 at 1:33 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote: > Jim, > > if ALS is of interest, and as far as weighed ALS is concerned (since we > already have trivial regularized ALS in the "decompositions" package), > here's uncommitted samsara-compatible patch from a while back: > https://issues.apache.org/jira/browse/MAHOUT-1365 [MAHOUT-1365] Weighted ALS-WR iterator for Spark - ASF JIRA<https://issues.apache.org/jira/browse/MAHOUT-1365> issues.apache.org Given preference P and confidence C distributed sparse matrices, compute ALS-WR solution for implicit feedback (Spark Bagel version). Following Hu-Koren-Volynsky ... > > it combines weights on both data points (a.k.a "implicit feedback" als) > and regularization rates (paper references are given). We combine both > approaches in one (which is novel, i guess, but yet simple enough). > Obviously the final solver can also be used as pure reg rate regularized if > wanted, making it equivalent to one of the papers. > > You may know implicit feedback paper from mllib's implicit als, but unlike > it was done over there (as a use case sort problem that takes input before > even features were extracted), we split the problem into pure algebraic > solver (double-weighed ALS math) and leave the feature extraction outside > of this issue per se (it can be added as a separate adapter). > > The reason for that is that the specific use-case oriented implementation > does not necessarily leave the space for feature extraction that is > different from described use case of partially consumed streamed videos in > the paper. (e.g., instead of videos one could count visits or clicks or > add-to-cart events which may need additional hyperparameter found for them > as part of feature extraction and converting observations into "weghts"). > > The biggest problem with these ALS methods however is that all > hyperparameters require multidimensional crossvalidation and optimization. > I think i mentioned it before as list of desired solutions, as it stands, > Mahout does not have hyperarameter fitting routine. > > In practice, when using these kind of ALS, we have a case of > multidimensional hyperparameter optimization. One of them comes from the > fitter (reg rate, or base reg rate in case of weighed regularization), and > the others come from feature extraction process. E.g., in original paper > they introduce (at least) 2 formulas to extract measure weighs from the > streaming video observations, and each of them had one parameter, alhpa, > which in context of the whole problem becomes effectively yet another > hyperparameter to fit. In other use cases when your confidence measurement > may be coming from different sources and observations, the confidence > extraction may actually have even more hyperparameters to fit than just > one. And when we have a multidimensional case, simple approaches (like grid > or random search) become either cost prohibitive or ineffective, due to the > curse of dimensionality. > > At the time i was contributing that method, i was using it in conjunction > with multidimensional bayesian optimizer, but the company that i wrote it > for did not have it approved for contribution (unlike weighed als) at that > time. > > Anyhow, perhaps you could read the algebra in both ALS papers there and > ask questions, and we could worry about hyperparameter optimization a bit > later and performance a bit later. > > On the feature extraction front (as in implicit feedback als per Koren > etc.), this is an ideal use case for more general R-like formula approach, > which is also on desired list of things to have. > > So i guess we have 3 problems really here: > (1) double-weighed ALS > (2) bayesian optimization and crossvalidation in an n-dimensional > hyperparameter space > (3) feature extraction per (preferrably R-like) formula. > > > -d > > > On Fri, Feb 17, 2017 at 10:11 AM, Andrew Palumbo <ap....@outlook.com> > wrote: > >> +1 to glms >> >> >> >> Sent from my Verizon Wireless 4G LTE smartphone >> >> >> -------- Original message -------- >> From: Trevor Grant <trevor.d.gr...@gmail.com> >> Date: 02/17/2017 6:56 AM (GMT-08:00) >> To: dev@mahout.apache.org >> Subject: Re: Contributing an algorithm for samsara >> >> Jim is right, and I would take it one further and say, it would be best to >> implement GLMs https://en.wikipedia.org/wiki/Generalized_linear_model , [http://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Biologist_and_statistician_Ronald_Fisher.jpg/200px-Biologist_and_statistician_Ronald_Fisher.jpg]<https://en.wikipedia.org/wiki/Generalized_linear_model> Generalized linear model - Wikipedia<https://en.wikipedia.org/wiki/Generalized_linear_model> en.wikipedia.org Part of a series on Statistics: Regression analysis; Models; Linear regression; Simple regression; Ordinary least squares; Polynomial regression; General linear model >> from there a Logistic regression is a trivial extension. >> >> Buyer beware- GLMs will be a bit of work- doable, but that would be >> jumping >> in neck first for both Jim and Saikat... >> >> MAHOUT-1928 and MAHOUT-1929 >> >> https://issues.apache.org/jira/browse/MAHOUT-1925?jql=projec >> t%20%3D%20MAHOUT%20AND%20component%20%3D%20Algorithms%20AND% >> 20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C% >> 20priority%20DESC%2C%20created%20ASC >> >> ^^ currently open JIRAs around Algorithms- you'll see Logistic and GLMs >> are >> in there. >> >> If you have an algorithm you are particularly intimate with, or explicitly >> need/want- feel free to open a JIRA and assign to yourself. >> >> There is also a case to be made for implementing the ALS... >> >> 1) It's a much better 'beginner' project. >> 2) Mahout has some world class Recommenders, a toy ALS implementation >> might >> help us think through how the other reccomenders (e.g. CCO) will 'fit' >> into >> the framework. E.g. ALS being the toy-prototype reccomender that helps us >> think through building out that section of the framework. >> >> >> >> Trevor Grant >> Data Scientist >> https://github.com/rawkintrevo [https://avatars3.githubusercontent.com/u/5852441?v=3&s=400]<https://github.com/rawkintrevo> rawkintrevo (Trevor Grant) · GitHub<https://github.com/rawkintrevo> github.com rawkintrevo has 22 repositories available. Follow their code on GitHub. >> http://stackexchange.com/users/3002022/rawkintrevo User rawkintrevo - Stack Exchange<http://stackexchange.com/users/3002022/rawkintrevo> stackexchange.com Fortuna Audaces Iuvat ~Chance Favors the Bold. top accounts reputation activity favorites subscriptions. Top Questions >> http://trevorgrant.org [https://s0.wp.com/i/blank.jpg]<http://trevorgrant.org/> The musings of rawkintrevo<http://trevorgrant.org/> trevorgrant.org Hot-rodder, opera enthusiast, mad data scientist; a man for all seasons. >> >> *"Fortunate is he, who is able to know the causes of things." -Virgil* >> >> >> On Fri, Feb 17, 2017 at 7:59 AM, Jim Jagielski <j...@jagunet.com> wrote: >> >> > My own thoughts are that logistic regression seems a more "generalized" >> > and hence more useful algo to be factored in... At least in the >> > use cases that I've been toying with. >> > >> > So I'd like to help out with that if wanted... >> > >> > > On Feb 9, 2017, at 3:59 PM, Saikat Kanjilal <sxk1...@hotmail.com> >> wrote: >> > > >> > > Trevor et al, >> > > >> > > I'd like to contribute an algorithm or two in samsara using spark as I >> > would like to do a compare and contrast with mahout with R server for a >> > data science pipeline, machine learning repo that I'm working on, in >> > looking at the list of algorithms (https://mahout.apache.org/ >> > users/basics/algorithms.html) is there an algorithm for spark that would >> > be beneficial for the community, my use cases would typically be around >> > clustering or real time machine learning for building recommendations on >> > the fly. The algorithms I see that could potentially be useful are: >> 1) >> > Matrix Factorization with ALS 2) Logistic regression with SVD. >> > > >> > > Apache Mahout: Scalable machine learning and data mining< >> > https://mahout.apache.org/users/basics/algorithms.html> >> > > mahout.apache.org >> > > Mahout 0.12.0 Features by Engine¶ Single Machine MapReduce Spark H2O >> > Flink; Mahout Math-Scala Core Library and Scala DSL >> > > >> > > >> > > >> > > Any thoughts/guidance or recommendations would be very helpful. >> > > Thanks in advance. >> > >> > >> > >
