When will Spark 1.4 be available exactly? To answer to "Model selection can be achieved through high lambda resulting lots of zero in the coefficients" : Do you mean that putting a high lambda as a parameter of the logistic regression keeps only a few significant variables and "deletes" the others with a zero in the coefficients? What is a high lambda for you? Is the lambda a parameter available in Spark 1.4 only or can I see it in Spark 1.3?
2015-05-23 0:04 GMT+02:00 Joseph Bradley <jos...@databricks.com>: > If you want to select specific variable combinations by hand, then you > will need to modify the dataset before passing it to the ML algorithm. The > DataFrame API should make that easy to do. > > If you want to have an ML algorithm select variables automatically, then I > would recommend using L1 regularization for now and possibly elastic net > after 1.4 is release, per DB's suggestion. > > If you want detailed model statistics similar to what R provides, I've > created a JIRA for discussing how we should add that functionality to > MLlib. Those types of stats will be added incrementally, but feedback > would be great for prioritization: > https://issues.apache.org/jira/browse/SPARK-7674 > > To answer your question: "How are the weights calculated: is there a > correlation calculation with the variable of interest?" > --> Weights are calculated as with all logistic regression algorithms, by > using convex optimization to minimize a regularized log loss. > > Good luck! > Joseph > > On Fri, May 22, 2015 at 1:07 PM, DB Tsai <dbt...@dbtsai.com> wrote: > >> In Spark 1.4, Logistic Regression with elasticNet is implemented in ML >> pipeline framework. Model selection can be achieved through high >> lambda resulting lots of zero in the coefficients. >> >> Sincerely, >> >> DB Tsai >> ------------------------------------------------------- >> Blog: https://www.dbtsai.com >> >> >> On Fri, May 22, 2015 at 1:19 AM, SparknewUser >> <melanie.galloi...@gmail.com> wrote: >> > I am new in MLlib and in Spark.(I use Scala) >> > >> > I'm trying to understand how LogisticRegressionWithLBFGS and >> > LogisticRegressionWithSGD work. >> > I usually use R to do logistic regressions but now I do it on Spark >> > to be able to analyze Big Data. >> > >> > The model only returns weights and intercept. My problem is that I have >> no >> > information about which variable is significant and which variable I had >> > better >> > to delete to improve my model. I only have the confusion matrix and the >> AUC >> > to evaluate the performance. >> > >> > Is there any way to have information about the variables I put in my >> model? >> > How can I try different variable combinations, do I have to modify the >> > dataset >> > of origin (e.g. delete one or several columns?) >> > How are the weights calculated: is there a correlation calculation with >> the >> > variable >> > of interest? >> > >> > >> > >> > -- >> > View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-how-to-get-the-best-model-with-only-the-most-significant-explanatory-variables-in-LogisticRegr-tp22993.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > -- *Mélanie*
