Hi Xin, If you take a look at the model you trained, the intercept from Spark is significantly smaller than StatsModel, and the intercept represents a prior on categories in LOR which causes the low accuracy in Spark implementation. In LogisticRegressionWithLBFGS, the intercept is regularized due to the implementation of Updater, and the intercept should not be regularized.
In the new pipleline APIs, a LOR with elasticNet is implemented, and the intercept is properly handled. https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala As you can see the tests, https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala the result is exactly the same as R now. BTW, in both version, the feature scalings are done before training, and we train the model in scaled space but transform the model weights back to original space. The only difference is in the mllib version, LogisticRegressionWithLBFGS regularizes the intercept while in the ml version, the intercept is excluded from regularization. As a result, if lambda is zero, the model should be the same. On Wed, May 20, 2015 at 3:42 PM, Xin Liu <liuxin...@gmail.com> wrote: > Hi, > > I have tried a few models in Mllib to train a LogisticRegression model. > However, I consistently get much better results using other libraries such > as statsmodel (which gives similar results as R) in terms of AUC. For > illustration purpose, I used a small data (I have tried much bigger data) > http://www.ats.ucla.edu/stat/data/binary.csv in > http://www.ats.ucla.edu/stat/r/dae/logit.htm > > Here is the snippet of my usage of LogisticRegressionWithLBFGS. > > val algorithm = new LogisticRegressionWithLBFGS > algorithm.setIntercept(true) > algorithm.optimizer > .setNumIterations(100) > .setRegParam(0.01) > .setConvergenceTol(1e-5) > val model = algorithm.run(training) > model.clearThreshold() > val scoreAndLabels = test.map { point => > val score = model.predict(point.features) > (score, point.label) > } > val metrics = new BinaryClassificationMetrics(scoreAndLabels) > val auROC = metrics.areaUnderROC() > > I did a (0.6, 0.4) split for training/test. The response is "admit" and > features are "GRE score", "GPA", and "college Rank". > > Spark: > Weights (GRE, GPA, Rank): > [0.0011576276331509304,0.048544858567336854,-0.394202150286076] > Intercept: -0.6488972641282202 > Area under ROC: 0.6294070512820512 > > StatsModel: > Weights [0.0018, 0.7220, -0.3148] > Intercept: -3.5913 > Area under ROC: 0.69 > > The weights from statsmodel seems more reasonable if you consider for a one > unit increase in gpa, the log odds of being admitted to graduate school > increases by 0.72 in statsmodel than 0.04 in Spark. > > I have seen much bigger difference with other data. So my question is has > anyone compared the results with other libraries and is anything wrong with > my code to invoke LogisticRegressionWithLBFGS? > > As the real data I am processing is pretty big and really want to use Spark > to get this to work. Please let me know if you have similar experience and > how you resolve it. > > Thanks, > Xin --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
