Re: Trainning a spark ml linear regresion model fail after migrating from 1.5.2 to 1.6.1
Yes, WeightedLeastSquares can not solve some ill-conditioned problem currently, the community members have paid some efforts to resolve it (SPARK-13777). For the work around, you can set the solver to "l-bfgs" which will train the LogisticRegressionModel by L-BFGS optimization method. 2016-06-09 7:37 GMT-07:00 chaz2505 : > I ran into this problem too - it's because WeightedLeastSquares (added in > 1.6.0 SPARK-10668) is being used on an ill-conditioned problem > (SPARK-11918). I guess because of the one hot encoding. To get around it > you > need to ensure WeightedLeastSquares isn't used. Set parameters to make the > following false: > > $(solver) == "auto" && $(elasticNetParam) == 0.0 && > numFeatures <= WeightedLeastSquares.MAX_NUM_FEATURES) || $(solver) == > "normal" > > Hope this helps > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111p27128.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Trainning a spark ml linear regresion model fail after migrating from 1.5.2 to 1.6.1
I ran into this problem too - it's because WeightedLeastSquares (added in 1.6.0 SPARK-10668) is being used on an ill-conditioned problem (SPARK-11918). I guess because of the one hot encoding. To get around it you need to ensure WeightedLeastSquares isn't used. Set parameters to make the following false: $(solver) == "auto" && $(elasticNetParam) == 0.0 && numFeatures <= WeightedLeastSquares.MAX_NUM_FEATURES) || $(solver) == "normal" Hope this helps -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111p27128.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Trainning a spark ml linear regresion model fail after migrating from 1.5.2 to 1.6.1
here is a gist with the minimal code and data http://gist.github.com/anonymous/aca8ba5841404ea092f9efcc658c5d57 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111p27116.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Trainning a spark ml linear regresion model fail after migrating from 1.5.2 to 1.6.1
Hi, Is it me only to *not* see the snippets? Could you please gist 'em => https://gist.github.com ? Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Wed, Jun 8, 2016 at 10:22 AM, philippe v wrote: > I use spark-ml to train a linear regression model. It worked perfectly with > spark version 1.5.2 but now with 1.6.1 I get the following error : > > > > Here is a minimal code : > > > > And input.csv data > > > > the pom.xml > > > > > How can I fix it ? > > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org