Hi DB, Thanks for doing the comparison! What were the running times for fortran/breeze/riso?
Best, Xiangrui On Thu, Mar 6, 2014 at 4:21 PM, DB Tsai <dbt...@alpinenow.com> wrote: > Hi David, > > I can converge to the same result with your breeze LBFGS and Fortran > implementations now. Probably, I made some mistakes when I tried > breeze before. I apologize that I claimed it's not stable. > > See the test case in BreezeLBFGSSuite.scala > https://github.com/AlpineNow/spark/tree/dbtsai-breezeLBFGS > > This is training multinomial logistic regression against iris dataset, > and both optimizers can train the models with 98% training accuracy. > > There are two issues to use Breeze in Spark, > > 1) When the gradientSum and lossSum are computed distributively in > custom defined DiffFunction which will be passed into your optimizer, > Spark will complain LBFGS class is not serializable. In > BreezeLBFGS.scala, I've to convert RDD to array to make it work > locally. It should be easy to fix by just having LBFGS to implement > Serializable. > > 2) Breeze computes redundant gradient and loss. See the following log > from both Fortran and Breeze implementations. > > Thanks. > > Fortran: > Iteration -1: loss 1.3862943611198926, diff 1.0 > Iteration 0: loss 1.5846343143210866, diff 0.14307193024217352 > Iteration 1: loss 1.1242501524477688, diff 0.29053004039012126 > Iteration 2: loss 1.0930151243303563, diff 0.027782962952189336 > Iteration 3: loss 1.054036932835569, diff 0.03566113127440601 > Iteration 4: loss 0.9907956302751622, diff 0.05999907649459571 > Iteration 5: loss 0.9184205380342829, diff 0.07304737423337761 > Iteration 6: loss 0.8259870936519937, diff 0.10064381175132982 > Iteration 7: loss 0.6327447552109574, diff 0.23395293458364716 > Iteration 8: loss 0.5534101162436359, diff 0.1253815427665277 > Iteration 9: loss 0.4045020086612566, diff 0.26907321376758075 > Iteration 10: loss 0.3078824990823728, diff 0.23885980452569627 > > Breeze: > Iteration -1: loss 1.3862943611198926, diff 1.0 > Mar 6, 2014 3:59:11 PM com.github.fommil.netlib.BLAS <clinit> > WARNING: Failed to load implementation from: > com.github.fommil.netlib.NativeSystemBLAS > Mar 6, 2014 3:59:11 PM com.github.fommil.netlib.BLAS <clinit> > WARNING: Failed to load implementation from: > com.github.fommil.netlib.NativeRefBLAS > Iteration 0: loss 1.3862943611198926, diff 0.0 > Iteration 1: loss 1.5846343143210866, diff 0.14307193024217352 > Iteration 2: loss 1.1242501524477688, diff 0.29053004039012126 > Iteration 3: loss 1.1242501524477688, diff 0.0 > Iteration 4: loss 1.1242501524477688, diff 0.0 > Iteration 5: loss 1.0930151243303563, diff 0.027782962952189336 > Iteration 6: loss 1.0930151243303563, diff 0.0 > Iteration 7: loss 1.0930151243303563, diff 0.0 > Iteration 8: loss 1.054036932835569, diff 0.03566113127440601 > Iteration 9: loss 1.054036932835569, diff 0.0 > Iteration 10: loss 1.054036932835569, diff 0.0 > Iteration 11: loss 0.9907956302751622, diff 0.05999907649459571 > Iteration 12: loss 0.9907956302751622, diff 0.0 > Iteration 13: loss 0.9907956302751622, diff 0.0 > Iteration 14: loss 0.9184205380342829, diff 0.07304737423337761 > Iteration 15: loss 0.9184205380342829, diff 0.0 > Iteration 16: loss 0.9184205380342829, diff 0.0 > Iteration 17: loss 0.8259870936519939, diff 0.1006438117513297 > Iteration 18: loss 0.8259870936519939, diff 0.0 > Iteration 19: loss 0.8259870936519939, diff 0.0 > Iteration 20: loss 0.6327447552109576, diff 0.233952934583647 > Iteration 21: loss 0.6327447552109576, diff 0.0 > Iteration 22: loss 0.6327447552109576, diff 0.0 > Iteration 23: loss 0.5534101162436362, diff 0.12538154276652747 > Iteration 24: loss 0.5534101162436362, diff 0.0 > Iteration 25: loss 0.5534101162436362, diff 0.0 > Iteration 26: loss 0.40450200866125635, diff 0.2690732137675816 > Iteration 27: loss 0.40450200866125635, diff 0.0 > Iteration 28: loss 0.40450200866125635, diff 0.0 > Iteration 29: loss 0.30788249908237314, diff 0.23885980452569502 > > Sincerely, > > DB Tsai > Machine Learning Engineer > Alpine Data Labs > -------------------------------------- > Web: http://alpinenow.com/ > > > On Wed, Mar 5, 2014 at 2:00 PM, David Hall <d...@cs.berkeley.edu> wrote: >> On Wed, Mar 5, 2014 at 1:57 PM, DB Tsai <dbt...@alpinenow.com> wrote: >> >>> Hi David, >>> >>> On Tue, Mar 4, 2014 at 8:13 PM, dlwh <david.lw.h...@gmail.com> wrote: >>> > I'm happy to help fix any problems. I've verified at points that the >>> > implementation gives the exact same sequence of iterates for a few >>> different >>> > functions (with a particular line search) as the c port of lbfgs. So I'm >>> a >>> > little surprised it fails where Fortran succeeds... but only a little. >>> This >>> > was fixed late last year. >>> I'm working on a reproducible test case using breeze vs fortran >>> implementation to show the problem I've run into. The test will be in >>> one of the test cases in my Spark fork, is it okay for you to >>> investigate the issue? Or do I need to make it as a standalone test? >>> >> >> >> Um, as long as it wouldn't be too hard to pull out. >> >> >>> >>> Will send you the test later today. >>> >>> Thanks. >>> >>> Sincerely, >>> >>> DB Tsai >>> Machine Learning Engineer >>> Alpine Data Labs >>> -------------------------------------- >>> Web: http://alpinenow.com/ >>>