[
https://issues.apache.org/jira/browse/MADLIB-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480950#comment-16480950
]
Nikhil edited comment on MADLIB-1210 at 5/18/18 5:30 PM:
---------------------------------------------------------
We tested the model update code using 2d rosenbrock as the loss function. We
started our model at the same initial point (-1.2,1) as
https://jlmelville.github.io/mize/nesterov.html#testing_with_rosenbrock.
Here are the results for 100 iterations
No momentum
{code}
24.200000
5.352912
4.117790
4.114552
4.111402
4.108251
4.105099
4.101944
4.098789
4.095632
4.092474
4.089314
4.086153
4.082990
4.079825
4.076660
4.073493
4.070324
4.067154
4.063982
4.060809
4.057635
4.054458
4.051281
4.048102
4.044921
4.041739
4.038556
4.035371
4.032184
4.028996
4.025806
4.022615
4.019423
4.016229
4.013033
4.009836
4.006637
4.003437
4.000235
3.997032
3.993827
3.990620
3.987412
3.984203
3.980992
3.977779
3.974565
3.971349
3.968132
3.964913
3.961692
3.958470
3.955246
3.952021
3.948794
3.945566
3.942336
3.939104
3.935871
3.932636
3.929400
3.926162
3.922922
3.919681
3.916438
3.913193
3.909947
3.906699
3.903450
3.900199
3.896946
3.893692
3.890436
3.887178
3.883919
3.880658
3.877395
3.874131
3.870865
3.867598
3.864328
3.861057
3.857785
3.854510
3.851234
3.847957
3.844677
3.841396
3.838113
3.834829
3.831543
3.828255
3.824965
3.821674
3.818380
3.815086
3.811789
3.808491
3.805191
{code}
Momentum 0.95
{code}
24.200000
5.352912
25.541437
22.488091
4.321887
18.826923
17.960659
3.707286
13.955235
17.239317
7.697164
3.489292
11.875827
11.827455
3.901931
2.871476
5.476991
5.929791
3.806193
1.505419
1.248258
3.483557
6.629838
8.088645
6.408485
2.952019
0.718966
1.373433
3.877286
5.697909
5.039226
2.389898
0.400048
1.521285
4.366073
4.548812
1.820190
0.312452
1.549169
3.433944
3.729078
2.208632
0.560836
0.647038
2.241226
3.011759
1.885178
0.545087
0.568845
1.572851
2.280232
1.995975
1.078072
0.460328
0.727861
1.500870
1.845502
1.390100
0.704716
0.500882
0.868943
1.351409
1.475961
1.155795
0.705216
0.531766
0.763056
1.128218
1.235803
0.984102
0.642734
0.534194
0.714490
0.966279
1.034820
0.855987
0.596998
0.494346
0.625975
0.827827
0.864583
0.690345
0.490731
0.459419
0.591974
0.715626
0.682750
0.519027
0.391734
0.427605
0.561741
0.607965
0.494904
0.356307
0.346229
0.444850
0.501926
0.430699
0.311241
0.286977
{code}
Nesterov's Momentum 0.95
{code}
24.200000
34.960154
7.039334
5.020041
3.845406
3.746363
3.668701
3.592084
3.515410
3.438569
3.361510
3.284194
3.206569
3.128580
3.050171
2.971284
2.891866
2.811859
2.731208
2.649861
2.567765
2.484871
2.401137
2.316527
2.231016
2.144599
2.057289
1.969133
1.880221
1.790697
1.700774
1.610753
1.521032
1.432110
1.344572
1.259050
1.176144
1.096318
1.019791
0.946485
0.876085
0.808253
0.742921
0.680487
0.621746
0.567539
0.518352
0.474130
0.434415
0.398615
0.366191
0.336689
0.309729
0.285020
0.262372
0.241651
0.222727
0.205447
0.189651
0.175190
0.161932
0.149761
0.138582
0.128309
0.118864
0.110177
0.102180
0.094815
0.088027
0.081768
0.075992
0.070660
0.065736
0.061186
0.056979
0.053087
0.049485
0.046151
0.043062
0.040199
0.037545
0.035083
0.032798
0.030677
0.028708
0.026877
0.025176
0.023593
0.022121
0.020751
0.019475
0.018287
0.017179
0.016147
0.015184
0.014285
0.013446
0.012663
0.011931
0.011248
{code}
was (Author: nikhilkak):
We tested the model update code using rosenbrock as the loss function. We
started our model at the same initial point (-1.2,1) as
https://jlmelville.github.io/mize/nesterov.html#testing_with_rosenbrock.
Here are the results for 100 iterations
No momentum
{code}
24.200000
5.352912
4.117790
4.114552
4.111402
4.108251
4.105099
4.101944
4.098789
4.095632
4.092474
4.089314
4.086153
4.082990
4.079825
4.076660
4.073493
4.070324
4.067154
4.063982
4.060809
4.057635
4.054458
4.051281
4.048102
4.044921
4.041739
4.038556
4.035371
4.032184
4.028996
4.025806
4.022615
4.019423
4.016229
4.013033
4.009836
4.006637
4.003437
4.000235
3.997032
3.993827
3.990620
3.987412
3.984203
3.980992
3.977779
3.974565
3.971349
3.968132
3.964913
3.961692
3.958470
3.955246
3.952021
3.948794
3.945566
3.942336
3.939104
3.935871
3.932636
3.929400
3.926162
3.922922
3.919681
3.916438
3.913193
3.909947
3.906699
3.903450
3.900199
3.896946
3.893692
3.890436
3.887178
3.883919
3.880658
3.877395
3.874131
3.870865
3.867598
3.864328
3.861057
3.857785
3.854510
3.851234
3.847957
3.844677
3.841396
3.838113
3.834829
3.831543
3.828255
3.824965
3.821674
3.818380
3.815086
3.811789
3.808491
3.805191
{code}
Momentum 0.95
{code}
24.200000
5.352912
25.541437
22.488091
4.321887
18.826923
17.960659
3.707286
13.955235
17.239317
7.697164
3.489292
11.875827
11.827455
3.901931
2.871476
5.476991
5.929791
3.806193
1.505419
1.248258
3.483557
6.629838
8.088645
6.408485
2.952019
0.718966
1.373433
3.877286
5.697909
5.039226
2.389898
0.400048
1.521285
4.366073
4.548812
1.820190
0.312452
1.549169
3.433944
3.729078
2.208632
0.560836
0.647038
2.241226
3.011759
1.885178
0.545087
0.568845
1.572851
2.280232
1.995975
1.078072
0.460328
0.727861
1.500870
1.845502
1.390100
0.704716
0.500882
0.868943
1.351409
1.475961
1.155795
0.705216
0.531766
0.763056
1.128218
1.235803
0.984102
0.642734
0.534194
0.714490
0.966279
1.034820
0.855987
0.596998
0.494346
0.625975
0.827827
0.864583
0.690345
0.490731
0.459419
0.591974
0.715626
0.682750
0.519027
0.391734
0.427605
0.561741
0.607965
0.494904
0.356307
0.346229
0.444850
0.501926
0.430699
0.311241
0.286977
{code}
Nesterov's Momentum 0.95
{code}
24.200000
34.960154
7.039334
5.020041
3.845406
3.746363
3.668701
3.592084
3.515410
3.438569
3.361510
3.284194
3.206569
3.128580
3.050171
2.971284
2.891866
2.811859
2.731208
2.649861
2.567765
2.484871
2.401137
2.316527
2.231016
2.144599
2.057289
1.969133
1.880221
1.790697
1.700774
1.610753
1.521032
1.432110
1.344572
1.259050
1.176144
1.096318
1.019791
0.946485
0.876085
0.808253
0.742921
0.680487
0.621746
0.567539
0.518352
0.474130
0.434415
0.398615
0.366191
0.336689
0.309729
0.285020
0.262372
0.241651
0.222727
0.205447
0.189651
0.175190
0.161932
0.149761
0.138582
0.128309
0.118864
0.110177
0.102180
0.094815
0.088027
0.081768
0.075992
0.070660
0.065736
0.061186
0.056979
0.053087
0.049485
0.046151
0.043062
0.040199
0.037545
0.035083
0.032798
0.030677
0.028708
0.026877
0.025176
0.023593
0.022121
0.020751
0.019475
0.018287
0.017179
0.016147
0.015184
0.014285
0.013446
0.012663
0.011931
0.011248
{code}
> Add momentum methods to MLP
> ---------------------------
>
> Key: MADLIB-1210
> URL: https://issues.apache.org/jira/browse/MADLIB-1210
> Project: Apache MADlib
> Issue Type: New Feature
> Components: Module: Neural Networks
> Reporter: Frank McQuillan
> Priority: Major
> Fix For: v1.15
>
>
> Story
> As a data scientist,
> I want to use momentum methods in MLP,
> so that I get significantly better convergence behavior.
> Details
> Adding momentum will get the MADlib MLP algorithm closer to state of the art.
> 1) Implement momentum term, default value ~0.9
> Ref [1]:
> "Momentum update is another approach that almost always enjoys better
> converge rates on deep networks."
> 2) Implement Nesterov momentum, default TRUE
> Ref [1]:
> "Nesterov Momentum is a slightly different version of the momentum update
> that has recently been gaining popularity. It enjoys stronger theoretical
> converge guarantees for convex functions and in practice it also consistently
> works slightly better than standard momentum."
> Ref [2]
> "Nesterov’s accelerated gradient (abbrv. NAG; Nesterov, 1983) is a
> first-order optimization method which is proven to have a better convergence
> rate guarantee than gradient descent for general convex functions with
> Lipshitz-continuous derivatives (O(1/T2) versus O(1/T))"
> Interface
> There are 2 new optimization params for momentum, which apply for both
> classification and regression:
> {code}
> 'learning_rate_init = <value>,
> learning_rate_policy = <value>,
> gamma = <value>,
> power = <value>,
> iterations_per_step = <value>,
> n_iterations = <value>,
> n_tries = <value>,
> lambda = <value>,
> tolerance = <value>,
> batch_size = <value>,
> n_epochs = <value>,
> momentum = <value>,
> nesterov_momentum= <value>'
> momentum
> Default: 0.9. Momentum can help accelerate learning and
> avoid local minima when using gradient descent. Value must be in the
> range 0 to 1, where 0 means no momentum.
> nesterov_momentum
> Default: TRUE. Nesterov momentum can provide better results than using
> classical momentum alone, due to its look ahead characteristics.
> In classical momentum you first correct velocity and step with that
> velocity, whereas in Nesterov momentum you first step in the velocity
> direction then make a correction to the velocity vector based on
> new location.
> Nesterov momentum is only used when the 'momentum' parameter is > 0.
> {code}
> Open questions
> 1) Does momentum and Nesterov momentum work equally well with and without
> mini-batching?
> Is there any guidance we need to give to users on this?
> Acceptance
> [1] Compare the usefulness of momentum with and without Nesterov, mini-batch,
> and SGD. Use a 2D Rosenbrock function to compare in a similar way to test
> ref [100] in the comment further down, i.e., loss by iteration number.
> [2] Use another well behaved function (TBD) and run similar tests as in [1]
> above.
> [3] Test with MNIST.
> [4] Test with CIFAR-10 or CIFAR-100
> http://www.cs.toronto.edu/~kriz/cifar.html
> References
> [1] http://cs231n.github.io/neural-networks-3/#sgd
> [2] http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf, a
> link from previous source.
> [3]
> http://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms
> [4]
> http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html
> [5] https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
