[
https://issues.apache.org/jira/browse/SPARK-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225295#comment-14225295
]
Reza Zadeh commented on SPARK-1503:
-----------------------------------
Thanks for this design doc Aaron.
It looks good for the first implementation to support composite objectives, A&T
updates, but I'm not sure about backtracking.
Have you thought about how many passes through the data backtracking can
require? As you mention: per backtracking inner loop iteration, we need 2
shuffles. But how many iterations of the inner backtracking loop can be
typical? Could it be better in a distributed environment to avoid backtracking
and use a constant step size? Especially for the well-behaved objectives we
have (e.g. logistic regression). If a constant step size works fast enough, we
should do that first - what do you think?
Please try a constant step size first - if it works, that will bring down the
communication cost greatly.
It’s fine that the initial implementation will not include the linear operator
optimizations present in TFOCS. That’s a good call. In general let’s try to
keep the first PR as simple as possible.
Please make sure your code adheres to this example for LBFGS, so we can swap
out the Optimizer with your contribution:
http://spark.apache.org/docs/latest/mllib-optimization.html#l-bfgs
> Implement Nesterov's accelerated first-order method
> ---------------------------------------------------
>
> Key: SPARK-1503
> URL: https://issues.apache.org/jira/browse/SPARK-1503
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Xiangrui Meng
> Assignee: Aaron Staple
>
> Nesterov's accelerated first-order method is a drop-in replacement for
> steepest descent but it converges much faster. We should implement this
> method and compare its performance with existing algorithms, including SGD
> and L-BFGS.
> TFOCS (http://cvxr.com/tfocs/) is a reference implementation of Nesterov's
> method and its variants on composite objectives.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
