yuxiangw commented on a change in pull request #9220: Signum optimizer
URL: https://github.com/apache/incubator-mxnet/pull/9220#discussion_r159721868
##########
File path: python/mxnet/optimizer.py
##########
@@ -529,6 +534,69 @@ def update_multi_precision(self, index, weight, grad,
state):
self._update_impl(index, weight, grad, state,
multi_precision=use_multi_precision)
+
+
+@register
+class Signum(Optimizer):
+ """The SGD optimizer with momentum and weight decay.
+
+ The optimizer updates the weight by::
+
+ rescaled_grad = lr * rescale_grad * clip(grad, clip_gradient) + wd *
weight
+ state = momentum * state + (1-momentum)*rescaled_grad
+ weight = weight - sign(state) - wd_lh * weight
+
+ For details of the update algorithm see
+ :class:`~mxnet.ndarray.signsgd_update` and
:class:`~mxnet.ndarray.signum_update`.
+
+ This optimizer accepts the following parameters in addition to those
accepted
+ by :class:`.Optimizer`.
+
+ Parameters
+ ----------
+ momentum : float, optional
+ The momentum value.
+ wd_lh : float, optitional
+ The amount of decoupled weight decay regularization.
Review comment:
added the temp link to pdf hosted on jeremy's site. will update to arxiv or
a published version when they are ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
