yuxiangw commented on a change in pull request #9220: Signum optimizer URL: https://github.com/apache/incubator-mxnet/pull/9220#discussion_r160519689
########## File path: python/mxnet/optimizer.py ########## @@ -534,6 +535,66 @@ def update_multi_precision(self, index, weight, grad, state): self._update_impl(index, weight, grad, state, multi_precision=use_multi_precision) +@register +class Signum(Optimizer): + """The Signum optimizer that takes the sign of gradient or momentum. + + The optimizer updates the weight by: + + rescaled_grad = rescale_grad * clip(grad, clip_gradient) + wd * weight + state = momentum * state + (1-momentum)*rescaled_grad + weight = (1 - lr * wd_lh) * weight - lr * sign(state) Review comment: It is an alternative weight decay. See the descriptions. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services