wangwei created SINGA-71:
----------------------------
Summary: Fix a buf from repeated averaging gradients from workers
Key: SINGA-71
URL: https://issues.apache.org/jira/browse/SINGA-71
Project: Singa
Issue Type: Bug
Reporter: wangwei
Assignee: wangwei
For workers from the same group, if they share parameters, gradients of
parameters should be averaged. SINGA averages the gradients by
* aggregates gradients from all workers
* calls Updater::Update() passing the scale_grad = 1.0 / worker_group_size.
* The Update function multiplies the gradient with scale_grad.
The bug was from the Trainer::HandleUpdate() which averaged the gradients from
local workers. The bug has been fixed by removing the averaging operation.
Gradients from local workers are (only) summed before sending to servers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
