ymjiang edited a comment on issue #14485: Any suggestion to accelerate parameter update on PS for distributed training? URL: https://github.com/apache/incubator-mxnet/issues/14485#issuecomment-475472581 @apeforest Thank you very much for your suggestions! I will try `MXNET_KVSTORE_REDUCTION_NTHREADS` to see if it works. I have a question In terms of option-2. I do not find any code in `kvstore_dist_server.h` that is related to `update_on_kvstore`. So the parameter server will still keep updating the gradients, no matter what value you set `update_on_kvstore` to. In this case, I actually do not understand "the worker that will update weights locally with the gradients". How could this happen if the parameter server already aggregates and updates the gradients? ----updates---- It seems that `MXNET_KVSTORE_REDUCTION_NTHREADS` does not improve performance.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
