chrishkchris commented on a change in pull request #468: Distributted module
URL: https://github.com/apache/incubator-singa/pull/468#discussion_r317889630
##########
File path: python/singa/autograd.py
##########
@@ -1286,25 +1287,26 @@ def set_params(self, **parameters):
class _BatchNorm2d(Operation):
- def __init__(self, handle, name=None):
+ def __init__(self, handle, running_mean, running_var, name=None):
super(_BatchNorm2d, self).__init__(name)
self.handle = handle
+ self.running_mean = running_mean.data
+ self.running_var = running_var.data
- def forward(self, x, scale, bias, running_mean, running_var):
- self.running_mean = running_mean
- self.running_var = running_var
+ def forward(self, x, scale, bias):
if training:
if isinstance(self.handle, singa.CudnnBatchNormHandle):
y, mean, var = singa.GpuBatchNormForwardTraining(
- self.handle, x, scale, bias, running_mean, running_var
+ self.handle, x, scale, bias, self.running_mean,
self.running_var
Review comment:
The following is the resnet18 training using CPU on CIFAR10 in the first few
epochs. CPU is slow so I trained only a few epochs
```
Start intialization............
Epoch=0:
100%|████████████████████████████████████████████████████████████████████████|
1562/1562 [2:09:57<00:00, 5.03s/it]
Training loss = 2233.394769, training accuracy = 0.490297
Test accuracy = 0.636218
Epoch=1:
100%|███████████████████████████████████████████████████████████████████████████████████████████|
1562/1562 [2:10:00<00:00, 4.98s/it]
Training loss = 1474.432049, training accuracy = 0.666633
Test accuracy = 0.678986
Epoch=2:
100%|███████████████████████████████████████████████████████████████████████████████████████████|
1562/1562 [2:10:11<00:00, 5.00s/it]
Training loss = 1163.035850, training accuracy = 0.741717
Test accuracy = 0.738181
Epoch=3:
100%|███████████████████████████████████████████████████████████████████████████████████████████|
1562/1562 [2:10:31<00:00, 5.03s/it]
Training loss = 979.977119, training accuracy = 0.782570
Test accuracy = 0.800581
Epoch=4:
100%|███████████████████████████████████████████████████████████████████████████████████████████|
1562/1562 [2:10:10<00:00, 4.98s/it]
Training loss = 872.811802, training accuracy = 0.806098
Test accuracy = 0.813902
Epoch=5:
100%|███████████████████████████████████████████████████████████████████████████████████████████|
1562/1562 [2:10:05<00:00, 4.99s/it]
Training loss = 782.525783, training accuracy = 0.826144
Test accuracy = 0.832232
```
The training loss decreases normally. Therefore seems the CPU batch norm is
working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
