KexinFeng edited a comment on issue #20293:
URL:
https://github.com/apache/incubator-mxnet/issues/20293#issuecomment-866448263
**Analysis:**
The issue is solved mainly by comparing elemwise_add with elemwise_mul at
operation registration. @barry-jin pointed out an important discrepancy
between these two operators today. The discrepancy is in “registering the
function for creating the node of the operator in a backward pass”:
https://github.com/apache/incubator-mxnet/blob/da4ff3a4dc0bd6a54af3d75c492021d18ba1867b/src/operator/tensor/elemwise_binary_op_basic.cc#L111
https://github.com/apache/incubator-mxnet/blob/da4ff3a4dc0bd6a54af3d75c492021d18ba1867b/src/operator/tensor/elemwise_binary_op_basic.cc#L238
elemwise_mul uses ElemwiseGradUseIn, since calculation of _back_mul depends
on input value. However, elemwise_add uses CloneGradient, the reason of which
is not clear yet. But ElemwiseGradUseNone is another valid option for
elemwise_add . So I tested it by replacing CloneGradient with
ElemwiseGradUseNone .
**Result:**
**The issue is solved!** ./demo2 0 1 specifies the gradient request as
following, and 2021 is the hardcoding head gradient.
```
uint32_t grad_req_1[1] = {0};
uint32_t grad_req_2[1] = {1};
```
```
$~/mxnet/issue2/build2$ ./demo2 0 1
[19:45:23] ../src/nnvm/legacy_json_util.cc:209: Loading symbol saved by
previous version v0.8.0. Attempting to upgrade...
[19:45:23] ../src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[19:45:23] ../src/engine/engine.cc:55: MXNet start using engine: NaiveEngine
[19:45:23] ../src/storage/storage.cc:199: Using Pooled (Naive)
StorageManager for CPU
INPUT 1: 0.4
INPUT 2: 0.5
OUTPUT: 0.9
OUTGRAD: 2021
GRAD 1: 0
GRAD 2: 2021
$~/mxnet/issue2/build2$ ./demo2 1 0
[19:49:59] ../src/nnvm/legacy_json_util.cc:209: Loading symbol saved by
previous version v0.8.0. Attempting to upgrade...
[19:49:59] ../src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[19:49:59] ../src/engine/engine.cc:55: MXNet start using engine: NaiveEngine
[19:49:59] ../src/storage/storage.cc:199: Using Pooled (Naive)
StorageManager for CPU
INPUT 1: 0.4
INPUT 2: 0.5
OUTPUT: 0.9
OUTGRAD: 2021
GRAD 1: 2021
GRAD 2: 0
$~/mxnet/issue2/build2$ ./demo2 1 1
[19:50:11] ../src/nnvm/legacy_json_util.cc:209: Loading symbol saved by
previous version v0.8.0. Attempting to upgrade...
[19:50:11] ../src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[19:50:11] ../src/engine/engine.cc:55: MXNet start using engine: NaiveEngine
[19:50:11] ../src/storage/storage.cc:199: Using Pooled (Naive)
StorageManager for CPU
INPUT 1: 0.4
INPUT 2: 0.5
OUTPUT: 0.9
OUTGRAD: 2021
GRAD 1: 2021
GRAD 2: 2021
```
**Further investigation:**
This result shows CloneGradient is the most possible reason that causes
this issue. So the remaining check is
1. In bug report , the sym is built by:
```
{
"nodes": [
{
\"op\":\"null\",
\"name\":\".Inputs.Input1\",
\"inputs\":[]
},
{
\"op\":\"null\",
\"name\":\".Inputs.Input2\",
\"inputs\":[]
},
{
\"op\":\"elemwise_add\",
\"name\":\".$0\",
\"inputs\":[[0,0,0],[1,0,0]]
},
{
\"op\":\"_copy\",
\"name\":\".Outputs.Output\",
\"inputs\":[[2,0,0]]
}
],
\"arg_nodes\":[0,1],
\"heads\":[[3,0,0]]
}
```
It is possible that "_copy" operation intrigues this issue in
CloneGradient. This helps understand why it fails.
2. Other usages of CloneGradient in mxnet\src\operator\contrib\stes_op.cc
```
CloneGradient{"_backward_round_ste"}
CloneGradient{"_backward_sign_ste"}
```
It is possible they have the similar bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
