MarisaKirisame commented on issue #1996: [RFC][WIP] Tensor Expression level automatic differentiation URL: https://github.com/apache/incubator-tvm/issues/1996#issuecomment-595890287 I see fundamental problem in this PR. Jacobian(Y, W): tensor compute.jacobian{0x165b360}[0] : float32 [32, 3000, 3000, 10000] axes (i : [0, 31], j : [0, 2999], jac_i0 : [0, 2999], jac_i1 : [0, 9999]) Reduction identity [0.000000f] lhs [x.der] rhs [y.der] combiner [(x.der + y.der)] axes (k : [0, 9999]) condition (uint1)1 source[0] = (X(i, k)*float32(((jac_i0 == j) && (jac_i1 == k)))) This is a really, really big tensor, and the approach this PR take has a "cliff of death" performance chart. This PR then rely on simplification to eliminate all those tensor. If any tensor is not eliminated(which seems to be the case for more complex tensor) the performance will be very bad. Reverse mode automatic differentiation should only calculate vector jacobian product. The jacobian of Y, W, should be dW times jacobian Y W. the Jacobian should simply never be manifested. can this be fixed so the algorithm will not be algorithmically slower without optimization?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
