connorgoggins commented on a change in pull request #17617: [Large Tensor] Fixed Spatial Transformer op URL: https://github.com/apache/incubator-mxnet/pull/17617#discussion_r381202221
########## File path: src/operator/spatial_transformer.cc ########## @@ -58,10 +58,10 @@ inline void BilinearSamplingForward(const Tensor<cpu, 4, DType> &output, const DType top_left_x_w = 1.0 - (x_real - top_left_x); const index_t data_index = n * i_c * i_h * i_w + c * i_h * i_w + top_left_y * i_w + top_left_x; - index_t top_left_v = 0; - index_t top_right_v = 0; - index_t bottom_left_v = 0; - index_t bottom_right_v = 0; + DType top_left_v = 0; Review comment: @ChaiBapchya changing DType to index_t for these specific variables makes the op give incorrect on standard inputs (e.g. the inputs in the CI run) - the values generated in the output NDArray are all integers instead of floats. This is due to the fact that these variables do not represent the index positions (as I also originally believed), but the underlying values at the vertices. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services