samskalicky opened a new issue #19655: URL: https://github.com/apache/incubator-mxnet/issues/19655
## Description Theres a race condition between loading a model and the model's params being available by a backend run with `optimize_for` like: ``` block = SymbolBlock.imports(symbol_file, input_names, param_file, mxnet.cpu()) block.optimize_for(inputs, backend='myBackend') ``` Where the data in the ndarrays have garbage data in the backend. ### Fix We need to loop over the NDArrays for args/aux and call `wait_to_read` on each to ensure any previous operations pushed to the engine are complete: https://github.com/apache/incubator-mxnet/blob/master/src/c_api/c_api_symbolic.cc#L1309-L1310 ### Workaround In the meantime we can simply call `mx.nd.waitall()` between loading the model and `optimize_for` ``` block = SymbolBlock.imports(symbol_file, input_names, param_file, mxnet.cpu()) mx.nd.waitall() block.optimize_for(inputs, backend='myBackend') ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org For additional commands, e-mail: issues-h...@mxnet.apache.org
