JohnLee168 commented on a change in pull request #12456: [MXNET-910] Multithreading inference. URL: https://github.com/apache/incubator-mxnet/pull/12456#discussion_r261053384
########## File path: src/c_api/c_predict_api.cc ########## @@ -232,24 +223,117 @@ int MXPredCreatePartialOut(const char* symbol_json_str, } aux_arrays.push_back(nd); } - ret->arg_arrays = arg_arrays; - ret->aux_arrays = aux_arrays; // bind - { - std::map<std::string, Context> ctx_map; - std::vector<NDArray> grad_store(arg_arrays.size()); - std::vector<OpReqType> grad_req(arg_arrays.size(), kNullOp); - - - ret->exec.reset(Executor::Bind(sym, ctx, ctx_map, - arg_arrays, - grad_store, grad_req, - aux_arrays)); + for (int i = 0; i < num_threads; i++) { + std::unique_ptr<MXAPIPredictor> ret(new MXAPIPredictor()); + ret->sym = sym; + ret->ctx = ctx; + ret->key2arg = key2arg; + ret->arg_arrays = arg_arrays; + ret->aux_arrays = aux_arrays; ret->out_shapes = out_shapes; - ret->out_arrays = ret->exec->outputs(); + + if (!lazy) { Review comment: > The fundamental problem here is that if we create multiple executors in the same thread (e.g., in the main thread), these executors will share the same temporary resources, which leads to race condition when these executors are used in different threads. To fix this problem, here we avoid creating executors when we create predictors in the main thread. The executors are actually created when the predictor is used in the worker thread for the first time. As long as the executor is always used in this worker thread, there won't be race condition. If I use 10 different PredictorHandles created by MXPredCreate() in the main thread and for each PredictorHandle calling MXPredSetInput(), MXPredForward() and MXPredGetOutput() functions to inference in 10 threads, Is it safe? Thread can get one current available PredictorHandle to inference, therefore, one certain thread may get different PredictorHandle to inference. Is this safe? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services