JohnLee168 commented on a change in pull request #12456: [MXNET-910]
Multithreading inference.
URL: https://github.com/apache/incubator-mxnet/pull/12456#discussion_r261053384
##########
File path: src/c_api/c_predict_api.cc
##########
@@ -232,24 +223,117 @@ int MXPredCreatePartialOut(const char* symbol_json_str,
}
aux_arrays.push_back(nd);
}
- ret->arg_arrays = arg_arrays;
- ret->aux_arrays = aux_arrays;
// bind
- {
- std::map<std::string, Context> ctx_map;
- std::vector<NDArray> grad_store(arg_arrays.size());
- std::vector<OpReqType> grad_req(arg_arrays.size(), kNullOp);
-
-
- ret->exec.reset(Executor::Bind(sym, ctx, ctx_map,
- arg_arrays,
- grad_store, grad_req,
- aux_arrays));
+ for (int i = 0; i < num_threads; i++) {
+ std::unique_ptr<MXAPIPredictor> ret(new MXAPIPredictor());
+ ret->sym = sym;
+ ret->ctx = ctx;
+ ret->key2arg = key2arg;
+ ret->arg_arrays = arg_arrays;
+ ret->aux_arrays = aux_arrays;
ret->out_shapes = out_shapes;
- ret->out_arrays = ret->exec->outputs();
+
+ if (!lazy) {
Review comment:
> The fundamental problem here is that if we create multiple executors in
the same thread (e.g., in the main thread), these executors will share the same
temporary resources, which leads to race condition when these executors are
used in different threads. To fix this problem, here we avoid creating
executors when we create predictors in the main thread. The executors are
actually created when the predictor is used in the worker thread for the first
time. As long as the executor is always used in this worker thread, there won't
be race condition.
If I use 10 different PredictorHandles created by
MXPredCreate() in the main thread and for each PredictorHandle calling
MXPredSetInput(), MXPredForward() and MXPredGetOutput() functions to inference
in 10 threads, Is it safe?
Thread can get one current available PredictorHandle to inference,
therefore, one certain thread may get different PredictorHandle to inference.
Is this safe?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
