ZhennanQin commented on issue #15465: [RFC] Integrate TVM into Apache MXNet URL: https://github.com/apache/incubator-mxnet/issues/15465#issuecomment-509582962 @junrushao1994 Let me clarify my concerns for TVM thread pool. 1> On default, threaded engine will only create single worker for normal CPU operator. At this scenario, user will specify `OMP_NUM_THREADS = physical_core_num` to maximize the computing capacity of CPU. A OMP operator followed by TVM operator will have omp thread spinning issue, causing TVM operator executes slowly. This is a major issue need to address because it's a typical scenario in MXNet. 2> when user specify `MXNET_CPU_WORKER_NTHREADS=2`, then 2 CPU workers are activated. It's possible to run 2 independent CPU operators in parallel. For this case, user can set `OMP_NUM_THREADS = physical_core_num / 2 ` to let each worker to use half physical cores. I want to know if TVM can do the same thing. 3> Even if we resolved above problem, there's a thread switching penalty when maintaining 2 different threading pool. The penalty can't be ignored for latency sensitive task(e.g. BS=1 inference). From my perspective, mixed using 2 kinds of thread pool isn't a good idea. It's hard to manage thread binding for multi-workers, and will have thread switching overhead between different thread pools. It's better to use same threading management as MXNet.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
