I am trying to train mnist using tvm, and I hit an issue: There is two function, loss and infer, which one calculate loss and gradient, and one just does infer. However, create_executor/aot/vm all only take one single entry point. If there is multiple entry point, the passes will be called multiple time. Furthurmore, I think the notion of 'compile then execute' is not enough for all the deep learning workload. For example, using our partial evaluator to specialize training/validation/testing data mean we must compile only after we had loaded all the data. Even if we accept that, since PE is slow, an optimization (at the client side) will be to pipeline the PEs: only do them per minibatch at epoch 0, then reuse. Or just has a thread that constantly PE things and fire in NON-PE mode if it did not PE that batch yet. Another example is NAS/AdaNet where the Network structure change all the time.
I think we can make the passes incremental (they are given a set of new things whenever invoked), and have the module keep track of the compiled vs uncompiled. @jroesch @tqchen @zhiics @wweic @vinx13 @junrushao1994 what do you guys think? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/dmlc/tvm/issues/4054
