FrozenGene opened a new pull request #4657: [CodeGen] Generate blob use LLVM directly URL: https://github.com/apache/incubator-tvm/pull/4657 This is one prior work of[ Module based Model Runtime Interface RFC](https://discuss.tvm.ai/t/discuss-module-based-model-runtime-interface/5025). In this RFC, we want to serialize params (weight) into shared library directly. However, previous way of serializing the blob is to use `PackImportToC`, i.e. we will serialize the blob into char array `__tvm_dev_blob`, which stored the serialized result (0x...0x...), then we write this into `dev.cc` and pass to the compiler compiling. If we don't serialize params, it is ok. However, when we serialize params, the `__tvm_dev_blob` and `dev.cc` will become very large (resnet18 workload of `dev.cc` will be 380M, without serializing weight is 2.6M). The compiling time will increase very much (**Even 10x slower**). According to the investigation, the bottleneck is the parsing time (see [GCC time report](https://discuss.tvm.ai/t/discuss-module-based-model-runtime-interface/5025/5?u=frozengene)). So we decide to avoid the C++ compiler (i.e. avoid the parsing time) and generate LLVM module directly. After testing, even we we serialize the params into shared library, the compiling time of generating blob is very fast. The compiling time of resnet18 on one cloud machine I tested before could **boost from 128.46s to 13.28s**. see[ previous test](https://discuss.tvm.ai/t/discuss-module-based-model-runtime-interface/5025/17?u=frozengene) After this work, I find it also have some a little improvement of compiling time compared previous way testing on my local machine (Intel i7-7700 CPU @ 3.60GHz). Workload is restnet18 on CUDA. Run 10 times and get the average | PackImportToC | PackImportToLLVM | | ---------------------| ----------------| | 8.586s | 8.082s | @tqchen @zhiics @yzhliu @icemelon9
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
