access2rohit opened a new issue #20088: URL: https://github.com/apache/incubator-mxnet/issues/20088
## Description Current CD pipeline installs unnecessary libs during test stage which causes incorrect testing of wheels. Any statically linked and packaged dependency should not be installed while testing the wheels. Currently this script(https://github.com/apache/incubator-mxnet/blob/v1.8.x/ci/docker/install/ubuntu_core.sh) executes inside container used for testing the wheels which already installs openblas, lapack, opencv etc. when most of these dependencies are either statically linked to the binary or packaged into the wheel file: ## Recent Issue Libopenblas wasn't packaged correctly and CD test phase didn't catch it #20068 . Also dnnl_version.h and dnnl_config.h were not being packaged but none of the tests failed. in the link: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fmxnet-stable-release-job/detail/mxnet-stable-release-job/86/pipeline/737/ check section `set +eecho "Unpacked lib/libmxnet.so, licenses/*, lib/libgfortran.so.3, 3rdparty/mkldnn/build/install/include/dnnl_version.h, 3rdparty/mkldnn/build/install/include/dnnl_config.h from mxnet_cpu"for i in $(echo lib/libmxnet.so, licenses/*, lib/libgfortran.so.3, 3rdparty/mkldnn/build/install/include/dnnl_version.h, 3rdparty/mkldnn/build/install/include/dnnl_config.h | sed -e 's/,/ /g'); do md5sum $i; donereturn 0` ``` — Shell Script<1s [2021-03-22T22:00:14.052Z] + set +e [2021-03-22T22:00:14.052Z] + echo Unpacked lib/libmxnet.so, licenses/*, lib/libgfortran.so.3, 3rdparty/mkldnn/build/install/include/dnnl_version.h, 3rdparty/mkldnn/build/install/include/dnnl_config.h from mxnet_cpu [2021-03-22T22:00:14.052Z] Unpacked lib/libmxnet.so, licenses/*, lib/libgfortran.so.3, 3rdparty/mkldnn/build/install/include/dnnl_version.h, 3rdparty/mkldnn/build/install/include/dnnl_config.h from mxnet_cpu [2021-03-22T22:00:14.052Z] + echo lib/libmxnet.so, licenses/*, lib/libgfortran.so.3, 3rdparty/mkldnn/build/install/include/dnnl_version.h, 3rdparty/mkldnn/build/install/include/dnnl_config.h [2021-03-22T22:00:14.052Z] + sed -e s/,/ /g [2021-03-22T22:00:14.052Z] + md5sum lib/libmxnet.so [2021-03-22T22:00:14.052Z] 607cec1cefcba577e8013089c09bce02 lib/libmxnet.so [2021-03-22T22:00:14.052Z] + md5sum licenses/BOOST1_0 [2021-03-22T22:00:14.052Z] e4224ccaecb14d942c71d31bef20d78c licenses/BOOST1_0 [2021-03-22T22:00:14.052Z] + md5sum licenses/BSD2 [2021-03-22T22:00:14.052Z] 404484cd993e63faa93414e8ef81ca2e licenses/BSD2 [2021-03-22T22:00:14.052Z] + md5sum licenses/BSD3-cmake [2021-03-22T22:00:14.052Z] 79ee324fc8726f182bf4adae9b5dfeb7 licenses/BSD3-cmake [2021-03-22T22:00:14.052Z] + md5sum licenses/DISCLAIMER-WIP [2021-03-22T22:00:14.052Z] 31129fcab8ae551f0c3a6e8060df0e31 licenses/DISCLAIMER-WIP [2021-03-22T22:00:14.052Z] + md5sum licenses/LICENSE [2021-03-22T22:00:14.052Z] b740dd498bd21432989832ecc9d800bf licenses/LICENSE [2021-03-22T22:00:14.052Z] + md5sum licenses/LICENSE.binary.dependencies [2021-03-22T22:00:14.052Z] 18be783f393064af841be6704abfa562 licenses/LICENSE.binary.dependencies [2021-03-22T22:00:14.052Z] + md5sum licenses/MIT [2021-03-22T22:00:14.052Z] 228348db49a2c1d1f4acf2afaea3c2a0 licenses/MIT [2021-03-22T22:00:14.052Z] + md5sum licenses/NOTICE [2021-03-22T22:00:14.052Z] 7f78e09d1107692cb9c71c72ed6e173b licenses/NOTICE [2021-03-22T22:00:14.052Z] + md5sum licenses/OFL1_1 [2021-03-22T22:00:14.052Z] 6ed93967ff0dc6dd9c8d31c17f817a06 licenses/OFL1_1 [2021-03-22T22:00:14.052Z] + md5sum lib/libgfortran.so.3 [2021-03-22T22:00:14.052Z] 66de19a2dd61695f2439c8d53abec1e2 lib/libgfortran.so.3 [2021-03-22T22:00:14.052Z] + md5sum 3rdparty/mkldnn/build/install/include/dnnl_version.h -------> [2021-03-22T22:00:14.052Z] md5sum: 3rdparty/mkldnn/build/install/include/dnnl_version.h: No such file or directory [2021-03-22T22:00:14.052Z] + md5sum 3rdparty/mkldnn/build/install/include/dnnl_config.h -------> [2021-03-22T22:00:14.052Z] md5sum: 3rdparty/mkldnn/build/install/include/dnnl_config.h: No such file or directory [2021-03-22T22:00:14.052Z] + return 0 ``` ## What have you tried to solve it? 1. PR to fix the issue with MXNet-1.8.0 wheels #20086 ## Solution Remove unnecessary libraries from the CD test container. Adding new shell script to run is a solution. But using CI/CD like master branch and using build_spec.yaml to install only the necessary libraries is a much cleaner and maintainable solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
