Re: [Discussion] MXNet CMake build - raise minimal required version

2018-11-22 Thread Chris Olivier
yes that flag can be removed. profiler should always be. built in

On Thu, Nov 22, 2018 at 7:44 AM Anton Chernov  wrote:

> You can find relevant information regarding the profiling flag here:
>
> https://github.com/apache/incubator-mxnet/issues/11563
>
>
> чт, 22 нояб. 2018 г. в 16:06, Chris Olivier :
>
> > what is meant by:
> >
> >
> > *Profiling*
> > The profiler is always on even for production release builds, because
> MXNet
> > can not be build without it [2].  ?
> >
> > you mean it is always built or it is turned on (ie recording and saving
> > profiling information)?  I am not aware of it being turned on by default.
> >
> >
> > profiler has no overhead when built in but not turned on.
> >
> >
> > On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov 
> wrote:
> >
> > > Dear MXNet community,
> > >
> > > I propose to raise the minimal required cmake version that is needed to
> > > build MXNet to 3.10 which was tagged on March 16 2018 [1].
> > >
> > > The effort of repairing cmake scripts in general is targeting to
> > deprecate
> > > make and maintain only 1 build system.
> > >
> > > *Need*
> > >
> > > The build system is the foundation of every software project. It's
> > quality
> > > is directly impacting the quality of the project. The MXNet build
> system
> > is
> > > fragile, partially broken and not maintained.
> > >
> > > Users of MXNet and developers are confused by the fact that 2 build
> > systems
> > > exist at the same time: make and CMake.
> > >
> > > The main functional areas which are impacted by the current state of
> the
> > > cmake files are:
> > >
> > > *OpenMP*
> > > The current CMake files mix OpenMP libraries from different compliers
> > which
> > > is undefined behaviour. It leads to indeterministic crashes on some
> > > platforms. Build and deployment are very hard. No evidence exists that
> > > proves that there is any benefit of having llvm OpenMP library as a
> > > submodule in MXNet.
> > >
> > > *BLAS and LAPACK*
> > > Basic math library usage is mixed up. It is hard and confusing to
> > configure
> > > and the choosing logic of the most optimal library is not present. MKL
> > and
> > > OpenBLAS are intermixed in an unpredictable manner.
> > >
> > > *Profiling*
> > > The profiler is always on even for production release builds, because
> > MXNet
> > > can not be build without it [2].
> > >
> > > *CUDA*
> > > CUDA is detected by 3 different files in the current cmake scripts and
> > the
> > > choice of those is based on a obscure logic with involves different
> > > versions of cmake and platforms which it's building on
> > >
> > > * CMakeLists.txt
> > > * cmake/FirstClassLangCuda.cmake
> > > * 3rdparty/mshadow/cmake/Cuda.cmake
> > >
> > >
> > > *Confusing and misleading cmake user options*
> > > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not
> do
> > > what they supposed to based on cmake generator version and version of
> > cmake
> > > [3].
> > > There are currently more than 30 build parameters for MXNet none of
> them
> > > documented. Some of them not even located in the main CMakeLists.txt
> > file,
> > > for example 'BLAS'.
> > >
> > >
> > > *Issues*
> > > There is a significant amount of github issues related to cmake or
> build
> > in
> > > general. New tickets are issued frequently.
> > >
> > > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702)
> > >  [DISCUSSION] Should we deprecate Makefile and only use CMake?
> > > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079)
> >  troubles
> > > building python interface on raspberry pi 3
> > > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722)
> >  problem:
> > > compile mxnet with hdfs
> > > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip
> > > package can be much faster (OpenCV version?)
> > > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417)
> > > libomp.so
> > > dependency (need REAL fix)
> > > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532)
> > >  mxnet-mkl
> > > (v0.12.0) crash when using (conda-installed) numpy with MKL //
> > (indirectly)
> > > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131)
> > > mxnet-cu92 low efficiency  // (indirectly)
> > > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA
> > > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix
> makefile
> > or
> > > Ninja generator
> > > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo
> > in
> > > cpp-package/CMakeLists.txt
> > > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737)
> Cmake
> > is
> > > running again when execute make install
> > > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543)
> Failed
> > > to
> > > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083:
> unabel
> > > to open file: “mxnet-cpp/op.h”: No such file or directory
> > > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217)
> > Building
> > > w

Re: [Discussion] MXNet CMake build - raise minimal required version

2018-11-22 Thread Pedro Larroy
Thanks Anton for putting this together and your efforts here. I think
it's crucial that we maintain and bring the CMake system forward. I
have spent a lot of time dealing with CMake issues on different
platforms, we really increase developer productivity and platform
support by having a streamlined build system.
On Thu, Nov 22, 2018 at 4:06 PM Chris Olivier  wrote:
>
> what is meant by:
>
>
> *Profiling*
> The profiler is always on even for production release builds, because MXNet
> can not be build without it [2].  ?
>
> you mean it is always built or it is turned on (ie recording and saving
> profiling information)?  I am not aware of it being turned on by default.
>
>
> profiler has no overhead when built in but not turned on.
>
>
> On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov  wrote:
>
> > Dear MXNet community,
> >
> > I propose to raise the minimal required cmake version that is needed to
> > build MXNet to 3.10 which was tagged on March 16 2018 [1].
> >
> > The effort of repairing cmake scripts in general is targeting to deprecate
> > make and maintain only 1 build system.
> >
> > *Need*
> >
> > The build system is the foundation of every software project. It's quality
> > is directly impacting the quality of the project. The MXNet build system is
> > fragile, partially broken and not maintained.
> >
> > Users of MXNet and developers are confused by the fact that 2 build systems
> > exist at the same time: make and CMake.
> >
> > The main functional areas which are impacted by the current state of the
> > cmake files are:
> >
> > *OpenMP*
> > The current CMake files mix OpenMP libraries from different compliers which
> > is undefined behaviour. It leads to indeterministic crashes on some
> > platforms. Build and deployment are very hard. No evidence exists that
> > proves that there is any benefit of having llvm OpenMP library as a
> > submodule in MXNet.
> >
> > *BLAS and LAPACK*
> > Basic math library usage is mixed up. It is hard and confusing to configure
> > and the choosing logic of the most optimal library is not present. MKL and
> > OpenBLAS are intermixed in an unpredictable manner.
> >
> > *Profiling*
> > The profiler is always on even for production release builds, because MXNet
> > can not be build without it [2].
> >
> > *CUDA*
> > CUDA is detected by 3 different files in the current cmake scripts and the
> > choice of those is based on a obscure logic with involves different
> > versions of cmake and platforms which it's building on
> >
> > * CMakeLists.txt
> > * cmake/FirstClassLangCuda.cmake
> > * 3rdparty/mshadow/cmake/Cuda.cmake
> >
> >
> > *Confusing and misleading cmake user options*
> > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do
> > what they supposed to based on cmake generator version and version of cmake
> > [3].
> > There are currently more than 30 build parameters for MXNet none of them
> > documented. Some of them not even located in the main CMakeLists.txt file,
> > for example 'BLAS'.
> >
> >
> > *Issues*
> > There is a significant amount of github issues related to cmake or build in
> > general. New tickets are issued frequently.
> >
> > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702)
> >  [DISCUSSION] Should we deprecate Makefile and only use CMake?
> > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079)   troubles
> > building python interface on raspberry pi 3
> > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722)   problem:
> > compile mxnet with hdfs
> > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip
> > package can be much faster (OpenCV version?)
> > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417)
> > libomp.so
> > dependency (need REAL fix)
> > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532)
> >  mxnet-mkl
> > (v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly)
> > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131)
> > mxnet-cu92 low efficiency  // (indirectly)
> > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA
> > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or
> > Ninja generator
> > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in
> > cpp-package/CMakeLists.txt
> > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is
> > running again when execute make install
> > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed
> > to
> > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel
> > to open file: “mxnet-cpp/op.h”: No such file or directory
> > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building
> > with OpenCV causes link errors
> > * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet
> > MKLDNN build dependency/flow discussion
> > * #10009 (https://github.com/apache/incubator-mxnet/issues/10009)
> > [CMAKE][IoT] Rem

Re: [Discussion] MXNet CMake build - raise minimal required version

2018-11-22 Thread Anton Chernov
You can find relevant information regarding the profiling flag here:

https://github.com/apache/incubator-mxnet/issues/11563


чт, 22 нояб. 2018 г. в 16:06, Chris Olivier :

> what is meant by:
>
>
> *Profiling*
> The profiler is always on even for production release builds, because MXNet
> can not be build without it [2].  ?
>
> you mean it is always built or it is turned on (ie recording and saving
> profiling information)?  I am not aware of it being turned on by default.
>
>
> profiler has no overhead when built in but not turned on.
>
>
> On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov  wrote:
>
> > Dear MXNet community,
> >
> > I propose to raise the minimal required cmake version that is needed to
> > build MXNet to 3.10 which was tagged on March 16 2018 [1].
> >
> > The effort of repairing cmake scripts in general is targeting to
> deprecate
> > make and maintain only 1 build system.
> >
> > *Need*
> >
> > The build system is the foundation of every software project. It's
> quality
> > is directly impacting the quality of the project. The MXNet build system
> is
> > fragile, partially broken and not maintained.
> >
> > Users of MXNet and developers are confused by the fact that 2 build
> systems
> > exist at the same time: make and CMake.
> >
> > The main functional areas which are impacted by the current state of the
> > cmake files are:
> >
> > *OpenMP*
> > The current CMake files mix OpenMP libraries from different compliers
> which
> > is undefined behaviour. It leads to indeterministic crashes on some
> > platforms. Build and deployment are very hard. No evidence exists that
> > proves that there is any benefit of having llvm OpenMP library as a
> > submodule in MXNet.
> >
> > *BLAS and LAPACK*
> > Basic math library usage is mixed up. It is hard and confusing to
> configure
> > and the choosing logic of the most optimal library is not present. MKL
> and
> > OpenBLAS are intermixed in an unpredictable manner.
> >
> > *Profiling*
> > The profiler is always on even for production release builds, because
> MXNet
> > can not be build without it [2].
> >
> > *CUDA*
> > CUDA is detected by 3 different files in the current cmake scripts and
> the
> > choice of those is based on a obscure logic with involves different
> > versions of cmake and platforms which it's building on
> >
> > * CMakeLists.txt
> > * cmake/FirstClassLangCuda.cmake
> > * 3rdparty/mshadow/cmake/Cuda.cmake
> >
> >
> > *Confusing and misleading cmake user options*
> > For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do
> > what they supposed to based on cmake generator version and version of
> cmake
> > [3].
> > There are currently more than 30 build parameters for MXNet none of them
> > documented. Some of them not even located in the main CMakeLists.txt
> file,
> > for example 'BLAS'.
> >
> >
> > *Issues*
> > There is a significant amount of github issues related to cmake or build
> in
> > general. New tickets are issued frequently.
> >
> > * #8702 (https://github.com/apache/incubator-mxnet/issues/8702)
> >  [DISCUSSION] Should we deprecate Makefile and only use CMake?
> > * #5079 (https://github.com/apache/incubator-mxnet/issues/5079)
>  troubles
> > building python interface on raspberry pi 3
> > * #1722 (https://github.com/apache/incubator-mxnet/issues/1722)
>  problem:
> > compile mxnet with hdfs
> > * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip
> > package can be much faster (OpenCV version?)
> > * #11417 (https://github.com/apache/incubator-mxnet/issues/11417)
> > libomp.so
> > dependency (need REAL fix)
> > * #8532 (https://github.com/apache/incubator-mxnet/issues/8532)
> >  mxnet-mkl
> > (v0.12.0) crash when using (conda-installed) numpy with MKL //
> (indirectly)
> > * #11131 (https://github.com/apache/incubator-mxnet/issues/11131)
> > mxnet-cu92 low efficiency  // (indirectly)
> > * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA
> > 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile
> or
> > Ninja generator
> > * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo
> in
> > cpp-package/CMakeLists.txt
> > * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake
> is
> > running again when execute make install
> > * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed
> > to
> > build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel
> > to open file: “mxnet-cpp/op.h”: No such file or directory
> > * #10217 (https://github.com/apache/incubator-mxnet/issues/10217)
> Building
> > with OpenCV causes link errors
> > * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet
> > MKLDNN build dependency/flow discussion
> > * #10009 (https://github.com/apache/incubator-mxnet/issues/10009)
> > [CMAKE][IoT] Remove pthread from android_arm64 build
> > * #9944 (https://github.com/apache/incubator-mxnet/issues/9944)   MXNet
> > MinGW-w64 build error // (indirectly)
> > *

Re: [Discussion] MXNet CMake build - raise minimal required version

2018-11-22 Thread Chris Olivier
what is meant by:


*Profiling*
The profiler is always on even for production release builds, because MXNet
can not be build without it [2].  ?

you mean it is always built or it is turned on (ie recording and saving
profiling information)?  I am not aware of it being turned on by default.


profiler has no overhead when built in but not turned on.


On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov  wrote:

> Dear MXNet community,
>
> I propose to raise the minimal required cmake version that is needed to
> build MXNet to 3.10 which was tagged on March 16 2018 [1].
>
> The effort of repairing cmake scripts in general is targeting to deprecate
> make and maintain only 1 build system.
>
> *Need*
>
> The build system is the foundation of every software project. It's quality
> is directly impacting the quality of the project. The MXNet build system is
> fragile, partially broken and not maintained.
>
> Users of MXNet and developers are confused by the fact that 2 build systems
> exist at the same time: make and CMake.
>
> The main functional areas which are impacted by the current state of the
> cmake files are:
>
> *OpenMP*
> The current CMake files mix OpenMP libraries from different compliers which
> is undefined behaviour. It leads to indeterministic crashes on some
> platforms. Build and deployment are very hard. No evidence exists that
> proves that there is any benefit of having llvm OpenMP library as a
> submodule in MXNet.
>
> *BLAS and LAPACK*
> Basic math library usage is mixed up. It is hard and confusing to configure
> and the choosing logic of the most optimal library is not present. MKL and
> OpenBLAS are intermixed in an unpredictable manner.
>
> *Profiling*
> The profiler is always on even for production release builds, because MXNet
> can not be build without it [2].
>
> *CUDA*
> CUDA is detected by 3 different files in the current cmake scripts and the
> choice of those is based on a obscure logic with involves different
> versions of cmake and platforms which it's building on
>
> * CMakeLists.txt
> * cmake/FirstClassLangCuda.cmake
> * 3rdparty/mshadow/cmake/Cuda.cmake
>
>
> *Confusing and misleading cmake user options*
> For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do
> what they supposed to based on cmake generator version and version of cmake
> [3].
> There are currently more than 30 build parameters for MXNet none of them
> documented. Some of them not even located in the main CMakeLists.txt file,
> for example 'BLAS'.
>
>
> *Issues*
> There is a significant amount of github issues related to cmake or build in
> general. New tickets are issued frequently.
>
> * #8702 (https://github.com/apache/incubator-mxnet/issues/8702)
>  [DISCUSSION] Should we deprecate Makefile and only use CMake?
> * #5079 (https://github.com/apache/incubator-mxnet/issues/5079)   troubles
> building python interface on raspberry pi 3
> * #1722 (https://github.com/apache/incubator-mxnet/issues/1722)   problem:
> compile mxnet with hdfs
> * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip
> package can be much faster (OpenCV version?)
> * #11417 (https://github.com/apache/incubator-mxnet/issues/11417)
> libomp.so
> dependency (need REAL fix)
> * #8532 (https://github.com/apache/incubator-mxnet/issues/8532)
>  mxnet-mkl
> (v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly)
> * #11131 (https://github.com/apache/incubator-mxnet/issues/11131)
> mxnet-cu92 low efficiency  // (indirectly)
> * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA
> 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or
> Ninja generator
> * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in
> cpp-package/CMakeLists.txt
> * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is
> running again when execute make install
> * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed
> to
> build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel
> to open file: “mxnet-cpp/op.h”: No such file or directory
> * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building
> with OpenCV causes link errors
> * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet
> MKLDNN build dependency/flow discussion
> * #10009 (https://github.com/apache/incubator-mxnet/issues/10009)
> [CMAKE][IoT] Remove pthread from android_arm64 build
> * #9944 (https://github.com/apache/incubator-mxnet/issues/9944)   MXNet
> MinGW-w64 build error // (indirectly)
> * #9868 (https://github.com/apache/incubator-mxnet/issues/9868)   MKL and
> CMake
> * #9516 (https://github.com/apache/incubator-mxnet/issues/9516)   cmake
> cuda arch issues
> * #9105 (https://github.com/apache/incubator-mxnet/issues/9105)
>  libmxnet.so load path error
> * #9096 (https://github.com/apache/incubator-mxnet/issues/9096)   MXNet
> built with GPerftools crashes
> * #8786 (https://github.com/apache/incubator-

Re: [Discussion] MXNet CMake build - raise minimal required version

2018-11-22 Thread Chris Olivier
i have not seen any proof that any crashes are due to llvm openmp usage.

On Thu, Nov 22, 2018 at 2:35 AM Anton Chernov  wrote:

> Dear MXNet community,
>
> I propose to raise the minimal required cmake version that is needed to
> build MXNet to 3.10 which was tagged on March 16 2018 [1].
>
> The effort of repairing cmake scripts in general is targeting to deprecate
> make and maintain only 1 build system.
>
> *Need*
>
> The build system is the foundation of every software project. It's quality
> is directly impacting the quality of the project. The MXNet build system is
> fragile, partially broken and not maintained.
>
> Users of MXNet and developers are confused by the fact that 2 build systems
> exist at the same time: make and CMake.
>
> The main functional areas which are impacted by the current state of the
> cmake files are:
>
> *OpenMP*
> The current CMake files mix OpenMP libraries from different compliers which
> is undefined behaviour. It leads to indeterministic crashes on some
> platforms. Build and deployment are very hard. No evidence exists that
> proves that there is any benefit of having llvm OpenMP library as a
> submodule in MXNet.
>
> *BLAS and LAPACK*
> Basic math library usage is mixed up. It is hard and confusing to configure
> and the choosing logic of the most optimal library is not present. MKL and
> OpenBLAS are intermixed in an unpredictable manner.
>
> *Profiling*
> The profiler is always on even for production release builds, because MXNet
> can not be build without it [2].
>
> *CUDA*
> CUDA is detected by 3 different files in the current cmake scripts and the
> choice of those is based on a obscure logic with involves different
> versions of cmake and platforms which it's building on
>
> * CMakeLists.txt
> * cmake/FirstClassLangCuda.cmake
> * 3rdparty/mshadow/cmake/Cuda.cmake
>
>
> *Confusing and misleading cmake user options*
> For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do
> what they supposed to based on cmake generator version and version of cmake
> [3].
> There are currently more than 30 build parameters for MXNet none of them
> documented. Some of them not even located in the main CMakeLists.txt file,
> for example 'BLAS'.
>
>
> *Issues*
> There is a significant amount of github issues related to cmake or build in
> general. New tickets are issued frequently.
>
> * #8702 (https://github.com/apache/incubator-mxnet/issues/8702)
>  [DISCUSSION] Should we deprecate Makefile and only use CMake?
> * #5079 (https://github.com/apache/incubator-mxnet/issues/5079)   troubles
> building python interface on raspberry pi 3
> * #1722 (https://github.com/apache/incubator-mxnet/issues/1722)   problem:
> compile mxnet with hdfs
> * #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip
> package can be much faster (OpenCV version?)
> * #11417 (https://github.com/apache/incubator-mxnet/issues/11417)
> libomp.so
> dependency (need REAL fix)
> * #8532 (https://github.com/apache/incubator-mxnet/issues/8532)
>  mxnet-mkl
> (v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly)
> * #11131 (https://github.com/apache/incubator-mxnet/issues/11131)
> mxnet-cu92 low efficiency  // (indirectly)
> * #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA
> 9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or
> Ninja generator
> * #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in
> cpp-package/CMakeLists.txt
> * #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is
> running again when execute make install
> * #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed
> to
> build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel
> to open file: “mxnet-cpp/op.h”: No such file or directory
> * #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building
> with OpenCV causes link errors
> * #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet
> MKLDNN build dependency/flow discussion
> * #10009 (https://github.com/apache/incubator-mxnet/issues/10009)
> [CMAKE][IoT] Remove pthread from android_arm64 build
> * #9944 (https://github.com/apache/incubator-mxnet/issues/9944)   MXNet
> MinGW-w64 build error // (indirectly)
> * #9868 (https://github.com/apache/incubator-mxnet/issues/9868)   MKL and
> CMake
> * #9516 (https://github.com/apache/incubator-mxnet/issues/9516)   cmake
> cuda arch issues
> * #9105 (https://github.com/apache/incubator-mxnet/issues/9105)
>  libmxnet.so load path error
> * #9096 (https://github.com/apache/incubator-mxnet/issues/9096)   MXNet
> built with GPerftools crashes
> * #8786 (https://github.com/apache/incubator-mxnet/issues/8786)   Link
> failure on DEBUG=1 (static member symbol not defined) // (indirectly)
> * #8729 (https://github.com/apache/incubator-mxnet/issues/8729)   Build
> amalgamation using a docker // (indirectly)
> * #8667 (https://github.com/apache/incubator-mxnet/issues/8

[Discussion] MXNet CMake build - raise minimal required version

2018-11-22 Thread Anton Chernov
Dear MXNet community,

I propose to raise the minimal required cmake version that is needed to
build MXNet to 3.10 which was tagged on March 16 2018 [1].

The effort of repairing cmake scripts in general is targeting to deprecate
make and maintain only 1 build system.

*Need*

The build system is the foundation of every software project. It's quality
is directly impacting the quality of the project. The MXNet build system is
fragile, partially broken and not maintained.

Users of MXNet and developers are confused by the fact that 2 build systems
exist at the same time: make and CMake.

The main functional areas which are impacted by the current state of the
cmake files are:

*OpenMP*
The current CMake files mix OpenMP libraries from different compliers which
is undefined behaviour. It leads to indeterministic crashes on some
platforms. Build and deployment are very hard. No evidence exists that
proves that there is any benefit of having llvm OpenMP library as a
submodule in MXNet.

*BLAS and LAPACK*
Basic math library usage is mixed up. It is hard and confusing to configure
and the choosing logic of the most optimal library is not present. MKL and
OpenBLAS are intermixed in an unpredictable manner.

*Profiling*
The profiler is always on even for production release builds, because MXNet
can not be build without it [2].

*CUDA*
CUDA is detected by 3 different files in the current cmake scripts and the
choice of those is based on a obscure logic with involves different
versions of cmake and platforms which it's building on

* CMakeLists.txt
* cmake/FirstClassLangCuda.cmake
* 3rdparty/mshadow/cmake/Cuda.cmake


*Confusing and misleading cmake user options*
For example, USE_CUDA / USE_OLDCMAKECUDA. Some of them will do or not do
what they supposed to based on cmake generator version and version of cmake
[3].
There are currently more than 30 build parameters for MXNet none of them
documented. Some of them not even located in the main CMakeLists.txt file,
for example 'BLAS'.


*Issues*
There is a significant amount of github issues related to cmake or build in
general. New tickets are issued frequently.

* #8702 (https://github.com/apache/incubator-mxnet/issues/8702)
 [DISCUSSION] Should we deprecate Makefile and only use CMake?
* #5079 (https://github.com/apache/incubator-mxnet/issues/5079)   troubles
building python interface on raspberry pi 3
* #1722 (https://github.com/apache/incubator-mxnet/issues/1722)   problem:
compile mxnet with hdfs
* #11549 (https://github.com/apache/incubator-mxnet/issues/11549) Pip
package can be much faster (OpenCV version?)
* #11417 (https://github.com/apache/incubator-mxnet/issues/11417) libomp.so
dependency (need REAL fix)
* #8532 (https://github.com/apache/incubator-mxnet/issues/8532)   mxnet-mkl
(v0.12.0) crash when using (conda-installed) numpy with MKL // (indirectly)
* #11131 (https://github.com/apache/incubator-mxnet/issues/11131)
mxnet-cu92 low efficiency  // (indirectly)
* #10743 (https://github.com/apache/incubator-mxnet/issues/10743) CUDA
9.1.xx failed if not set OLDCMAKECUDA on cmake 3.10.3 with unix makefile or
Ninja generator
* #10742 (https://github.com/apache/incubator-mxnet/issues/10742) typo in
cpp-package/CMakeLists.txt
* #10737 (https://github.com/apache/incubator-mxnet/issues/10737) Cmake is
running again when execute make install
* #10543 (https://github.com/apache/incubator-mxnet/issues/10543) Failed to
build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel
to open file: “mxnet-cpp/op.h”: No such file or directory
* #10217 (https://github.com/apache/incubator-mxnet/issues/10217) Building
with OpenCV causes link errors
* #10175 (https://github.com/apache/incubator-mxnet/issues/10175) MXNet
MKLDNN build dependency/flow discussion
* #10009 (https://github.com/apache/incubator-mxnet/issues/10009)
[CMAKE][IoT] Remove pthread from android_arm64 build
* #9944 (https://github.com/apache/incubator-mxnet/issues/9944)   MXNet
MinGW-w64 build error // (indirectly)
* #9868 (https://github.com/apache/incubator-mxnet/issues/9868)   MKL and
CMake
* #9516 (https://github.com/apache/incubator-mxnet/issues/9516)   cmake
cuda arch issues
* #9105 (https://github.com/apache/incubator-mxnet/issues/9105)
 libmxnet.so load path error
* #9096 (https://github.com/apache/incubator-mxnet/issues/9096)   MXNet
built with GPerftools crashes
* #8786 (https://github.com/apache/incubator-mxnet/issues/8786)   Link
failure on DEBUG=1 (static member symbol not defined) // (indirectly)
* #8729 (https://github.com/apache/incubator-mxnet/issues/8729)   Build
amalgamation using a docker // (indirectly)
* #8667 (https://github.com/apache/incubator-mxnet/issues/8667)
 Compiler/linker error while trying to build from source on Mac OSX Sierra
10.12.6
* #8295 (https://github.com/apache/incubator-mxnet/issues/8295)   Building
with cmake - error
* #7852 (https://github.com/apache/incubator-mxnet/issues/7852)   Trouble
installing MXNet on Raspberry Pi 3
* #13303 (https://github.com/apache