I have noticed that our build jobs have been taking about 2+ hr to build and test for every PR and merge. I have observed that the windows build and unit tests take about 30 minutes to build and 50 minutes to run the unit tests. There is also CUDA7.5 + cuDNN and MKL builds that take 30+ minutes each respectively to just build. Though they run in parallel there is not always enough capacity to handle these.
Can someone who has expertise in these areas to help explain what part of these builds are taking so long and help reduce the build times. One area I see is running the Python Windows GPU Unit tests that can be parallelized for Python2 and Python3 on the same machine by using Conda Environments(i think the slaves already have Conda Environments), We are also considering to modify our PR builds to not build immediately after a PR is submitted and instead run only sanity tests and wait for a reviewer to tag it as "ready to build". This could alleviate some load off and only run when the PR is really ready. You can view the various builds time by clicking through the stages at the top of this sample build. https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/v0.11.0/5/pipeline/49 yajiedesign@, can you please take a look to see if we can improve Windows build/test times? Thanks, Naveen
