Re: New AMIs for CI

2020-02-21 Thread Lausen, Leonard
Thanks Pedro for you work to improve the CI setup. For Windows, let's see if fixing the deterministic failure of GPU tests with the updated toolchain helps to fix the non-deterministic failures in the old toolchain. Best regards Leonard On Fri, 2020-02-21 at 15:37 -0800, Pedro Larroy wrote: > C

Re: New AMIs for CI

2020-02-21 Thread Pedro Larroy
CI is back to normal. We haven't updated Windows AMIs due to issues with GPU unit tests. You might need to retrigger your PRs. Thanks for your patience. On Wed, Feb 19, 2020 at 5:54 PM Pedro Larroy wrote: > I reverted the CI rollout due to the following issues: > > https://github.com/apache/in

Re: New AMIs for CI

2020-02-19 Thread Pedro Larroy
I reverted the CI rollout due to the following issues: https://github.com/apache/incubator-mxnet/issues/17633 https://github.com/apache/incubator-mxnet/issues/17635 I would need help from the community to fix them as we can't even compile in debug mode in windows as the above, and also due to ol

New AMIs for CI

2020-02-18 Thread Pedro Larroy
Hi Tomorrow I will be updating the CI environment with new AMIs, and deploying updated autoscaling logic with fixes, expect some disruptions in CI runs. The Linux AMIs will be updated to Ubuntu 18.04 with updated GPU drivers, this won't affect Linux container builds. The new Windows AMI comes wi