Before any binding tests are moved to nightly, I think we need to figure out how the community can get proper notifications of failure and success on those nightly runs. Otherwise, I think that breakages would go unnoticed.
-Carin On Tue, Aug 13, 2019 at 7:47 PM Pedro Larroy <pedro.larroy.li...@gmail.com> wrote: > Hi > > Seems we are hitting some problems in CI. I propose the following action > items to remedy the situation and accelerate turn around times in CI, > reduce cost, complexity and probability of failure blocking PRs and > frustrating developers: > > * Upgrade Windows visual studio from VS 2015 to VS 2017. The > build_windows.py infrastructure should easily work with the new version. > Currently some PRs are blocked by this: > https://github.com/apache/incubator-mxnet/issues/13958 > * Move Gluon Model zoo tests to nightly. Tracked at > https://github.com/apache/incubator-mxnet/issues/15295 > * Move non-python bindings tests to nightly. If a commit is touching other > bindings, the reviewer should ask for a full run which can be done locally, > use the label bot to trigger a full CI build, or defer to nightly. > * Provide a couple of basic sanity performance tests on small models that > are run on CI and can be echoed by the label bot as a comment for PRs. > * Address unit tests that take more than 10-20s, streamline them or move > them to nightly if it can't be done. > * Open sourcing the remaining CI infrastructure scripts so the community > can contribute. > > I think our goal should be turnaround under 30min. > > I would also like to touch base with the community that some PRs are not > being followed up by committers asking for changes. For example this PR is > importtant and is hanging for a long time. > > https://github.com/apache/incubator-mxnet/pull/15051 > > This is another, less important but more trivial to review: > > https://github.com/apache/incubator-mxnet/pull/14940 > > I think comitters requesting changes and not folllowing up in reasonable > time is not healthy for the project. I suggest configuring github > Notifications for a good SNR and following up. > > Regards. > > Pedro. >
