Aaron, IMO tutorials have a specific purpose, to introduce concepts and APIs to the users and I think converting examples to tutorials would overwhelm the users, we should carefully choose which examples we want to turn into tutorials.
I agree that today examples are graveyard of untested code, my suggestion is to add some testing to the example when you touch the example - at the least to check the functionality. These can be run once a week. On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham <aaron.s.mark...@gmail.com> wrote: > I've been actively promoting moving examples to tutorials during reviews. > That way they fall under the testing umbrella and get added to the website. > > Many times there's not really a great distinction as to why something is in > the examples folder, other than it's like a graveyard of untested sample > code. > > I would suggest a starting strategy of when doing updates on examples, see > if with just a little more effort, ask yourself, can it be converted to a > tutorial? > > The last thing CI needs is more flaky tutorial tests, so whatever is done > here should use the more robust approaches that are being discussed. > > Cheers, > Aaron > > On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy < > sandeep.krishn...@gmail.com wrote: > > > Thanks, Ankit for bringing this up. @Anirudh - All the concerns you > raised > > are very valid. Here are my thoughts: > > 1. There were several examples that were crashing or had compiler errors. > > This is a very bad user experience. All example scripts should be at > > least runnable! > > 2. While I agree examples are too diverse (python scripts, notebooks, > > epochs, print statements etc..) We can always start small, we can start > > with 5 examples. We can use this to streamline all examples to be python > > scripts, print statements, with the main function invoker that can take > > params like epoch, dataset etc. > > 3. We can start with running weekly tests to avoid too long nightly test > > pipeline. > > 4. One possible issue can be on a few examples that depend on a large or > > controlled dataset. I am not sure yet, how to solve this, but, we can > > think. > > > > Any suggestions? > > Best, > > Sandeep > > > > > > > > On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <anirudhk...@gmail.com> > > wrote: > > > > > Hi Ankit, > > > > > > I have a few concerns about testing examples. Before writing tests for > > > examples, > > > > > > - you will need to first decide what constitutes a test for an > > example, > > > because examples are not API calls, which will have return > statements > > > and > > > the test can just call the API and assert for certain values. Just > > > testing > > > if an example is a compilable python script will not add much value > in > > > my > > > opinion. > > > - And testing for example output and results will require a re-write > > of > > > many of the examples, because many of them currently just have print > > > statements as outputs and does not return any value as such. I am > not > > > sure > > > if it is worth the dev-effort. > > > - the current set of examples in the mxnet repo are very diverse - > > some > > > are written as python notebooks, some are just python scripts with > > paper > > > implementations, and some are just illustrations of certain mxnet > > > features. > > > I am curious to know how you will write tests for these things. > > > > > > > > > Looking forward to seeing the design of this test bed/framework. > > > > > > > > > Thanks > > > Anirudh Acharya > > > > > > On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu > > > <marco.g.ab...@googlemail.com.invalid> wrote: > > > > > > > Hello Ankit, > > > > > > > > that's a great idea! Using the tutorial tests as reference is a great > > > > starting point. If you are interested, please don't hesitate to > attend > > > the > > > > Berlin user group in case you would like to discuss your first > thoughts > > > > in-person before drafting a design. > > > > > > > > -Marco > > > > > > > > > > > > Am Fr., 9. Nov. 2018, 23:23 hat khedia.an...@gmail.com < > > > > khedia.an...@gmail.com> geschrieben: > > > > > > > > > Hi MXNet community, > > > > > > > > > > Recently, I and a few other contributors focussed on fixing > examples > > in > > > > > our repository which were not working out of the box as expected. > > > > > https://github.com/apache/incubator-mxnet/issues/12800 > > > > > https://github.com/apache/incubator-mxnet/issues/11895 > > > > > https://github.com/apache/incubator-mxnet/pull/13196 > > > > > > > > > > Some of the examples failed after API changes and remained uncaught > > > until > > > > > a user reported the issue. While the community is actively working > on > > > > > fixing it, it might re-occur after few days if we don’t have a > proper > > > > > mechanism to catch regressions. > > > > > > > > > > So, I would like to propose to enable nightly/weekly tests for the > > > > > examples similar to what we have for tutorials to catch any such > > > > > regressions. The test could check only basic > functionalities/working > > of > > > > the > > > > > examples. It can run small examples completely whereas it can run > > long > > > > > training examples for only few epochs. > > > > > > > > > > Any thoughts from the community? Any other suggestions for fixing > the > > > > same? > > > > > > > > > > Regards, > > > > > Ankit Khedia > > > > > > > > > > > > > > > > > > -- > > Sandeep Krishnamurthy > > >
