Assume that we are able to “package” MTT so it can be upstreamed to the OpenHPC folks (as they have requested). It then becomes a little more “salable” as a generalized testing tool, which means that the tests in our repo could be used by others without having to deal with the MTT-specific harness calls.
At that point, it seems to me one could make a case for a general MPI test repo that supported MTT, and OpenHPC could even be a reasonable home for it. I can certainly raise the question there, if you like. > On May 19, 2016, at 8:24 AM, Josh Hursey <jjhur...@open-mpi.org> wrote: > > I think talking to the MPICH folks about creating a common test pool might be > useful. More useful would be to get the MPI Forum to 'bless' it and take > input from all of the MPI venders. Maybe you all can talk about that in a > sidebar at the MPI Forum - depending on what we figure out here. > > The MPICH tests [1] contains a few test suites that are freely available as > tarball downloads on their site (though they seem a bit stale - last updated > 2012). They also have the 'tests' subdirectory in their repo [2]. > > > From my perspective, the benefits of creating a new public test repo on > GitHub would be (in no particular order, numbers provided to aid in reference > during followup discussion): > > (1) As we create new MPI test cases for issues, we file them publicly. > Currently, we drop them in ompi-tests for convenience. As a result, the > public cannot see/use them. Same for new test suites that we might create as > part of a new MPI standard interface proposal (e.g., FT, Sessions, or even > back to Nonblocking collectives). > > (2) GitHub allows us to foster a discussion about the tests in a way that is > easily traceable. More so than using a mailing list and discussing a tarball > drop of a test suite. > > (3) We can associate a MTT snippet of code for Test Build/Run that can be > used to run that test suite. Instead of putting this in with the MTT repo. > This would make these snippets slightly easier to find. > > (4) We can contain references to other test suites, even if we don't copy > them into our public test suite. This would allow us to say "hey, there are > these great MPICH/Netpipe/IMB/... tests at this URL" then give them > instructions on how we run them in MTT (or outside of MTT). If we have any > patches for that test suite we could collect them in the sub-directory until > (if) they get incorporated upstream. We could also associate notes on runtime > parameters that help tune Open MPI for a particular performance test at > various scales. > > (5) Generally, it would make it easier for the community to pick up some > unit tests and run them to do a 'make deep-check' on their MPI install. I > think there is a hesitation to use the test suite from another MPI > implementation with Open MPI as it might not be as consistent as one would > like. Any failures would have to be investigated as a difference of > implementation (do their tests use our CLI correctly?), difference of MPI > Standard interpretation (as often happens as the MPI Forum rolls out errata), > or actual bug in Open MPI. For an end user that is tough to determine. For > MPI implementers it is easier since we live those differences. > > > Let me play a little devils advocate here too on a couple points [not that I > need to really, I'm sure I'll get some help here]. (#) reference the points > above. > > (1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI Forum > to decide, and might be difficult to achieve given the diversity of > stakeholders at that level. However, if there is a public repo for MPI tests > that the MPI Forum can all agree on then that might be a good place to start. > I still think (4) would apply, but maybe we can solve that differently. > > (2) How much do we really debate the correctness of our tests? Early on in > the project, quite often. Lately it still happens (NULL datatype issue, for > example). That discussion can only happen internally since the test cases we > reference are not public unless we expose them (which is toe'ing the line of > redistribution). There is value in broadening the discussion outside of the > Open MPI developers on some of those test cases. At the very least for > educational benefit of the community about some of the more subtle edges of > the MPI standard. > > (3) Shouldn't that sample code live with MTT? Maybe. I can see both sides of > this one. I do think there is some value in separating out the functional > code of MTT from the test suite configurations. > > > -- Josh > > [1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html > <http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html> > [2] http://trac.mpich.org/projects/mpich/browser/test > <http://trac.mpich.org/projects/mpich/browser/test> > > > On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com > <mailto:jsquy...@cisco.com>> wrote: > I was initially for this idea, but now I find myself conflicted. > > Specifically: what's the value in yet another MPI test suite? > > Sure, we've got a bunch of tests that no one else has (i.e., the things we've > home-grown over the years). Some are great tests. Some will need to be > cleaned up and generalized. Some are user-contributed. Some are ompi-specific. > > Should these tests be contributed to some other test suite? > > Specifically: I'm wondering about the MPICH test suite - that seems to be the > only remaining big MPI test suite these days. Is it worth having a discussion > w the MPICH folks to see if a) their test suite is general enough for all MPI > implementations, and B) if they would accept a bunch of random tests from us? > > And if not, I think I'd like to understand better the value add that we can > provide by making another MPI test suite. > > Sent from my phone. No type good. > > > On May 18, 2016, at 11:54 PM, Josh Hursey <jjhur...@open-mpi.org > > <mailto:jjhur...@open-mpi.org>> wrote: > > > > WHAT: Create a public test repo (ompi-tests-public) to collect > > > > WHY: ompi-tests is private, and difficult/impossible to open up. There is a > > demand for a public collection of unit tests. This repo would allow us to > > cultivate such a collection of unit tests. > > > > WHERE: open-mpi GitHub project > > > > TIMEOUT: Teleconf, Tue., May 24, 2016 > > > > MORE DETAIL: > > > > Over the years we have had periodic public requests for access to our > > ompi-tests repo. Opening up ompi-tests to the public is nearly impossible > > since, given the age of some of these tests, determining if we can > > redistribute these tests has become complicated. > > > > Recently we had two different requests on the MTT users and Open MPI devel > > list about access to the ompi-tests repo for testing. This got me thinking > > that we could try to cultivate a public set of tests with a clear lineage > > and license. > > > > > > Below are my current thoughts for structure and maintenance of the repo. > > Certainly up for discussion. > > > > Similar to the existing ompi-tests repo structure. > > - Call the repository "ompi-tests-public" > > - The repo will contain at least one test suite ('misc' - see below). > > - Each test suite can have its own build system > > - Each test suite should contain a README-MTT.md which will contain a > > sample Test Build and Test Run section for use in MTT. > > > > All tests contributed will be covered under the Open MPI license agreement > > unless a specific test suite has a different, but compatible license. To > > contribute a test to the repo a developer must sign the contributor > > agreement. Contributions must go through a PR + Review process (similar to > > how we maintain ompi-release). This is meant to provide an opportunity to > > review the tests for correctness before acceptance into the repo. > > > > We will seed the repo with a 'misc' test suite. This test suite is meant to > > collect all of the miscellaneous tests created by Open MPI developers > > including those regression tests that emerge as part of Issues or MPI Forum > > discussions, for example. It will contain the unit tests currently in the > > Open MPI's examples directory - what we have been calling the 'trivial' > > test suite. I was thinking about breaking it down into roughly MPI Standard > > chapters. > > > > If someone wants to contribute a whole suite of tests they can do so in a > > separate directory in the repo with it's own build system. The license must > > be compatible with the Open MPI license, and, in particular, allow the code > > to be freely distributed. > > > > > > Let me know what you think. Certainly everything here is open for > > discussion, and we will likely need to refine aspects as we go. > > > > -- Josh > > > > > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org <mailto:de...@open-mpi.org> > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > > <https://www.open-mpi.org/mailman/listinfo.cgi/devel> > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2016/05/18997.php > > <http://www.open-mpi.org/community/lists/devel/2016/05/18997.php> > _______________________________________________ > devel mailing list > de...@open-mpi.org <mailto:de...@open-mpi.org> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > <https://www.open-mpi.org/mailman/listinfo.cgi/devel> > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/05/19000.php > <http://www.open-mpi.org/community/lists/devel/2016/05/19000.php> > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/05/19002.php
