Re: [OMPI devel] RFC: Public Test Repo

[Josh Hursey]

If OpenHPC folks want to cultivate such a repo I think that would be great.

If such a repo existed then I think we should work with them on making sure
that it integrates nicely into MTT so we can add it to the nightly
regressions. Getting a release of MTT is something I would like to see
happen sooner rather than later. Now that we have more folks working in the
code, we might be able to do that this year. But that's a discussion for
the MTT list.


The things I would want in such a repo would be:
 (1) Ability to submit new individual unit tests and test suites
 (2) Ability to discuss existing and proposed tests in a similar way to
what we have on GitHub
 (3) Ability to create some 'best practice' notes on running a test suite
with a particular MPI implementation. This will become important for any
performance regression tests that might get pushed there.

If such a repo existed, then we (Open MPI) could come up with a nice
FAQ/wikipage describing how 'ompi-tests' is private, but if one wanted a
broader range of tests then look at the public repo from OpenHPC. Then give
some instructions on the MTT side about how to drop it in to those
configurations.


On Thu, May 19, 2016 at 5:24 PM, Ralph Castain  wrote:

> Assume that we are able to “package” MTT so it can be upstreamed to the
> OpenHPC folks (as they have requested). It then becomes a little more
> “salable” as a generalized testing tool, which means that the tests in our
> repo could be used by others without having to deal with the MTT-specific
> harness calls.
>
> At that point, it seems to me one could make a case for a general MPI test
> repo that supported MTT, and OpenHPC could even be a reasonable home for
> it. I can certainly raise the question there, if you like.
>
>
> On May 19, 2016, at 8:24 AM, Josh Hursey  wrote:
>
> I think talking to the MPICH folks about creating a common test pool might
> be useful. More useful would be to get the MPI Forum to 'bless' it and take
> input from all of the MPI venders. Maybe you all can talk about that in a
> sidebar at the MPI Forum - depending on what we figure out here.
>
> The MPICH tests [1] contains a few test suites that are freely available
> as tarball downloads on their site (though they seem a bit stale - last
> updated 2012). They also have the 'tests' subdirectory in their repo [2].
>
>
> From my perspective, the benefits of creating a new public test repo on
> GitHub would be (in no particular order, numbers provided to aid in
> reference during followup discussion):
>
>  (1) As we create new MPI test cases for issues, we file them publicly.
> Currently, we drop them in ompi-tests for convenience. As a result, the
> public cannot see/use them. Same for new test suites that we might create
> as part of a new MPI standard interface proposal (e.g., FT, Sessions, or
> even back to Nonblocking collectives).
>
>  (2) GitHub allows us to foster a discussion about the tests in a way that
> is easily traceable. More so than using a mailing list and discussing a
> tarball drop of a test suite.
>
>  (3) We can associate a MTT snippet of code for Test Build/Run that can be
> used to run that test suite. Instead of putting this in with the MTT repo.
> This would make these snippets slightly easier to find.
>
>  (4) We can contain references to other test suites, even if we don't copy
> them into our public test suite. This would allow us to say "hey, there are
> these great MPICH/Netpipe/IMB/... tests at this URL" then give them
> instructions on how we run them in MTT (or outside of MTT). If we have any
> patches for that test suite we could collect them in the sub-directory
> until (if) they get incorporated upstream. We could also associate notes on
> runtime parameters that help tune Open MPI for a particular performance
> test at various scales.
>
>  (5) Generally, it would make it easier for the community to pick up some
> unit tests and run them to do a 'make deep-check' on their MPI install. I
> think there is a hesitation to use the test suite from another MPI
> implementation with Open MPI as it might not be as consistent as one would
> like. Any failures would have to be investigated as a difference of
> implementation (do their tests use our CLI correctly?), difference of MPI
> Standard interpretation (as often happens as the MPI Forum rolls out
> errata), or actual bug in Open MPI. For an end user that is tough to
> determine. For MPI implementers it is easier since we live those
> differences.
>
>
> Let me play a little devils advocate here too on a couple points [not that
> I need to really, I'm sure I'll get some help here]. (#) reference the
> points above.
>
> (1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI
> Forum to decide, and might be difficult to achieve given the diversity of
> stakeholders at that level. However, if there is a public repo for MPI
> tests that the MPI Forum can all agree on then that might be a 

Re: [OMPI devel] RFC: Public Test Repo

[Ralph Castain]

Assume that we are able to “package” MTT so it can be upstreamed to the OpenHPC 
folks (as they have requested). It then becomes a little more “salable” as a 
generalized testing tool, which means that the tests in our repo could be used 
by others without having to deal with the MTT-specific harness calls.

At that point, it seems to me one could make a case for a general MPI test repo 
that supported MTT, and OpenHPC could even be a reasonable home for it. I can 
certainly raise the question there, if you like.


> On May 19, 2016, at 8:24 AM, Josh Hursey  wrote:
> 
> I think talking to the MPICH folks about creating a common test pool might be 
> useful. More useful would be to get the MPI Forum to 'bless' it and take 
> input from all of the MPI venders. Maybe you all can talk about that in a 
> sidebar at the MPI Forum - depending on what we figure out here.
> 
> The MPICH tests [1] contains a few test suites that are freely available as 
> tarball downloads on their site (though they seem a bit stale - last updated 
> 2012). They also have the 'tests' subdirectory in their repo [2].
> 
> 
> From my perspective, the benefits of creating a new public test repo on 
> GitHub would be (in no particular order, numbers provided to aid in reference 
> during followup discussion):
> 
>  (1) As we create new MPI test cases for issues, we file them publicly. 
> Currently, we drop them in ompi-tests for convenience. As a result, the 
> public cannot see/use them. Same for new test suites that we might create as 
> part of a new MPI standard interface proposal (e.g., FT, Sessions, or even 
> back to Nonblocking collectives).
> 
>  (2) GitHub allows us to foster a discussion about the tests in a way that is 
> easily traceable. More so than using a mailing list and discussing a tarball 
> drop of a test suite.
> 
>  (3) We can associate a MTT snippet of code for Test Build/Run that can be 
> used to run that test suite. Instead of putting this in with the MTT repo. 
> This would make these snippets slightly easier to find.
> 
>  (4) We can contain references to other test suites, even if we don't copy 
> them into our public test suite. This would allow us to say "hey, there are 
> these great MPICH/Netpipe/IMB/... tests at this URL" then give them 
> instructions on how we run them in MTT (or outside of MTT). If we have any 
> patches for that test suite we could collect them in the sub-directory until 
> (if) they get incorporated upstream. We could also associate notes on runtime 
> parameters that help tune Open MPI for a particular performance test at 
> various scales.
> 
>  (5) Generally, it would make it easier for the community to pick up some 
> unit tests and run them to do a 'make deep-check' on their MPI install. I 
> think there is a hesitation to use the test suite from another MPI 
> implementation with Open MPI as it might not be as consistent as one would 
> like. Any failures would have to be investigated as a difference of 
> implementation (do their tests use our CLI correctly?), difference of MPI 
> Standard interpretation (as often happens as the MPI Forum rolls out errata), 
> or actual bug in Open MPI. For an end user that is tough to determine. For 
> MPI implementers it is easier since we live those differences.
> 
> 
> Let me play a little devils advocate here too on a couple points [not that I 
> need to really, I'm sure I'll get some help here]. (#) reference the points 
> above.
> 
> (1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI Forum 
> to decide, and might be difficult to achieve given the diversity of 
> stakeholders at that level. However, if there is a public repo for MPI tests 
> that the MPI Forum can all agree on then that might be a good place to start. 
> I still think (4) would apply, but maybe we can solve that differently.
> 
> (2) How much do we really debate the correctness of our tests? Early on in 
> the project, quite often. Lately it still happens (NULL datatype issue, for 
> example). That discussion can only happen internally since the test cases we 
> reference are not public unless we expose them (which is toe'ing the line of 
> redistribution). There is value in broadening the discussion outside of the 
> Open MPI developers on some of those test cases. At the very least for 
> educational benefit of the community about some of the more subtle edges of 
> the MPI standard.
> 
> (3) Shouldn't that sample code live with MTT? Maybe. I can see both sides of 
> this one. I do think there is some value in separating out the functional 
> code of MTT from the test suite configurations.
> 
> 
> -- Josh
> 
> [1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html 
> 
> [2] http://trac.mpich.org/projects/mpich/browser/test 
> 
> 
> 
> On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres) 

Re: [OMPI devel] RFC: Public Test Repo

[Josh Hursey]

I think talking to the MPICH folks about creating a common test pool might
be useful. More useful would be to get the MPI Forum to 'bless' it and take
input from all of the MPI venders. Maybe you all can talk about that in a
sidebar at the MPI Forum - depending on what we figure out here.

The MPICH tests [1] contains a few test suites that are freely available as
tarball downloads on their site (though they seem a bit stale - last
updated 2012). They also have the 'tests' subdirectory in their repo [2].


>From my perspective, the benefits of creating a new public test repo on
GitHub would be (in no particular order, numbers provided to aid in
reference during followup discussion):

 (1) As we create new MPI test cases for issues, we file them publicly.
Currently, we drop them in ompi-tests for convenience. As a result, the
public cannot see/use them. Same for new test suites that we might create
as part of a new MPI standard interface proposal (e.g., FT, Sessions, or
even back to Nonblocking collectives).

 (2) GitHub allows us to foster a discussion about the tests in a way that
is easily traceable. More so than using a mailing list and discussing a
tarball drop of a test suite.

 (3) We can associate a MTT snippet of code for Test Build/Run that can be
used to run that test suite. Instead of putting this in with the MTT repo.
This would make these snippets slightly easier to find.

 (4) We can contain references to other test suites, even if we don't copy
them into our public test suite. This would allow us to say "hey, there are
these great MPICH/Netpipe/IMB/... tests at this URL" then give them
instructions on how we run them in MTT (or outside of MTT). If we have any
patches for that test suite we could collect them in the sub-directory
until (if) they get incorporated upstream. We could also associate notes on
runtime parameters that help tune Open MPI for a particular performance
test at various scales.

 (5) Generally, it would make it easier for the community to pick up some
unit tests and run them to do a 'make deep-check' on their MPI install. I
think there is a hesitation to use the test suite from another MPI
implementation with Open MPI as it might not be as consistent as one would
like. Any failures would have to be investigated as a difference of
implementation (do their tests use our CLI correctly?), difference of MPI
Standard interpretation (as often happens as the MPI Forum rolls out
errata), or actual bug in Open MPI. For an end user that is tough to
determine. For MPI implementers it is easier since we live those
differences.


Let me play a little devils advocate here too on a couple points [not that
I need to really, I'm sure I'll get some help here]. (#) reference the
points above.

(1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI Forum
to decide, and might be difficult to achieve given the diversity of
stakeholders at that level. However, if there is a public repo for MPI
tests that the MPI Forum can all agree on then that might be a good place
to start. I still think (4) would apply, but maybe we can solve that
differently.

(2) How much do we really debate the correctness of our tests? Early on in
the project, quite often. Lately it still happens (NULL datatype issue, for
example). That discussion can only happen internally since the test cases
we reference are not public unless we expose them (which is toe'ing the
line of redistribution). There is value in broadening the discussion
outside of the Open MPI developers on some of those test cases. At the very
least for educational benefit of the community about some of the more
subtle edges of the MPI standard.

(3) Shouldn't that sample code live with MTT? Maybe. I can see both sides
of this one. I do think there is some value in separating out the
functional code of MTT from the test suite configurations.


-- Josh

[1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html
[2] http://trac.mpich.org/projects/mpich/browser/test


On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres)  wrote:

> I was initially for this idea, but now I find myself conflicted.
>
> Specifically: what's the value in yet another MPI test suite?
>
> Sure, we've got a bunch of tests that no one else has (i.e., the things
> we've home-grown over the years). Some are great tests.  Some will need to
> be cleaned up and generalized. Some are user-contributed. Some are
> ompi-specific.
>
> Should these tests be contributed to some other test suite?
>
> Specifically: I'm wondering about the MPICH test suite - that seems to be
> the only remaining big MPI test suite these days. Is it worth having a
> discussion w the MPICH folks to see if a) their test suite is general
> enough for all MPI implementations, and B) if they would accept a bunch of
> random tests from us?
>
> And if not, I think I'd like to understand better the value add that we
> can provide by making another MPI test suite.
>
> Sent from 

Re: [OMPI devel] RFC: Public Test Repo

[Jeff Squyres (jsquyres)]

I was initially for this idea, but now I find myself conflicted. 

Specifically: what's the value in yet another MPI test suite?

Sure, we've got a bunch of tests that no one else has (i.e., the things we've 
home-grown over the years). Some are great tests.  Some will need to be cleaned 
up and generalized. Some are user-contributed. Some are ompi-specific.  

Should these tests be contributed to some other test suite?

Specifically: I'm wondering about the MPICH test suite - that seems to be the 
only remaining big MPI test suite these days. Is it worth having a discussion w 
the MPICH folks to see if a) their test suite is general enough for all MPI 
implementations, and B) if they would accept a bunch of random tests from us?

And if not, I think I'd like to understand better the value add that we can 
provide by making another MPI test suite. 

Sent from my phone. No type good. 

> On May 18, 2016, at 11:54 PM, Josh Hursey  wrote:
> 
> WHAT: Create a public test repo (ompi-tests-public) to collect
> 
> WHY: ompi-tests is private, and difficult/impossible to open up. There is a 
> demand for a public collection of unit tests. This repo would allow us to 
> cultivate such a collection of unit tests.
> 
> WHERE: open-mpi GitHub project
> 
> TIMEOUT: Teleconf, Tue., May 24, 2016
> 
> MORE DETAIL:
> 
> Over the years we have had periodic public requests for access to our 
> ompi-tests repo. Opening up ompi-tests to the public is nearly impossible 
> since, given the age of some of these tests, determining if we can 
> redistribute these tests has become complicated.
> 
> Recently we had two different requests on the MTT users and Open MPI devel 
> list about access to the ompi-tests repo for testing. This got me thinking 
> that we could try to cultivate a public set of tests with a clear lineage and 
> license.
> 
> 
> Below are my current thoughts for structure and maintenance of the repo. 
> Certainly up for discussion.
> 
> Similar to the existing ompi-tests repo structure.
>  - Call the repository "ompi-tests-public"
>  - The repo will contain at least one test suite ('misc' - see below).
>  - Each test suite can have its own build system
>  - Each test suite should contain a README-MTT.md which will contain a sample 
> Test Build and Test Run section for use in MTT.
> 
> All tests contributed will be covered under the Open MPI license agreement 
> unless a specific test suite has a different, but compatible license. To 
> contribute a test to the repo a developer must sign the contributor 
> agreement. Contributions must go through a PR + Review process (similar to 
> how we maintain ompi-release). This is meant to provide an opportunity to 
> review the tests for correctness before acceptance into the repo.
> 
> We will seed the repo with a 'misc' test suite. This test suite is meant to 
> collect all of the miscellaneous tests created by Open MPI developers 
> including those regression tests that emerge as part of Issues or MPI Forum 
> discussions, for example. It will contain the unit tests currently in the 
> Open MPI's examples directory - what we have been calling the 'trivial' test 
> suite. I was thinking about breaking it down into roughly MPI Standard 
> chapters.
> 
> If someone wants to contribute a whole suite of tests they can do so in a 
> separate directory in the repo with it's own build system. The license must 
> be compatible with the Open MPI license, and, in particular, allow the code 
> to be freely distributed.
> 
> 
> Let me know what you think. Certainly everything here is open for discussion, 
> and we will likely need to refine aspects as we go.
> 
> -- Josh
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/05/18997.php

Re: [OMPI devel] RFC: Public Test Repo

[Christopher Samuel]

Hi Josh,

On 19/05/16 13:54, Josh Hursey wrote:

> Let me know what you think. Certainly everything here is open for
> discussion, and we will likely need to refine aspects as we go.

I think having an open test suite in conjunction with the current
private one is a great way to go, I think it sends the right message
about openness and hopefully allows a community to build around MPI
testing in general.

Certainly happy to try it out!

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci