Re: [OMPI devel] RFC: Public Test Repo

[Christopher Samuel]

Hi Josh,

On 19/05/16 13:54, Josh Hursey wrote:

> Let me know what you think. Certainly everything here is open for
> discussion, and we will likely need to refine aspects as we go.

I think having an open test suite in conjunction with the current
private one is a great way to go, I think it sends the right message
about openness and hopefully allows a community to build around MPI
testing in general.

Certainly happy to try it out!

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [OMPI devel] RFC: Public Test Repo

[Jeff Squyres (jsquyres)]

I was initially for this idea, but now I find myself conflicted. 

Specifically: what's the value in yet another MPI test suite?

Sure, we've got a bunch of tests that no one else has (i.e., the things we've 
home-grown over the years). Some are great tests.  Some will need to be cleaned 
up and generalized. Some are user-contributed. Some are ompi-specific.  

Should these tests be contributed to some other test suite?

Specifically: I'm wondering about the MPICH test suite - that seems to be the 
only remaining big MPI test suite these days. Is it worth having a discussion w 
the MPICH folks to see if a) their test suite is general enough for all MPI 
implementations, and B) if they would accept a bunch of random tests from us?

And if not, I think I'd like to understand better the value add that we can 
provide by making another MPI test suite. 

Sent from my phone. No type good. 

> On May 18, 2016, at 11:54 PM, Josh Hursey  wrote:
> 
> WHAT: Create a public test repo (ompi-tests-public) to collect
> 
> WHY: ompi-tests is private, and difficult/impossible to open up. There is a 
> demand for a public collection of unit tests. This repo would allow us to 
> cultivate such a collection of unit tests.
> 
> WHERE: open-mpi GitHub project
> 
> TIMEOUT: Teleconf, Tue., May 24, 2016
> 
> MORE DETAIL:
> 
> Over the years we have had periodic public requests for access to our 
> ompi-tests repo. Opening up ompi-tests to the public is nearly impossible 
> since, given the age of some of these tests, determining if we can 
> redistribute these tests has become complicated.
> 
> Recently we had two different requests on the MTT users and Open MPI devel 
> list about access to the ompi-tests repo for testing. This got me thinking 
> that we could try to cultivate a public set of tests with a clear lineage and 
> license.
> 
> 
> Below are my current thoughts for structure and maintenance of the repo. 
> Certainly up for discussion.
> 
> Similar to the existing ompi-tests repo structure.
>  - Call the repository "ompi-tests-public"
>  - The repo will contain at least one test suite ('misc' - see below).
>  - Each test suite can have its own build system
>  - Each test suite should contain a README-MTT.md which will contain a sample 
> Test Build and Test Run section for use in MTT.
> 
> All tests contributed will be covered under the Open MPI license agreement 
> unless a specific test suite has a different, but compatible license. To 
> contribute a test to the repo a developer must sign the contributor 
> agreement. Contributions must go through a PR + Review process (similar to 
> how we maintain ompi-release). This is meant to provide an opportunity to 
> review the tests for correctness before acceptance into the repo.
> 
> We will seed the repo with a 'misc' test suite. This test suite is meant to 
> collect all of the miscellaneous tests created by Open MPI developers 
> including those regression tests that emerge as part of Issues or MPI Forum 
> discussions, for example. It will contain the unit tests currently in the 
> Open MPI's examples directory - what we have been calling the 'trivial' test 
> suite. I was thinking about breaking it down into roughly MPI Standard 
> chapters.
> 
> If someone wants to contribute a whole suite of tests they can do so in a 
> separate directory in the repo with it's own build system. The license must 
> be compatible with the Open MPI license, and, in particular, allow the code 
> to be freely distributed.
> 
> 
> Let me know what you think. Certainly everything here is open for discussion, 
> and we will likely need to refine aspects as we go.
> 
> -- Josh
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/05/18997.php

[OMPI devel] Fwd: Errored: open-mpi/ompi#1160 (master - 50b3775)

[George Bosilca]

Travis seems to be broken due to a ethtool issue.

  George.


-- Forwarded message --
From: Travis CI 
List-Post: devel@lists.open-mpi.org
Date: Thu, May 19, 2016 at 11:06 AM
Subject: Errored: open-mpi/ompi#1160 (master - 50b3775)
To: bosi...@icl.utk.edu


*open-mpi / ompi * (master
)
Build #1160 has errored.

27 minutes and 43 seconds
*George Bosilca* 50b3775

Changeset
→ 
  Don't overwrite the function argument.
In a MPMD setup the app in the jdata can be NULL, so make sure we
don't leave the main argument to an inconsistent value.

*Want to know about upcoming build environment updates?*

Would you like to stay up-to-date with the upcoming Travis CI build
environment updates? We set up a mailing list for you! Sign up here
.
Documentation  about Travis CI
For help please join our IRC channel irc.freenode.net#travis.
Choose who receives these build notification emails in your configuration
file .

*Would you like to test your private code?*

Travis CI for Private Projects

could be your new best friend!

Re: [OMPI devel] RFC: Public Test Repo

[Josh Hursey]

I think talking to the MPICH folks about creating a common test pool might
be useful. More useful would be to get the MPI Forum to 'bless' it and take
input from all of the MPI venders. Maybe you all can talk about that in a
sidebar at the MPI Forum - depending on what we figure out here.

The MPICH tests [1] contains a few test suites that are freely available as
tarball downloads on their site (though they seem a bit stale - last
updated 2012). They also have the 'tests' subdirectory in their repo [2].


>From my perspective, the benefits of creating a new public test repo on
GitHub would be (in no particular order, numbers provided to aid in
reference during followup discussion):

 (1) As we create new MPI test cases for issues, we file them publicly.
Currently, we drop them in ompi-tests for convenience. As a result, the
public cannot see/use them. Same for new test suites that we might create
as part of a new MPI standard interface proposal (e.g., FT, Sessions, or
even back to Nonblocking collectives).

 (2) GitHub allows us to foster a discussion about the tests in a way that
is easily traceable. More so than using a mailing list and discussing a
tarball drop of a test suite.

 (3) We can associate a MTT snippet of code for Test Build/Run that can be
used to run that test suite. Instead of putting this in with the MTT repo.
This would make these snippets slightly easier to find.

 (4) We can contain references to other test suites, even if we don't copy
them into our public test suite. This would allow us to say "hey, there are
these great MPICH/Netpipe/IMB/... tests at this URL" then give them
instructions on how we run them in MTT (or outside of MTT). If we have any
patches for that test suite we could collect them in the sub-directory
until (if) they get incorporated upstream. We could also associate notes on
runtime parameters that help tune Open MPI for a particular performance
test at various scales.

 (5) Generally, it would make it easier for the community to pick up some
unit tests and run them to do a 'make deep-check' on their MPI install. I
think there is a hesitation to use the test suite from another MPI
implementation with Open MPI as it might not be as consistent as one would
like. Any failures would have to be investigated as a difference of
implementation (do their tests use our CLI correctly?), difference of MPI
Standard interpretation (as often happens as the MPI Forum rolls out
errata), or actual bug in Open MPI. For an end user that is tough to
determine. For MPI implementers it is easier since we live those
differences.


Let me play a little devils advocate here too on a couple points [not that
I need to really, I'm sure I'll get some help here]. (#) reference the
points above.

(1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI Forum
to decide, and might be difficult to achieve given the diversity of
stakeholders at that level. However, if there is a public repo for MPI
tests that the MPI Forum can all agree on then that might be a good place
to start. I still think (4) would apply, but maybe we can solve that
differently.

(2) How much do we really debate the correctness of our tests? Early on in
the project, quite often. Lately it still happens (NULL datatype issue, for
example). That discussion can only happen internally since the test cases
we reference are not public unless we expose them (which is toe'ing the
line of redistribution). There is value in broadening the discussion
outside of the Open MPI developers on some of those test cases. At the very
least for educational benefit of the community about some of the more
subtle edges of the MPI standard.

(3) Shouldn't that sample code live with MTT? Maybe. I can see both sides
of this one. I do think there is some value in separating out the
functional code of MTT from the test suite configurations.


-- Josh

[1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html
[2] http://trac.mpich.org/projects/mpich/browser/test


On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres)  wrote:

> I was initially for this idea, but now I find myself conflicted.
>
> Specifically: what's the value in yet another MPI test suite?
>
> Sure, we've got a bunch of tests that no one else has (i.e., the things
> we've home-grown over the years). Some are great tests.  Some will need to
> be cleaned up and generalized. Some are user-contributed. Some are
> ompi-specific.
>
> Should these tests be contributed to some other test suite?
>
> Specifically: I'm wondering about the MPICH test suite - that seems to be
> the only remaining big MPI test suite these days. Is it worth having a
> discussion w the MPICH folks to see if a) their test suite is general
> enough for all MPI implementations, and B) if they would accept a bunch of
> random tests from us?
>
> And if not, I think I'd like to understand better the value add that we
> can provide by making another MPI test suite.
>
> Sent from my phone. No type good

Re: [OMPI devel] Fwd: Errored: open-mpi/ompi#1160 (master - 50b3775)

[Gilles Gouaillardet]

Note this affects OS X only.

in opal/util/ethtool.c, ethtool_cmd_speed must not be defined if not
HAVE_STRUCT_ETHTOOL_CMD

I will not be able to push a fix until tomorrow

Cheers,

Gilles

On Friday, May 20, 2016, George Bosilca  wrote:

> Travis seems to be broken due to a ethtool issue.
>
>   George.
>
>
> -- Forwarded message --
> From: Travis CI  >
> Date: Thu, May 19, 2016 at 11:06 AM
> Subject: Errored: open-mpi/ompi#1160 (master - 50b3775)
> To: bosi...@icl.utk.edu
> 
>
>
> *open-mpi / ompi * (master
> )
> Build #1160 has errored.
> 
> 27 minutes and 43 seconds
> *George Bosilca* 50b3775
> 
>  Changeset
> → 
>   Don't overwrite the function argument.
> In a MPMD setup the app in the jdata can be NULL, so make sure we
> don't leave the main argument to an inconsistent value.
>
> *Want to know about upcoming build environment updates?*
>
> Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you! Sign up here
> .
> Documentation  about Travis CI
> For help please join our IRC channel irc.freenode.net#travis.
> Choose who receives these build notification emails in your configuration
> file .
>
> *Would you like to test your private code?*
>
> Travis CI for Private Projects
> 
> could be your new best friend!
>
>

Re: [OMPI devel] Fwd: Errored: open-mpi/ompi#1160 (master - 50b3775)

[Jeff Squyres (jsquyres)]

I have a fix coming shortly; just about to push the PR...

> On May 19, 2016, at 11:29 AM, Gilles Gouaillardet 
>  wrote:
> 
> Note this affects OS X only.
> 
> in opal/util/ethtool.c, ethtool_cmd_speed must not be defined if not 
> HAVE_STRUCT_ETHTOOL_CMD
> 
> I will not be able to push a fix until tomorrow
> 
> Cheers,
> 
> Gilles
> 
> On Friday, May 20, 2016, George Bosilca  wrote:
> Travis seems to be broken due to a ethtool issue.
> 
>   George.
> 
> 
> -- Forwarded message --
> From: Travis CI 
> Date: Thu, May 19, 2016 at 11:06 AM
> Subject: Errored: open-mpi/ompi#1160 (master - 50b3775)
> To: bosi...@icl.utk.edu
> 
> 
>  open-mpi / ompi (master)
> 
> Build #1160 has errored.   27 minutes and 43 seconds
>   George Bosilca  50b3775 Changeset →
>   Don't overwrite the function argument.
> In a MPMD setup the app in the jdata can be NULL, so make sure we
> don't leave the main argument to an inconsistent value.
> Want to know about upcoming build environment updates?
> Would you like to stay up-to-date with the upcoming Travis CI build 
> environment updates? We set up a mailing list for you! Sign up here.
>   Documentation about Travis CI
> For help please join our IRC channel irc.freenode.net#travis.
> Choose who receives these build notification emails in your configuration 
> file.
> 
> Would you like to test your private code?
> Travis CI for Private Projects could be your new best friend!
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/05/19003.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI devel] RFC: Public Test Repo

[Ralph Castain]

Assume that we are able to “package” MTT so it can be upstreamed to the OpenHPC 
folks (as they have requested). It then becomes a little more “salable” as a 
generalized testing tool, which means that the tests in our repo could be used 
by others without having to deal with the MTT-specific harness calls.

At that point, it seems to me one could make a case for a general MPI test repo 
that supported MTT, and OpenHPC could even be a reasonable home for it. I can 
certainly raise the question there, if you like.


> On May 19, 2016, at 8:24 AM, Josh Hursey  wrote:
> 
> I think talking to the MPICH folks about creating a common test pool might be 
> useful. More useful would be to get the MPI Forum to 'bless' it and take 
> input from all of the MPI venders. Maybe you all can talk about that in a 
> sidebar at the MPI Forum - depending on what we figure out here.
> 
> The MPICH tests [1] contains a few test suites that are freely available as 
> tarball downloads on their site (though they seem a bit stale - last updated 
> 2012). They also have the 'tests' subdirectory in their repo [2].
> 
> 
> From my perspective, the benefits of creating a new public test repo on 
> GitHub would be (in no particular order, numbers provided to aid in reference 
> during followup discussion):
> 
>  (1) As we create new MPI test cases for issues, we file them publicly. 
> Currently, we drop them in ompi-tests for convenience. As a result, the 
> public cannot see/use them. Same for new test suites that we might create as 
> part of a new MPI standard interface proposal (e.g., FT, Sessions, or even 
> back to Nonblocking collectives).
> 
>  (2) GitHub allows us to foster a discussion about the tests in a way that is 
> easily traceable. More so than using a mailing list and discussing a tarball 
> drop of a test suite.
> 
>  (3) We can associate a MTT snippet of code for Test Build/Run that can be 
> used to run that test suite. Instead of putting this in with the MTT repo. 
> This would make these snippets slightly easier to find.
> 
>  (4) We can contain references to other test suites, even if we don't copy 
> them into our public test suite. This would allow us to say "hey, there are 
> these great MPICH/Netpipe/IMB/... tests at this URL" then give them 
> instructions on how we run them in MTT (or outside of MTT). If we have any 
> patches for that test suite we could collect them in the sub-directory until 
> (if) they get incorporated upstream. We could also associate notes on runtime 
> parameters that help tune Open MPI for a particular performance test at 
> various scales.
> 
>  (5) Generally, it would make it easier for the community to pick up some 
> unit tests and run them to do a 'make deep-check' on their MPI install. I 
> think there is a hesitation to use the test suite from another MPI 
> implementation with Open MPI as it might not be as consistent as one would 
> like. Any failures would have to be investigated as a difference of 
> implementation (do their tests use our CLI correctly?), difference of MPI 
> Standard interpretation (as often happens as the MPI Forum rolls out errata), 
> or actual bug in Open MPI. For an end user that is tough to determine. For 
> MPI implementers it is easier since we live those differences.
> 
> 
> Let me play a little devils advocate here too on a couple points [not that I 
> need to really, I'm sure I'll get some help here]. (#) reference the points 
> above.
> 
> (1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI Forum 
> to decide, and might be difficult to achieve given the diversity of 
> stakeholders at that level. However, if there is a public repo for MPI tests 
> that the MPI Forum can all agree on then that might be a good place to start. 
> I still think (4) would apply, but maybe we can solve that differently.
> 
> (2) How much do we really debate the correctness of our tests? Early on in 
> the project, quite often. Lately it still happens (NULL datatype issue, for 
> example). That discussion can only happen internally since the test cases we 
> reference are not public unless we expose them (which is toe'ing the line of 
> redistribution). There is value in broadening the discussion outside of the 
> Open MPI developers on some of those test cases. At the very least for 
> educational benefit of the community about some of the more subtle edges of 
> the MPI standard.
> 
> (3) Shouldn't that sample code live with MTT? Maybe. I can see both sides of 
> this one. I do think there is some value in separating out the functional 
> code of MTT from the test suite configurations.
> 
> 
> -- Josh
> 
> [1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html 
> 
> [2] http://trac.mpich.org/projects/mpich/browser/test 
> 
> 
> 
> On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres)