Re: [OMPI devel] MTT / Open MPI Visibility of New Failures

[Josh Hursey]

We are putting a pin in this for now. Amazon had an idea they wanted to run
with and report back. Once they have more info then we'll try to setup
another meeting.

On Tue, Jul 11, 2017 at 3:20 PM, Josh Hursey  wrote:

> In the Open MPI face-to-face meeting we had a long discussion about how to
> better harness MTT such that new failures are identified and the community
> alerted to their existence. The current manual way is not working. Our
> ultimate goal here is to know if we are making forward progress, and if
> something new fails the community knows about it immediately and
> automatically.
>
> We had a bunch of discussion and decided to think it over some more then
> come back to a teleconf in the next week or two to make a plan. The MTT
> group is scheduled to meet on Thursday, July 20 at 4 pm EST. We already
> know that doesn't work for everyone interested so I setup a doodle poll to
> pick a time for this specific discussion.
>
>   https://doodle.com/poll/8zdetnnv4iaaawg4
>
> *Please fill out the doodle poll by Thursday, July 13, 2017 at 5 pm EST.
> I'll pick a time then.*
>
>
>
> Three topics were identified to make progress.
>  1. Move MTT (Perl) Client to new server submission interface
>  2. Adding an "expected fail" list to MTT Client.
>  3. Drive the number of MTT reported failures to 0
>
> (1) This will start the movement to the new RESTful MTT server. Eventually
> we want to disable the old PHP submission mechanism. This is a first step.
> Josh needs to re-test this interface to confirm it is still working as
> expected with the existing 'v3' database.
>
> (2) For each branch/test-suite/runtime-configuration identify a test run
> as "known to fail". These will be flagged in the MTT Database and MTT
> Viewer/Reporter as separate from other failures ("failed, but we expected
> to pass"). Is this part of the INI file or a separate 'thing'?
>
> (3) The idea is that the "failed, but we expected to pass" number should
> be 0 for all sites. The individual site is responsible for maintaining
> their "known to fail" list. If the "failed, but we expected to pass" number
> is >0 then this is a 'new failure' and an email to the community is
> generated.
>
>
> --
> Josh Hursey
> IBM Spectrum MPI Developer
>



-- 
Josh Hursey
IBM Spectrum MPI Developer
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] MTT / Open MPI Visibility of New Failures

[gilles]

i think we should only run minimal tests *by default* for each PR

then we can have jenkins interpret some specific bot commands (for 
example :bot:test:ibm/collective)

so additional tests can be performed *on demand*

Cheers,

Gilles

- Original Message -

Thanks for the idea. For IBM's internal CI that's similar to what we 
do (but I don't think we take advantage of the XML report - I'll mention 
that to our CI folks) - we run MTT for every internal PR. It takes a few 
hours per PR to complete though. I don't know how acceptable that is for 
the community, but we should talk about it. Maybe there is a subset of 
the ompi-tests repo that we can run in each PR then let full MTT runs 
test more throughly.

On Wed, Jul 12, 2017 at 12:53 AM, Gilles Gouaillardet  wrote:

Josh and all,

an other or complementary idea is to use MTT from jenkins and 
with the
junit plugin.
iirc, the python client can generate a junit xml report, and 
fwiw, i
made a simplistic proof of concept in the perl client.
the idea is the junit jenkins plugins display a summary of all 
tests
(ok, ko, skip) but also provides some tendencies (e.g. things 
are
going better or worst)

if we only want to use the junit plugin, an option is to have a
"dummy" jenkins job that query results or download xml reports 
from
the mtt server, so the results and tendencies can be displayed 
by
jenkins

Cheers,

Gilles

On Wed, Jul 12, 2017 at 5:20 AM, Josh Hursey  wrote:
> In the Open MPI face-to-face meeting we had a long discussion 
about how to
> better harness MTT such that new failures are identified and 
the community
> alerted to their existence. The current manual way is not 
working. Our
> ultimate goal here is to know if we are making forward 
progress, and if
> something new fails the community knows about it immediately 
and
> automatically.
>
> We had a bunch of discussion and decided to think it over some 
more then
> come back to a teleconf in the next week or two to make a plan.
 The MTT
> group is scheduled to meet on Thursday, July 20 at 4 pm EST. 
We already know
> that doesn't work for everyone interested so I setup a doodle 
poll to pick a
> time for this specific discussion.
>
>   https://doodle.com/poll/8zdetnnv4iaaawg4
>
> Please fill out the doodle poll by Thursday, July 13, 2017 at 
5 pm EST. I'll
> pick a time then.
>
>
>
> Three topics were identified to make progress.
>  1. Move MTT (Perl) Client to new server submission interface
>  2. Adding an "expected fail" list to MTT Client.
>  3. Drive the number of MTT reported failures to 0
>
> (1) This will start the movement to the new RESTful MTT server.
 Eventually
> we want to disable the old PHP submission mechanism. This is a 
first step.
> Josh needs to re-test this interface to confirm it is still 
working as
> expected with the existing 'v3' database.
>
> (2) For each branch/test-suite/runtime-configuration identify 
a test run as
> "known to fail". These will be flagged in the MTT Database and 
MTT
> Viewer/Reporter as separate from other failures ("failed, but 
we expected to
> pass"). Is this part of the INI file or a separate 'thing'?
>
> (3) The idea is that the "failed, but we expected to pass" 
number should be
> 0 for all sites. The individual site is responsible for 
maintaining their
> "known to fail" list. If the "failed, but we expected to pass" 
number is >0
> then this is a 'new failure' and an email to the community is 
generated.
>
>
> --
> Josh Hursey
> IBM Spectrum MPI Developer
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel




-- 
Josh Hursey
IBM Spectrum MPI Developer



___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] MTT / Open MPI Visibility of New Failures

[Josh Hursey]

Thanks for the idea. For IBM's internal CI that's similar to what we do
(but I don't think we take advantage of the XML report - I'll mention that
to our CI folks) - we run MTT for every internal PR. It takes a few hours
per PR to complete though. I don't know how acceptable that is for the
community, but we should talk about it. Maybe there is a subset of the
ompi-tests repo that we can run in each PR then let full MTT runs test more
throughly.

On Wed, Jul 12, 2017 at 12:53 AM, Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:

> Josh and all,
>
> an other or complementary idea is to use MTT from jenkins and with the
> junit plugin.
> iirc, the python client can generate a junit xml report, and fwiw, i
> made a simplistic proof of concept in the perl client.
> the idea is the junit jenkins plugins display a summary of all tests
> (ok, ko, skip) but also provides some tendencies (e.g. things are
> going better or worst)
>
> if we only want to use the junit plugin, an option is to have a
> "dummy" jenkins job that query results or download xml reports from
> the mtt server, so the results and tendencies can be displayed by
> jenkins
>
> Cheers,
>
> Gilles
>
> On Wed, Jul 12, 2017 at 5:20 AM, Josh Hursey 
> wrote:
> > In the Open MPI face-to-face meeting we had a long discussion about how
> to
> > better harness MTT such that new failures are identified and the
> community
> > alerted to their existence. The current manual way is not working. Our
> > ultimate goal here is to know if we are making forward progress, and if
> > something new fails the community knows about it immediately and
> > automatically.
> >
> > We had a bunch of discussion and decided to think it over some more then
> > come back to a teleconf in the next week or two to make a plan. The MTT
> > group is scheduled to meet on Thursday, July 20 at 4 pm EST. We already
> know
> > that doesn't work for everyone interested so I setup a doodle poll to
> pick a
> > time for this specific discussion.
> >
> >   https://doodle.com/poll/8zdetnnv4iaaawg4
> >
> > Please fill out the doodle poll by Thursday, July 13, 2017 at 5 pm EST.
> I'll
> > pick a time then.
> >
> >
> >
> > Three topics were identified to make progress.
> >  1. Move MTT (Perl) Client to new server submission interface
> >  2. Adding an "expected fail" list to MTT Client.
> >  3. Drive the number of MTT reported failures to 0
> >
> > (1) This will start the movement to the new RESTful MTT server.
> Eventually
> > we want to disable the old PHP submission mechanism. This is a first
> step.
> > Josh needs to re-test this interface to confirm it is still working as
> > expected with the existing 'v3' database.
> >
> > (2) For each branch/test-suite/runtime-configuration identify a test
> run as
> > "known to fail". These will be flagged in the MTT Database and MTT
> > Viewer/Reporter as separate from other failures ("failed, but we
> expected to
> > pass"). Is this part of the INI file or a separate 'thing'?
> >
> > (3) The idea is that the "failed, but we expected to pass" number should
> be
> > 0 for all sites. The individual site is responsible for maintaining their
> > "known to fail" list. If the "failed, but we expected to pass" number is
> >0
> > then this is a 'new failure' and an email to the community is generated.
> >
> >
> > --
> > Josh Hursey
> > IBM Spectrum MPI Developer
> >
> > ___
> > devel mailing list
> > devel@lists.open-mpi.org
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>



-- 
Josh Hursey
IBM Spectrum MPI Developer
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] MTT / Open MPI Visibility of New Failures

[Gilles Gouaillardet]

Josh and all,

an other or complementary idea is to use MTT from jenkins and with the
junit plugin.
iirc, the python client can generate a junit xml report, and fwiw, i
made a simplistic proof of concept in the perl client.
the idea is the junit jenkins plugins display a summary of all tests
(ok, ko, skip) but also provides some tendencies (e.g. things are
going better or worst)

if we only want to use the junit plugin, an option is to have a
"dummy" jenkins job that query results or download xml reports from
the mtt server, so the results and tendencies can be displayed by
jenkins

Cheers,

Gilles

On Wed, Jul 12, 2017 at 5:20 AM, Josh Hursey  wrote:
> In the Open MPI face-to-face meeting we had a long discussion about how to
> better harness MTT such that new failures are identified and the community
> alerted to their existence. The current manual way is not working. Our
> ultimate goal here is to know if we are making forward progress, and if
> something new fails the community knows about it immediately and
> automatically.
>
> We had a bunch of discussion and decided to think it over some more then
> come back to a teleconf in the next week or two to make a plan. The MTT
> group is scheduled to meet on Thursday, July 20 at 4 pm EST. We already know
> that doesn't work for everyone interested so I setup a doodle poll to pick a
> time for this specific discussion.
>
>   https://doodle.com/poll/8zdetnnv4iaaawg4
>
> Please fill out the doodle poll by Thursday, July 13, 2017 at 5 pm EST. I'll
> pick a time then.
>
>
>
> Three topics were identified to make progress.
>  1. Move MTT (Perl) Client to new server submission interface
>  2. Adding an "expected fail" list to MTT Client.
>  3. Drive the number of MTT reported failures to 0
>
> (1) This will start the movement to the new RESTful MTT server. Eventually
> we want to disable the old PHP submission mechanism. This is a first step.
> Josh needs to re-test this interface to confirm it is still working as
> expected with the existing 'v3' database.
>
> (2) For each branch/test-suite/runtime-configuration identify a test run as
> "known to fail". These will be flagged in the MTT Database and MTT
> Viewer/Reporter as separate from other failures ("failed, but we expected to
> pass"). Is this part of the INI file or a separate 'thing'?
>
> (3) The idea is that the "failed, but we expected to pass" number should be
> 0 for all sites. The individual site is responsible for maintaining their
> "known to fail" list. If the "failed, but we expected to pass" number is >0
> then this is a 'new failure' and an email to the community is generated.
>
>
> --
> Josh Hursey
> IBM Spectrum MPI Developer
>
> ___
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

[OMPI devel] MTT / Open MPI Visibility of New Failures

[Josh Hursey]

In the Open MPI face-to-face meeting we had a long discussion about how to
better harness MTT such that new failures are identified and the community
alerted to their existence. The current manual way is not working. Our
ultimate goal here is to know if we are making forward progress, and if
something new fails the community knows about it immediately and
automatically.

We had a bunch of discussion and decided to think it over some more then
come back to a teleconf in the next week or two to make a plan. The MTT
group is scheduled to meet on Thursday, July 20 at 4 pm EST. We already
know that doesn't work for everyone interested so I setup a doodle poll to
pick a time for this specific discussion.

  https://doodle.com/poll/8zdetnnv4iaaawg4

*Please fill out the doodle poll by Thursday, July 13, 2017 at 5 pm EST.
I'll pick a time then.*



Three topics were identified to make progress.
 1. Move MTT (Perl) Client to new server submission interface
 2. Adding an "expected fail" list to MTT Client.
 3. Drive the number of MTT reported failures to 0

(1) This will start the movement to the new RESTful MTT server. Eventually
we want to disable the old PHP submission mechanism. This is a first step.
Josh needs to re-test this interface to confirm it is still working as
expected with the existing 'v3' database.

(2) For each branch/test-suite/runtime-configuration identify a test run as
"known to fail". These will be flagged in the MTT Database and MTT
Viewer/Reporter as separate from other failures ("failed, but we expected
to pass"). Is this part of the INI file or a separate 'thing'?

(3) The idea is that the "failed, but we expected to pass" number should be
0 for all sites. The individual site is responsible for maintaining their
"known to fail" list. If the "failed, but we expected to pass" number is >0
then this is a 'new failure' and an email to the community is generated.


-- 
Josh Hursey
IBM Spectrum MPI Developer
___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel