i think we should only run minimal tests *by default* for each PR
then we can have jenkins interpret some specific bot commands (for
example :bot:test:ibm/collective)
so additional tests can be performed *on demand*
Cheers,
Gilles
----- Original Message -----
Thanks for the idea. For IBM's internal CI that's similar to what we
do (but I don't think we take advantage of the XML report - I'll mention
that to our CI folks) - we run MTT for every internal PR. It takes a few
hours per PR to complete though. I don't know how acceptable that is for
the community, but we should talk about it. Maybe there is a subset of
the ompi-tests repo that we can run in each PR then let full MTT runs
test more throughly.
On Wed, Jul 12, 2017 at 12:53 AM, Gilles Gouaillardet <gilles.
gouaillar...@gmail.com> wrote:
Josh and all,
an other or complementary idea is to use MTT from jenkins and
with the
junit plugin.
iirc, the python client can generate a junit xml report, and
fwiw, i
made a simplistic proof of concept in the perl client.
the idea is the junit jenkins plugins display a summary of all
tests
(ok, ko, skip) but also provides some tendencies (e.g. things
are
going better or worst)
if we only want to use the junit plugin, an option is to have a
"dummy" jenkins job that query results or download xml reports
from
the mtt server, so the results and tendencies can be displayed
by
jenkins
Cheers,
Gilles
On Wed, Jul 12, 2017 at 5:20 AM, Josh Hursey <jjhursey@open-mpi.
org> wrote:
> In the Open MPI face-to-face meeting we had a long discussion
about how to
> better harness MTT such that new failures are identified and
the community
> alerted to their existence. The current manual way is not
working. Our
> ultimate goal here is to know if we are making forward
progress, and if
> something new fails the community knows about it immediately
and
> automatically.
>
> We had a bunch of discussion and decided to think it over some
more then
> come back to a teleconf in the next week or two to make a plan.
The MTT
> group is scheduled to meet on Thursday, July 20 at 4 pm EST.
We already know
> that doesn't work for everyone interested so I setup a doodle
poll to pick a
> time for this specific discussion.
>
> https://doodle.com/poll/8zdetnnv4iaaawg4
>
> Please fill out the doodle poll by Thursday, July 13, 2017 at
5 pm EST. I'll
> pick a time then.
>
>
>
> Three topics were identified to make progress.
> 1. Move MTT (Perl) Client to new server submission interface
> 2. Adding an "expected fail" list to MTT Client.
> 3. Drive the number of MTT reported failures to 0
>
> (1) This will start the movement to the new RESTful MTT server.
Eventually
> we want to disable the old PHP submission mechanism. This is a
first step.
> Josh needs to re-test this interface to confirm it is still
working as
> expected with the existing 'v3' database.
>
> (2) For each branch/test-suite/runtime-configuration identify
a test run as
> "known to fail". These will be flagged in the MTT Database and
MTT
> Viewer/Reporter as separate from other failures ("failed, but
we expected to
> pass"). Is this part of the INI file or a separate 'thing'?
>
> (3) The idea is that the "failed, but we expected to pass"
number should be
> 0 for all sites. The individual site is responsible for
maintaining their
> "known to fail" list. If the "failed, but we expected to pass"
number is >0
> then this is a 'new failure' and an email to the community is
generated.
>
>
> --
> Josh Hursey
> IBM Spectrum MPI Developer
>
> _______________________________________________
> devel mailing list
> devel@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
--
Josh Hursey
IBM Spectrum MPI Developer
_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
