Re: [Rd] Running package tests and not stop on first fail

[Jan Gorecki]

Sorry for late reply. I like the stop-on-error.
Thanks for merging.
Glad to be R contributor!

On 4 November 2016 at 09:42, Oliver Keyes  wrote:
> On Friday, 4 November 2016, Martin Maechler 
> wrote:
>>
>> > Dirk Eddelbuettel 
>> > on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>>
>> > On 4 November 2016 at 16:24, Martin Maechler wrote: | My
>> > proposed name '--no-stop-on-error' was a quick shot; if |
>> > somebody has a more concise or better "English style"
>> > wording | (which is somewhat compatible with all the other
>> > options you see | from 'R CMD check --help'), | please
>> > speak up.
>>
>> > Why not keep it simple?  The similar feature this most
>> > resembles is 'make -k' and its help page has
>>
>> >-k, --keep-going
>>
>> >Continue as much as possible after an
>> > error.  While the target that failed, and those that
>> > depend on it, cannot be remade, the other dependencies of
>> > these targets can be processed all the same.
>>
>> Yes, that would be quite a bit simpler and nice in my view.
>> One may think it to be too vague,
>
>
> Mmn, I would agree on vagueness (and it breaks the pattern set by other
> flags of human-readability). Deep familiarity with make is probably not
> something we should ask of everyone who needs to test a package, too.
>
> I quite like stop-on-error=true (exactly the same as the previous suggestion
> but shaves off some characters by inverting the Boolean)
>
>> notably from Brian Pedersen's mentioning that the examples are
>> already continued in any case if they lead to an error.
>>
>> Other opinions?
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Hervé Pagès]

Thanks Martin.

These changes are great and improve the usefulness of 'R CMD check'.
Especially in the context of the Bioconductor daily builds where
we'll use --no-stop-on-test-error so developers will get a full
picture of all the errors in their package at once.

Cheers,
H.

To provide some context to my special interest for this,

On 11/08/2016 01:34 AM, Martin Maechler wrote:

Hervé Pagès 
on Mon, 7 Nov 2016 14:37:15 -0800 writes:


> On 11/05/2016 01:53 PM, Martin Maechler wrote:
>>> Oliver Keyes 
>>> on Fri, 4 Nov 2016 12:42:54 -0400 writes:
>>
>> > On Friday, 4 November 2016, Martin Maechler
>> >  wrote:
>>
>> >> > Dirk Eddelbuettel >
>> >> > on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>> >>
>> >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
>> >> My > proposed name '--no-stop-on-error' was a quick shot;
>> >> if | > somebody has a more concise or better "English
>> >> style" > wording | (which is somewhat compatible with all
>> >> the other > options you see | from 'R CMD check --help'),
>> >> | please > speak up.
>> >>
>> >> > Why not keep it simple?  The similar feature this most
>> >> > resembles is 'make -k' and its help page has
>> >>
>> >> > -k, --keep-going
>> >>
>> >> > Continue as much as possible after an > error.  While
>> >> the target that failed, and those that > depend on it,
>> >> cannot be remade, the other dependencies of > these
>> >> targets can be processed all the same.
>> >>
>> >> Yes, that would be quite a bit simpler and nice in my
>> >> view.  One may think it to be too vague,
>>
>> > Mmn, I would agree on vagueness (and it breaks the pattern
>> > set by other flags of human-readability). Deep familiarity
>> > with make is probably not something we should ask of
>> > everyone who needs to test a package, too.
>>
>> > I quite like stop-on-error=true (exactly the same as the
>> > previous suggestion but shaves off some characters by
>> > inverting the Boolean)
>>
>> Thank you, Brian, Dirk and Oliver for these (and some offline)
>> thoughts and suggestions!
>>
>> My current summary:
>>
>> 1) I really don't want a  --=value
>> but rather stay with logical/binary variables that "express
>> themselves"... in the same way I strongly prefer
>>
>> if (A_is_special)   
>> to
>> if (A_special == TRUE)  
>>
>> for a logical variable A_* .   Yes, this is mostly a matter
>> of taste,.. but related to how R style itself "works"
>>
>> 2) Brian mentioned that this is only about ./tests/ tests which
>> are continued, not about the Examples which are treated separately.
>> That's why we had contemplated additionally using 'tests' (because that's
>> the directory name used for unit/regression/.. tests) in the option
>> name.
>>
>> Even though Brian is correct, ideally we *would* want to also influence 
the
>> examples' running to *not* stop on a first error..   However that would
>> need more work, reorganizing how the examples are run and that may not be
>> worth the pain.   However it should be considered a goal in the long run.

> My name is Hervé, and I was not suggesting that what happens with the
> examples should be changed. I was just preaching consistency (again
> sorry) between what happens with the examples and what happens with
> the tests.

Thank you, Hervé and excuse me for not answering more focused on
what you said.
I think I do understand what you say (at least by now :-)) and
agree that consistency is something important and to be strived for,
also with these options.

> Why not simply change the latter?
> Do we really need an option to control this?

Very good questions.  If the change could be made much better,
I'd agree we'd not need a new option because the change could be
considerided uniformly better than the current (R 3.3.2, say) behavior.
However the change as it is currently, is not good enough to be
the only option (see below).

> The behavior was changed for the examples a couple of
> years ago and nobody felt the need to introduce an option
> to control this at the time.

Yes, that change was made very nicely (not by me) and I'd say
the result *was* uniformly better than the previous behavior, so
there did not seem much of a reason to still provide the old behavior.

  >> After all that, I tend to remain with the original proposed name. It 
is at
  >> least easy to pronounce and spell correctly...

> Unless --no-stop-on-error controls both (i.e. examples and tests), and
> both behave the same way by default, this is a misnomer.

Well, if you choose the option, there *is* no stop on errors anymore
... because the 

Re: [Rd] Running package tests and not stop on first fail

[Martin Maechler]

> Hervé Pagès 
> on Mon, 7 Nov 2016 14:37:15 -0800 writes:

> On 11/05/2016 01:53 PM, Martin Maechler wrote:
>>> Oliver Keyes 
>>> on Fri, 4 Nov 2016 12:42:54 -0400 writes:
>> 
>> > On Friday, 4 November 2016, Martin Maechler
>> >  wrote:
>> 
>> >> > Dirk Eddelbuettel >
>> >> > on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>> >>
>> >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
>> >> My > proposed name '--no-stop-on-error' was a quick shot;
>> >> if | > somebody has a more concise or better "English
>> >> style" > wording | (which is somewhat compatible with all
>> >> the other > options you see | from 'R CMD check --help'),
>> >> | please > speak up.
>> >>
>> >> > Why not keep it simple?  The similar feature this most
>> >> > resembles is 'make -k' and its help page has
>> >>
>> >> > -k, --keep-going
>> >>
>> >> > Continue as much as possible after an > error.  While
>> >> the target that failed, and those that > depend on it,
>> >> cannot be remade, the other dependencies of > these
>> >> targets can be processed all the same.
>> >>
>> >> Yes, that would be quite a bit simpler and nice in my
>> >> view.  One may think it to be too vague,
>> 
>> > Mmn, I would agree on vagueness (and it breaks the pattern
>> > set by other flags of human-readability). Deep familiarity
>> > with make is probably not something we should ask of
>> > everyone who needs to test a package, too.
>> 
>> > I quite like stop-on-error=true (exactly the same as the
>> > previous suggestion but shaves off some characters by
>> > inverting the Boolean)
>> 
>> Thank you, Brian, Dirk and Oliver for these (and some offline)
>> thoughts and suggestions!
>> 
>> My current summary:
>> 
>> 1) I really don't want a  --=value
>> but rather stay with logical/binary variables that "express
>> themselves"... in the same way I strongly prefer
>> 
>> if (A_is_special)   
>> to
>> if (A_special == TRUE)  
>> 
>> for a logical variable A_* .   Yes, this is mostly a matter
>> of taste,.. but related to how R style itself "works"
>> 
>> 2) Brian mentioned that this is only about ./tests/ tests which
>> are continued, not about the Examples which are treated separately.
>> That's why we had contemplated additionally using 'tests' (because that's
>> the directory name used for unit/regression/.. tests) in the option
>> name.
>> 
>> Even though Brian is correct, ideally we *would* want to also influence 
the
>> examples' running to *not* stop on a first error..   However that would
>> need more work, reorganizing how the examples are run and that may not be
>> worth the pain.   However it should be considered a goal in the long run.

> My name is Hervé, and I was not suggesting that what happens with the
> examples should be changed. I was just preaching consistency (again
> sorry) between what happens with the examples and what happens with
> the tests. 

Thank you, Hervé and excuse me for not answering more focused on
what you said.
I think I do understand what you say (at least by now :-)) and
agree that consistency is something important and to be strived for,
also with these options.

> Why not simply change the latter?
> Do we really need an option to control this? 

Very good questions.  If the change could be made much better,
I'd agree we'd not need a new option because the change could be
considerided uniformly better than the current (R 3.3.2, say) behavior.
However the change as it is currently, is not good enough to be
the only option (see below). 

> The behavior was changed for the examples a couple of
> years ago and nobody felt the need to introduce an option
> to control this at the time.

Yes, that change was made very nicely (not by me) and I'd say
the result *was* uniformly better than the previous behavior, so
there did not seem much of a reason to still provide the old behavior.

  >> After all that, I tend to remain with the original proposed name. It 
is at
  >> least easy to pronounce and spell correctly...

> Unless --no-stop-on-error controls both (i.e. examples and tests), and 
> both behave the same way by default, this is a misnomer.

Well, if you choose the option, there *is* no stop on errors anymore
... because the examples nowadays never stop the (R CMD check Pkg) 
from running on an error as we've mentioned above.

[[--- Digression on  Examples' checking 

  However / on the other hand: Because of the way the examples
  are run --- efficiently from one single R source file --- it
  is not so easy there, to let them run further: The first error
  from all the examples 

Re: [Rd] Running package tests and not stop on first fail

[Hervé Pagès]

On 11/05/2016 01:53 PM, Martin Maechler wrote:

Oliver Keyes 
on Fri, 4 Nov 2016 12:42:54 -0400 writes:


> On Friday, 4 November 2016, Martin Maechler
>  wrote:

>> > Dirk Eddelbuettel >
>> > on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>>
>> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
>> My > proposed name '--no-stop-on-error' was a quick shot;
>> if | > somebody has a more concise or better "English
>> style" > wording | (which is somewhat compatible with all
>> the other > options you see | from 'R CMD check --help'),
>> | please > speak up.
>>
>> > Why not keep it simple?  The similar feature this most
>> > resembles is 'make -k' and its help page has
>>
>> > -k, --keep-going
>>
>> > Continue as much as possible after an > error.  While
>> the target that failed, and those that > depend on it,
>> cannot be remade, the other dependencies of > these
>> targets can be processed all the same.
>>
>> Yes, that would be quite a bit simpler and nice in my
>> view.  One may think it to be too vague,

> Mmn, I would agree on vagueness (and it breaks the pattern
> set by other flags of human-readability). Deep familiarity
> with make is probably not something we should ask of
> everyone who needs to test a package, too.

> I quite like stop-on-error=true (exactly the same as the
> previous suggestion but shaves off some characters by
> inverting the Boolean)

Thank you, Brian, Dirk and Oliver for these (and some offline)
thoughts and suggestions!

My current summary:

1) I really don't want a  --=value
   but rather stay with logical/binary variables that "express
   themselves"... in the same way I strongly prefer

   if (A_is_special)   
   to
   if (A_special == TRUE)  

   for a logical variable A_* .   Yes, this is mostly a matter
   of taste,.. but related to how R style itself "works"

2) Brian mentioned that this is only about ./tests/ tests which
   are continued, not about the Examples which are treated separately.
   That's why we had contemplated additionally using 'tests' (because that's
   the directory name used for unit/regression/.. tests) in the option
   name.

Even though Brian is correct, ideally we *would* want to also influence the
examples' running to *not* stop on a first error..   However that would
need more work, reorganizing how the examples are run and that may not be
worth the pain.   However it should be considered a goal in the long run.


My name is Hervé, and I was not suggesting that what happens with the
examples should be changed. I was just preaching consistency (again
sorry) between what happens with the examples and what happens with
the tests. Why not simply change the latter? Do we really need an option
to control this? The behavior was changed for the examples a couple of
years ago and nobody felt the need to introduce an option to control
this at the time.



After all that, I tend to remain with the original proposed name. It is at
least easy to pronounce and spell correctly...


Unless --no-stop-on-error controls both (i.e. examples and tests), and 
both behave the same way by default, this is a misnomer.


If this option controls the tests only, it should be more something like
--no-stop-on-test-error. If in the long run, another option is added to
control the examples, then could be --no-stop-on-example-error.

But again, maybe all this is not needed at all. Maybe changing what
happens with the tests would be enough.

Cheers,
H.



Martin

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Martin Maechler]

> Dirk Eddelbuettel 
> on Fri, 4 Nov 2016 10:36:52 -0500 writes:

> On 4 November 2016 at 16:24, Martin Maechler wrote: | My
> proposed name '--no-stop-on-error' was a quick shot; if |
> somebody has a more concise or better "English style"
> wording | (which is somewhat compatible with all the other
> options you see | from 'R CMD check --help'), | please
> speak up.

> Why not keep it simple?  The similar feature this most
> resembles is 'make -k' and its help page has

>-k, --keep-going
   
>Continue as much as possible after an
> error.  While the target that failed, and those that
> depend on it, cannot be remade, the other dependencies of
> these targets can be processed all the same.

Yes, that would be quite a bit simpler and nice in my view.
One may think it to be too vague,
notably from Brian Pedersen's mentioning that the examples are
already continued in any case if they lead to an error.

Other opinions?

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Martin Maechler]

> Brian G Peterson 
> on Fri, 4 Nov 2016 10:37:18 -0500 writes:

> On Fri, 2016-11-04 at 16:24 +0100, Martin Maechler wrote:
>> > Jan Gorecki  > on Fri, 4
>> Nov 2016 11:20:37 + writes:
>> 
>> > Martin, I submitted very simple patch on >
>> https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176
>> 
>> > Herve, While I like your idea, I prefer to keep my
>> patch > simple, it is now exactly what Martin mentions. I
>> think it > is a good start that can eventually be
>> extended later for > what you are asking.
>> 
>> I tend to agree; this seems indeed much easier than I
>> anticipated.  Thank you, Jan!
>> 
>> I'm testing a version which uses the logical variable
>> 'stop_on_error' rather than 'no_stop_on_error' (because
>> !no_stop_on_error is hard to mentally parse quickly).
>> 
>> My proposed name '--no-stop-on-error' was a quick shot;
>> if somebody has a more concise or better "English style"
>> wording (which is somewhat compatible with all the other
>> options you see from 'R CMD check --help'), please speak
>> up.

> I might suggest

> --stop-tests-on-error

> with default=TRUE to match current functionality.

Thank you, Brian.

though that would be less concise and I think less matching the
'R CMD check' philosophy with many '--no-*' options to turn
*off* defaults. Note that most options have no " = " part, because
they are binary and I think that's easiest for use (when the 'binary' case 
is general enough). Also   R CMD check --help  
ends saying  "By default, all test sections are turned on."
which does fit the use of all those '--no-*' options.

OTOH, we also have  '--ignore-vignettes'
so we could consider

   --ignore-tests-errors

?


> This might avoid any confusion related to the behavior of
> continuing to run examples on error in R CMD check.

You are quite right on that, indeed.
Martin

> Regards,
> Brian

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Dirk Eddelbuettel]

On 4 November 2016 at 16:24, Martin Maechler wrote:
| My proposed name  '--no-stop-on-error'  was a quick shot; if
| somebody has a more concise or better "English style" wording
| (which is somewhat compatible with all the other options you see
| from 'R CMD check --help'),
| please speak up.

Why not keep it simple?  The similar feature this most resembles is 'make -k'
and its help page has

   -k, --keep-going
   
   Continue as much as possible after an error.  While the
   target that failed, and those that depend on it, cannot be
   remade, the other dependencies of these targets can be
   processed all the same.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Martin Maechler]

> Jan Gorecki 
> on Fri, 4 Nov 2016 11:20:37 + writes:

> Martin, I submitted very simple patch on
> https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176

> Herve, While I like your idea, I prefer to keep my patch
> simple, it is now exactly what Martin mentions. I think it
> is a good start that can eventually be extended later for
> what you are asking.

I tend to agree; this seems indeed much easier than I
anticipated.  Thank you, Jan!

I'm testing a version which uses the logical variable
'stop_on_error' rather than 'no_stop_on_error' (because
!no_stop_on_error is hard to mentally parse quickly).

My proposed name  '--no-stop-on-error'  was a quick shot; if
somebody has a more concise or better "English style" wording
(which is somewhat compatible with all the other options you see
from 'R CMD check --help'),
please speak up.

Martin

> Regards, Jan

> On 3 November 2016 at 17:25, Hervé Pagès
>  wrote:
>> 
>> Hi Martin, Jan,
>> 
>> On 11/03/2016 03:45 AM, Martin Maechler wrote:
 
 Jan Gorecki  on Tue, 1 Nov
 2016 22:51:28 + writes:
>>> 
>>> 
>>> > Hello community/devs, Is there an option to run
>>> package > tests during R CMD check and not stop on first
>>> error? I > know that testing frameworks (testhat and
>>> others) can do > that but asking about just R and base
>>> packages. Currently > when package check runs test
>>> scripts in ./tests directory > it will stop after first
>>> fail.  Do you think it could be > optionally available
>>> to continue to run tests after > failures?  Regards, Jan
>>> Gorecki
>>> 
>>> I agree that this would be a useful option sometimes.
>>> 
>>> So I would be supportive to get such an option, say,
>>> 
>>> R CMD check --no-stop-on-error 
>> 
>> 
>> A couple of years ago the behavior of 'R CMD check' was
>> changed to continue checking (e.g. the examples) after
>> many types of errors, and to output a summary count of
>> errors at the end if any have occurred.  So
>> --no-stop-on-error could easily be interpreted as an
>> option that controls this behavior (and would also
>> suggest that the default has been reverted back to what
>> it was prior to R 3.2.0), rather than an option that
>> specifically controls what should happen while running
>> the tests.
>> 
>> Cheers, H.
>> 
>>> 
>>> into R if someone provided (relatively small) patches to
>>> the R sources (i.e. subversion repos at
>>> https://svn.r-project.org/R/trunk/ ).  The relevant
>>> source code should basically all be in
>>> src/library/tools/R/testing.R
>>> 
>>> Note that this may be complicated, also because
>>> "parallel" checking is available in parts, via the
>>> TEST_MC_CORES environment variable ((which is currently
>>> only quickly documented in the 'R Administration ..'
>>> manual))
>>> 
>>> 
>>> Martin Maechler ETH Zurich
>>> 
>>> __
>>> R-devel@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>> 
>> 
>> --
>> Hervé Pagès
>> 
>> Program in Computational Biology Division of Public
>> Health Sciences Fred Hutchinson Cancer Research Center
>> 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA
>> 98109-1024
>> 
>> E-mail: hpa...@fredhutch.org Phone: (206) 667-5791 Fax:
>> (206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Jan Gorecki]

Martin,
I submitted very simple patch on
https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176

Herve,
While I like your idea, I prefer to keep my patch simple, it is now
exactly what Martin mentions. I think it is a good start that can
eventually be extended later for what you are asking.

Regards,
Jan

On 3 November 2016 at 17:25, Hervé Pagès  wrote:
>
> Hi Martin, Jan,
>
> On 11/03/2016 03:45 AM, Martin Maechler wrote:
>>>
>>> Jan Gorecki 
>>> on Tue, 1 Nov 2016 22:51:28 + writes:
>>
>>
>> > Hello community/devs, Is there an option to run package
>> > tests during R CMD check and not stop on first error? I
>> > know that testing frameworks (testhat and others) can do
>> > that but asking about just R and base packages. Currently
>> > when package check runs test scripts in ./tests directory
>> > it will stop after first fail.  Do you think it could be
>> > optionally available to continue to run tests after
>> > failures?  Regards, Jan Gorecki
>>
>> I agree that this would be a useful option sometimes.
>>
>> So I would be supportive to get such an option, say,
>>
>>R CMD check --no-stop-on-error  
>
>
> A couple of years ago the behavior of 'R CMD check' was changed to
> continue checking (e.g. the examples) after many types of errors, and
> to output a summary count of errors at the end if any have occurred.
> So --no-stop-on-error could easily be interpreted as an option that
> controls this behavior (and would also suggest that the default has
> been reverted back to what it was prior to R 3.2.0), rather than an
> option that specifically controls what should happen while running
> the tests.
>
> Cheers,
> H.
>
>>
>> into R if someone provided (relatively small) patches to the R
>> sources (i.e. subversion repos at https://svn.r-project.org/R/trunk/ ).
>> The relevant source code should basically all be in
>> src/library/tools/R/testing.R
>>
>> Note that this may be complicated, also because "parallel"
>> checking is available in parts, via the TEST_MC_CORES
>> environment variable ((which is currently only quickly
>> documented in the 'R Administration ..' manual))
>>
>>
>> Martin Maechler
>> ETH Zurich
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpa...@fredhutch.org
> Phone:  (206) 667-5791
> Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Hervé Pagès]

Hi Martin, Jan,

On 11/03/2016 03:45 AM, Martin Maechler wrote:

Jan Gorecki 
on Tue, 1 Nov 2016 22:51:28 + writes:


> Hello community/devs, Is there an option to run package
> tests during R CMD check and not stop on first error? I
> know that testing frameworks (testhat and others) can do
> that but asking about just R and base packages. Currently
> when package check runs test scripts in ./tests directory
> it will stop after first fail.  Do you think it could be
> optionally available to continue to run tests after
> failures?  Regards, Jan Gorecki

I agree that this would be a useful option sometimes.

So I would be supportive to get such an option, say,

   R CMD check --no-stop-on-error  


A couple of years ago the behavior of 'R CMD check' was changed to
continue checking (e.g. the examples) after many types of errors, and
to output a summary count of errors at the end if any have occurred.
So --no-stop-on-error could easily be interpreted as an option that
controls this behavior (and would also suggest that the default has
been reverted back to what it was prior to R 3.2.0), rather than an
option that specifically controls what should happen while running
the tests.

Cheers,
H.



into R if someone provided (relatively small) patches to the R
sources (i.e. subversion repos at https://svn.r-project.org/R/trunk/ ).
The relevant source code should basically all be in
src/library/tools/R/testing.R

Note that this may be complicated, also because "parallel"
checking is available in parts, via the TEST_MC_CORES
environment variable ((which is currently only quickly
documented in the 'R Administration ..' manual))


Martin Maechler
ETH Zurich

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Running package tests and not stop on first fail

[Martin Maechler]

> Jan Gorecki 
> on Tue, 1 Nov 2016 22:51:28 + writes:

> Hello community/devs, Is there an option to run package
> tests during R CMD check and not stop on first error? I
> know that testing frameworks (testhat and others) can do
> that but asking about just R and base packages. Currently
> when package check runs test scripts in ./tests directory
> it will stop after first fail.  Do you think it could be
> optionally available to continue to run tests after
> failures?  Regards, Jan Gorecki

I agree that this would be a useful option sometimes.

So I would be supportive to get such an option, say,

   R CMD check --no-stop-on-error  

into R if someone provided (relatively small) patches to the R
sources (i.e. subversion repos at https://svn.r-project.org/R/trunk/ ).
The relevant source code should basically all be in
src/library/tools/R/testing.R

Note that this may be complicated, also because "parallel"
checking is available in parts, via the TEST_MC_CORES
environment variable ((which is currently only quickly
documented in the 'R Administration ..' manual))


Martin Maechler
ETH Zurich

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Running package tests and not stop on first fail

[Jan Gorecki]

Hello community/devs,
Is there an option to run package tests during R CMD check and not stop on
first error? I know that testing frameworks (testhat and others) can do
that but asking about just R and base packages. Currently when package
check runs test scripts in ./tests directory it will stop after first fail.
Do you think it could be optionally available to continue to run tests
after failures?
Regards,
Jan Gorecki

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel