Re: Should Test2 maintain $! and $@?

[Chad Granum]

yeah, I am really not worried about the performance on this one, just noted
it for completeness.

What is nagging at me is what kentnl mentioned where something nested
catches an exception from something that failed to release it's context due
to the exception. Not because of these vars, but because it can cause other
problems, I need to look into it.

On Wed, Jan 13, 2016 at 11:58 AM, Sawyer X  wrote:

> [Top-posted]
>
> The extra cost would be:
> 1. Array storage
> 2. Push
> 3. Pop
>
> At the cost of supporting any level of nesting, I think it's a negligible
> cost, but I would profile it.
>
>
> On Wed, Jan 13, 2016 at 7:50 PM, Kent Fredric 
> wrote:
>
>> On 14 January 2016 at 07:39, Chad Granum  wrote:
>> > Right now the version I have up on cpan just stores them on creation,
>> and
>> > restores them on final release. Nothing happens for nested calls to
>> > context()/release(), all my downstream testing shows no breakages (not a
>> > full smoke, but does include several modules sensitive to $! and $@
>> > changes).
>>
>>
>> In the event some code like this dies:
>>
>>  sub foo {
>>   my $context = context();
>>   die "Bar";
>>  }
>>
>> What will happen with regards to $@ auto-stacking?
>>
>> If somebody catches the die in a higher context, what will $@ be?
>>
>>
>> --
>> Kent
>>
>> KENTNL - https://metacpan.org/author/KENTNL
>>
>
>

Re: Should Test2 maintain $! and $@?

[Chad Granum]

One remaining question.

Should the context maintain a stack, where each time context() is called
(nested) it pushes the vars to the stack, and each call to release pops it,
or should it just store them on creation, do nothing on retrieval, and only
restore them on the final release?

Right now the version I have up on cpan just stores them on creation, and
restores them on final release. Nothing happens for nested calls to
context()/release(), all my downstream testing shows no breakages (not a
full smoke, but does include several modules sensitive to $! and $@
changes).

This question is not about backwards compatibility, but rather about what
test tool authors might/should expect the behavior to be. Both maintain
backwards compatibility equally. However the stack form will have more of a
performance impact, probably minor though, would need to benchmark/profile.

-Chad

On Tue, Jan 12, 2016 at 6:12 PM, Sawyer X  wrote:

> [Top-posted]
>
> Chad, thank you for the detailed response. I think I now understand
> the scope of the problem and your solutions.
>
> I think it makes sense to put this in the guts inside the construction
> of a new context (or retrieval of current context) and in the release
> of that context. Kent, am I right to believe this also addresses the
> concerns you raised regarding testing of testing modules?
>
> I'm sorry to say I'm quite ignorant [also] when it comes to the
> testing underpinnings, so I'm not sure if there are additional
> concerns this does not address, but otherwise it sounds very
> reasonable to me.
>
> Thanks again for taking the time to clarify in detail. :)
> (It might be useful to start working on a document for the internals
> for anyone who wants to hack on it. This should at least be in the
> commit messages so it could be tracked down.)
>
> S.
>
>
>
> On Tue, Jan 12, 2016 at 11:26 PM, Chad Granum  wrote:
> > Yes, your understanding appears correct. And I can make it more clear.
> >
> > This is a very simple test tool in Test::Builder (the chrome):
> >
> >> my $TB = Test::Builder->new;  # Call to internals/guts (singleton)
> >>
> >> sub ok($;$) {
> >> my ($bool, $name) = @_;
> >> $TB->ok($bool, $name);# Call to internals/guts
> >> return $bool
> >> }
> >
> >
> > Here it is again using Test2 instead of Test::Builder (the chrome):
> >
> >> sub ok($;$) {
> >> my ($bool, $name) = @_;
> >> my $ctx = context();# Call to internals/guts (A)
> >> $ctx->ok($bool, $name); # another one (B)
> >> $ctx->release;  # another one (C)
> >> return $bool;
> >> }
> >
> >
> > The lines marked with 'Call to internals/guts' kick off a series of
> things
> > that read/write from filehandles, possibly opens them, evals code/catches
> > exceptions, and any number of other things that can squash $! and $@. It
> > should be noted that 'ok' is not the only method ever called on either
> > Test::Builder or $ctx, this is just a very simple illustration.
> >
> > Now for starters, Test::Builder uses a singleton, so it can do all its
> > initialization at load time, which allows it to leave several things
> > unprotected. The singleton was bad, so Test2 does not use one, which
> means
> > it has to be more protective of $! and $@ in more places to accomplish
> the
> > same end goal.
> >
> > History, what Test::Builder does: It localizes $! and $@ in an eval
> wrapper
> > called _try() that it uses to wrap things it expects could squash $! and
> $@.
> > It also localizes $SIG{__DIE__} for various reasons. In some places where
> > $SIG{__DIE__} should not be localized it will instead use local
> > independently of _try(). There is also extra logic for subtests to
> ensure $?
> > from the end of the subtest is carried-over into the regular testing
> outside
> > of the subtest. Some places also need to be careful of $? because they
> run
> > in an end block where squashing $? unintentionally is bad. (Yeah, $? is
> > involved in all this as well, but much less so)
> >
> > This results in a lot of places where things are localized, and several
> > places that run through an eval. People simply looking at the code may
> > overlook these things, and not know that the protection is happening. The
> > first time a new-dev will notice it is when tests start breaking because
> > they added an open/close/eval/etc in the wrong place. Thanfully there are
> > some tests for this, but not enough as I have found downstream things
> > (PerlIO::via::Timeout as an example) that break when $! is squashed in a
> way
> > Test::Builder never tests for.
> >
> > Test::Builder does not localize $! and $@ in all its public method.
> > Realistically it cannot for 2 reasons:
> >
> > Performance hit
> > Can mask real exceptions being thrown that are not caught by
> Test::Builder
> > itself.
> >
> > In short, this is a significant maintenance burden, with insufficient
> > testing, and no catch-all solution.
> >
> > 

Re: Should Test2 maintain $! and $@?

[Sawyer X]

[Top-posted]

The extra cost would be:
1. Array storage
2. Push
3. Pop

At the cost of supporting any level of nesting, I think it's a negligible
cost, but I would profile it.


On Wed, Jan 13, 2016 at 7:50 PM, Kent Fredric  wrote:

> On 14 January 2016 at 07:39, Chad Granum  wrote:
> > Right now the version I have up on cpan just stores them on creation, and
> > restores them on final release. Nothing happens for nested calls to
> > context()/release(), all my downstream testing shows no breakages (not a
> > full smoke, but does include several modules sensitive to $! and $@
> > changes).
>
>
> In the event some code like this dies:
>
>  sub foo {
>   my $context = context();
>   die "Bar";
>  }
>
> What will happen with regards to $@ auto-stacking?
>
> If somebody catches the die in a higher context, what will $@ be?
>
>
> --
> Kent
>
> KENTNL - https://metacpan.org/author/KENTNL
>

Re: Should Test2 maintain $! and $@?

[Kent Fredric]

On 14 January 2016 at 07:39, Chad Granum  wrote:
> Right now the version I have up on cpan just stores them on creation, and
> restores them on final release. Nothing happens for nested calls to
> context()/release(), all my downstream testing shows no breakages (not a
> full smoke, but does include several modules sensitive to $! and $@
> changes).


In the event some code like this dies:

 sub foo {
  my $context = context();
  die "Bar";
 }

What will happen with regards to $@ auto-stacking?

If somebody catches the die in a higher context, what will $@ be?


-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL