Branch: refs/heads/smoke-me/davem/rc_stack2 Home: https://github.com/Perl/perl5 Commit: 24b19b5ff0128e55c377933f81cbdada1ce4c4a7 https://github.com/Perl/perl5/commit/24b19b5ff0128e55c377933f81cbdada1ce4c4a7 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023)
Changed paths: M pp_ctl.c M t/op/svleak.t Log Message: ----------- fix leaks in new ${^HOOK}{require__*} facility In the initial implementation, a few temporary SVs ended up with a ref count one too high. Commit: af26e127b2471ce18f7cda719f4c91a10a51b3f1 https://github.com/Perl/perl5/commit/af26e127b2471ce18f7cda719f4c91a10a51b3f1 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M perl.c M perl.h Log Message: ----------- add PERL_RC_STACK define (not yet defined) This macro turns on stack reference counting. Add the macro, but commented out for now. In addition, temporarily define and enable PERL_XXX_TMP_NORC. This is a temporary transition measure: even in the presence of PERL_RC_STACK, do not actually modify reference counts yet. So PERL_RC_STACK will still define all the new static functions and code behaviour, except for the bits that actually modify reference counts in ways that assume the stack is reference counted. It allows this branch to have many commits which incrementally deal with adding support for PERL_RC_STACK, with each individual commit still working while half of the core is still not yet "ref count aware". Commit: 45c43ff3713fb60dcb508257dc402d4c0e8196e4 https://github.com/Perl/perl5/commit/45c43ff3713fb60dcb508257dc402d4c0e8196e4 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M lib/Internals.pod M universal.c Log Message: ----------- add Internals::stack_refcounted() This indicates whether perl has been built with PERL_RC_STACK. For now this should only be used by core; It's not yet decided whether CPAN modules have a use case too. Commit: 0315e7ccaf8c1d4fd605120946a774608a209ff6 https://github.com/Perl/perl5/commit/0315e7ccaf8c1d4fd605120946a774608a209ff6 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M ext/Devel-Peek/t/Peek.t Log Message: ----------- ext/Devel-Peek/t/Peek.t: fixup top refcount There are a few 'expected' Dump() output patterns which expect the refcount of the top-most SV in the tree to be fixed number like REFCNT = 1 Change all those occurrences to REFCNT = \d+ or similar, since a reference-counted stack will cause higher values Commit: eefb89f8710557e8293fe07268ac60118b4689c0 https://github.com/Perl/perl5/commit/eefb89f8710557e8293fe07268ac60118b4689c0 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c M scope.c Log Message: ----------- pp_dbstate: skip SAVESTACK_POS() On PERL_RC_STACK builds, skip the call to SAVESTACK_POS(). I suspect that saving the stack position is no longer required. It was added in 5.001 by: NETaa13155: &DB::DB left trash on the stack. From: Thomas Koenig Files patched: lib/perl5db.pl pp_ctl.c The call by pp_dbstate() to &DB::DB left trash on the stack. It now calls DB in list context, and DB returns (). but the details of what bug it fixed are long lost to history. SAVESTACK_POS() doesn't work well with stacks which may be split into partly reference-counted and partly not halves, so skip it and hope it doesn't cause any problems. Also, assert that SAVEt_STACK_POS isn't used anywhere any more. Commit: 483d0748e726d2f2fc3ac0bf08d1c46d00e04835 https://github.com/Perl/perl5/commit/483d0748e726d2f2fc3ac0bf08d1c46d00e04835 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- pp_mapwhile(): null out holes in stack >From time to time, pp_mapwhile() shifts up any remaining args on the stack, in order to leave enough space to add the current results to any accumulated results. This leaves a hole in the stack, which might not be fully filled until the last iteration. As well as making the output of perl -Ds confusing, any garbage left in the hole will double-count towards the reference count of each SV once the stack becomes reference-counted. So fill the newly-created hole in the stack with NULL pointers. Commit: 4450252fdf1e1cfb154100e816b3edf610af2306 https://github.com/Perl/perl5/commit/4450252fdf1e1cfb154100e816b3edf610af2306 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M av.c M embed.fnc M embed.h M inline.h M proto.h Log Message: ----------- Add av_remove_offset() core-only function This function abstracts away removing any AvARRAY(av) - AvALLOC(av) offset from an array (previously added by av_shift()). Also use Zero() rather than a loop to NULL out any elements in the offset area. Commit: bb78617877c4aa2e7d167b6934b394b1fc3fa679 https://github.com/Perl/perl5/commit/bb78617877c4aa2e7d167b6934b394b1fc3fa679 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M embed.fnc M embed.h M op.c M perl.c M pp_ctl.c M proto.h Log Message: ----------- create_eval_scope(): parameterise the stack pos Rather than always using PL_stack_sp to pass to cx_pushblock(), allow the caller of create_eval_scope() to specify what stack position is to be saved. This will be used shortly to fix a bug. Commit: 3b1df59e652df74b00eca1b7def7d5d96e899f06 https://github.com/Perl/perl5/commit/3b1df59e652df74b00eca1b7def7d5d96e899f06 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M perl.c Log Message: ----------- Perl_call_sv(G_EVAL): set oldsp correctly call_sv(cv, G_EVAL) pushes a CXt_EVAL context, fakes up an entersub op, then runs it, letting entersub consume the arguments pushed onto the current stack frame by call_sv()'s caller. So the C-level PUSHMARK(): PUSHs(arg1); ... PUSHs(argN); call_sv(cv, G_EVAL); is about equivalent to the perl-level: eval { cv($arg1,..$argN) } Except that in the latter, when the CXt_EVAL is pushed, it is done *before* any of the arguments have been pushed, so the eval context's blk_oldsp points to just before the arguments. Conversely in the case of call_sv(), blk_oldsp gets set to just *after* the arguments. During an exception, the context stack gets unwound until the CXt_EVAL is reached, then PL_stack_sp is restored to the blk_oldsp value. The difference currently makes no real functional difference, as call_sv() resets PL_stack_sp immediately after an exception anyway, so the bad just-restored value is discarded. But it does mean that PL_stack_sp briefly points to junk on the stack. When the stack becomes reference-counted, this will start to become important. So fix now. This commit changes it so that call_sv()'s blk_oldsp gets set to just before the arguments too. It also asserts that the stack was cleared back to blk_oldsp. Commit: fe247cc2a834f9061acd1e24a09107912a33d3b7 https://github.com/Perl/perl5/commit/fe247cc2a834f9061acd1e24a09107912a33d3b7 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M inline.h Log Message: ----------- assert that cxstack[i].oldsp are increasing The saved stack positions cxstack[0].oldsp cxstack[1].oldsp cxstack[2].oldsp ... effectively divide the argument stack into a series of stack frames. This unravels if an earlier oldsp is actually greater than a later one. It didn't really matter before, but for a reference-counted stack it becomes a mess when trying to unwind the stack after an exception. So assert that oldsp never decreases. This assert found one failure in core, which was fixed by the previous commit. Commit: 5eb1ca40a31a79e43b01e848930883ba035c611c https://github.com/Perl/perl5/commit/5eb1ca40a31a79e43b01e848930883ba035c611c Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M autodoc.pl M embed.fnc M embed.h M inline.h M proto.h Log Message: ----------- add rpp_foo() refcounted-stack manipulation fns The various new inline functions like rpp_push_1() are intended to be (approximate) replacements for the traditional PUSHs(), POPs() etc macros, except that when compiled under PERL_RC_STACK, they understand a reference-counted stack. Except currently they assume the stack is still not reference-counted in the presence of PERL_XXX_TMP_NORC. This will be turned off in a few commits' time. Commit: 84dc01ec7cdb97b321177cc328fba65f4b880fe3 https://github.com/Perl/perl5/commit/84dc01ec7cdb97b321177cc328fba65f4b880fe3 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_hot.c Log Message: ----------- use new rpp_is_lone() function This inline function checks for whether an SV is only referenced from the stack or temps stack. (Well, it only checks TEMPs currently, but will shortly handle a ref-counted stack too). In particular it tests for: SvTEMP(sv) && && SvREFCNT(sv) == 1 This commit uses it in various places to replace where the conditions are currently written out in full. This will allow those places to still work correctly when the stack becomes reference-counted. Commit: c8ef37a4579a926c1026205a1bf1d952ef58324a https://github.com/Perl/perl5/commit/c8ef37a4579a926c1026205a1bf1d952ef58324a Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M doop.c M embed.fnc M makedef.pl M pp.c M pp.h M pp_ctl.c M pp_hot.c M pp_pack.c M pp_sys.c M proto.h Log Message: ----------- wrap normal non-RC-stack-aware pp functions In the presence of PERL_RC_STACK, wrap most pp functions with a function that calls the real pp function with a (non-reference-counted) copy of the stack arguments. On return, the wrapper bumps up the reference count of the returned list and, as appropriate, shifts them down the stack. This allows a pp function which is not aware of a reference-counted stack to be called in a reference-counted stack environment. About 250 such functions are wrappable; the remaining 50 or so need individual handling and will be dealt with in the following few commits. This is a temporary band-aid which slows down calls to pp() functions. It is expected that over time these functions will be unwrapped and updated to to understand a reference-counted stack. But by temporarily wrapping them now, it allows perl to be (experimentally) shifted over to a reference-counted stack without having to review and modify all 30K lines of pp code first. (Except currently, the wrapper assumes the stack is still not reference-counted in the presence of PERL_XXX_TMP_NORC. This will be turned off in a few commits' time.) Note also that pp_wrap() is not yet complete. It simulates wrapping by making a copy of the args on entry to the wrapped function and shifting any returned values back down. But as well as being temporarily inhibited from actually modifying reference counts by PERL_XXX_TMP_NORC, there are extra considerations which will be taken account of by the "Allow stacks to be reference-counted" commit coming later. Commit: 0127060817386de4129d6fca3147bef7f4cc1e6f https://github.com/Perl/perl5/commit/0127060817386de4129d6fca3147bef7f4cc1e6f Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_hot.c Log Message: ----------- pp_method*: fixup wrapping The various method ops are sometimes naughty and substitute a different object at the base of the current stack frame. This is "below" the area of the stack normally processed by pp_wrap(), so the reference count adjusting must be done manually. Commit: 3f50405bb9fa401ba743487b928067b0b9e6fac0 https://github.com/Perl/perl5/commit/3f50405bb9fa401ba743487b928067b0b9e6fac0 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: regcomp Update pp_regcomp() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 1c04d885d33f04e11f9b7f276fd6b1ef244ec0a2 https://github.com/Perl/perl5/commit/1c04d885d33f04e11f9b7f276fd6b1ef244ec0a2 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M embed.fnc M makedef.pl M pp_hot.c M proto.h Log Message: ----------- make RC-stack-aware: entersub Update pp_entersub() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. As well as the expected changes to pp_entersub(), this commit also adds a function, xs_wrap(), which is similar in concept to pp_wrap(), but which makes a non-reference-counted copy of the arguments on the stack before calling any XS function which isn't RC-stack-aware yet (which is currently all of them). xs_wrap() will soon be used by pp_goto() too. In the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Note also that xs_wrap() is not yet complete. It simulates wrapping by making a copy of the args on entry to the wrapped function and shifting any returned values back down. But as well as being temporarily inhibited from actually modifying reference counts by PERL_XXX_TMP_NORC, there are extra considerations which will be taken account of by the "Allow stacks to be reference-counted" commit coming later. Commit: 8d4809e11c5faef6b7d0ab2501a30a99d3847494 https://github.com/Perl/perl5/commit/8d4809e11c5faef6b7d0ab2501a30a99d3847494 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c M pp_hot.c Log Message: ----------- make RC-stack-aware: db/nextstate, unstack Update pp_dbstate(), pp_nextstate() and pp_unstack() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 8b2bf0708150757b9125e287f998edbfe06d26f1 https://github.com/Perl/perl5/commit/8b2bf0708150757b9125e287f998edbfe06d26f1 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c M pp_hot.c M t/op/grep.t Log Message: ----------- make RC-stack-aware: map and grep Update pp_grepstart(), pp_grepwhile() and pp_mapwhile() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Also add some tests to op/grep.t to check for stuff being freed at an appropriate time (they weren't being at one iteration of this commit). Commit: bab8138a304058df60a2959e20127fbe8b743583 https://github.com/Perl/perl5/commit/bab8138a304058df60a2959e20127fbe8b743583 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: enteriter, leaveloop Update pp_enteriter() and pp_leveloop() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: c5c6ee9f72326d6040f9907a8cd87f0a32491e2e https://github.com/Perl/perl5/commit/c5c6ee9f72326d6040f9907a8cd87f0a32491e2e Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c M pp_hot.c Log Message: ----------- make RC-stack-aware: pp_leavefoo() Update some pp_leavefoo() functions to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 3dd66a643c9d044534c6a495d0faf0f076a4fc8d https://github.com/Perl/perl5/commit/3dd66a643c9d044534c6a495d0faf0f076a4fc8d Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: last Update pp_last() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: a710a39b8ec62ecb8dfd014c8e94f576d98fdd05 https://github.com/Perl/perl5/commit/a710a39b8ec62ecb8dfd014c8e94f576d98fdd05 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: goto Update pp_goto() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: d58a9687ece49682574c3a8b0ed24b0126d2053c https://github.com/Perl/perl5/commit/d58a9687ece49682574c3a8b0ed24b0126d2053c Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: require,eval Update pp_entereval() et al to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 2bdae954afd170109645a528fc0b0d7169557d4c https://github.com/Perl/perl5/commit/2bdae954afd170109645a528fc0b0d7169557d4c Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: enter/leavegiven etc Update pp_entergiven() et al to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 9ea8db092b53cf229caa321b171d526e0656c9ae https://github.com/Perl/perl5/commit/9ea8db092b53cf229caa321b171d526e0656c9ae Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_sys.c Log Message: ----------- make RC-stack-aware: enter/leavewrite Update pp_enterwrite() and pp_leavewrite() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 1d026c9a9bef8fd6dfb68f1b87980f4c73d15aee https://github.com/Perl/perl5/commit/1d026c9a9bef8fd6dfb68f1b87980f4c73d15aee Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_sort.c M t/op/sort.t Log Message: ----------- make RC-stack-aware: pp_sort() Update pp_sort() to handle a reference-counted stack environment in the presence of PERL_RC_STACK. (But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time.) In addition to replacing all the POPs type stuff with rpp_foo() inline function calls, three slight functional changes were done to make this work. 1) The stack entry pointing to the SV containing the sort function name is now NULLed out rather than being left on the stack, just after being freed. It is usually overwritten later on in the process. 2) For an in-place sort (@a = sort @a), the AV is now left on the stack so that it isn't prematurely freed. This is okay, because... 3) Also for an in-place sort, after the results are stored in @a, the stack frame is cleared. This is safe because the optimisation is only ever done in void context. Previously the stack would be be cleared by the following nextstate op or similar anyway. Commit: ffbd35f20eb7a9f2a352ab8fd49f8f08b548d511 https://github.com/Perl/perl5/commit/ffbd35f20eb7a9f2a352ab8fd49f8f08b548d511 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_sort.c Log Message: ----------- make RC-stack-aware: sort cmp function invokers Update S_sortcv() etc to handle a reference-counted stack environment in the presence of PERL_RC_STACK. These three static functions are used by pp_sort() to invoke a suitable comparison subroutine. (But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time.) Commit: 3a3aca281e76e5041e18616c8b20c3236538972d https://github.com/Perl/perl5/commit/3a3aca281e76e5041e18616c8b20c3236538972d Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M doio.c M pp_sys.c Log Message: ----------- make RC-stack-aware: stat and filetest ops Update the pp functions which do stat and filetest operations to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 55469df3b1439b5fd94b277db8a3450c5cb5699c https://github.com/Perl/perl5/commit/55469df3b1439b5fd94b277db8a3450c5cb5699c Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M inline.h M pp.c Log Message: ----------- make RC-stack-aware: misc pp() functions Update the remaining miscellaneous pp() functions to handle a reference-counted stack environment in the presence of PERL_RC_STACK. But in the presence of PERL_XXX_TMP_NORC, don't actually manipulate reference counts yet. This will be turned off in a few commits' time. Commit: 8f76dea6161eeb96da03a5593ce485a6da444aec https://github.com/Perl/perl5/commit/8f76dea6161eeb96da03a5593ce485a6da444aec Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M perl.c Log Message: ----------- make RC-stack-aware: call_sv() and similar Update the functions which directly or indirectly call a CV, to handle a reference-counted stack environment. Note that call_sv(), eval_pv() etc are slightly unusual in that they can be called from either a reference-counted or non-RC stack environment, so must be able to handle either case. This is done mostly by relying on CALLRUNOPS() to fix things up, but for anything pushed/pulled around that, by checking rpp_stack_is_rc(): which currently always returns false, but once ref-counting is enabled in a few commits' time, may start returning true. Commit: 0f64491c9cdbd9e79417d197149fc445b2dc4146 https://github.com/Perl/perl5/commit/0f64491c9cdbd9e79417d197149fc445b2dc4146 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M perl.c Log Message: ----------- make RC-stack-aware: eval_sv() Similar to the previous commit which fixed up call_sv() and similar, the commit updates eval_sv() to handle a reference-counted stack environment. Note that these functions are slightly unusual in that they can be called from either a reference-counted or non-RC stack environment, so must be able to handle either case. This is done mostly by relying on CALLRUNOPS() to fix things up, but for anything pushed/pulled around that, by checking rpp_stack_is_rc(): which currently always returns false, but once ref-counting is enabled in a few commits' time, may start returning true. Commit: 76ad30a8202531cbab9b21b3188e2c9c70558d33 https://github.com/Perl/perl5/commit/76ad30a8202531cbab9b21b3188e2c9c70558d33 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- make RC-stack-aware: defer The defer keyword saves the address of an optree, which is then run during leave_scope(). Since that can happen in unpredictable places (e.g. during a croak), we won't know until runtime where the stack is currently reference-counted. So handle both cases. Commit: 55bb823071f6875e1909fe98cfce3f545654ce19 https://github.com/Perl/perl5/commit/55bb823071f6875e1909fe98cfce3f545654ce19 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_hot.c Log Message: ----------- make RC-stack-aware: unwrap pp_add() Remove the temporary wrapper from pp_add() and update it to directly work in a refcounted-stack environment. This is mainly a proof of concept to check that all the new rpp_foo() inline functions and macros are suitable for working with a typical binary operator. Commit: 8b55d99e490ddf171372987ccfd06d63a2dae1ed https://github.com/Perl/perl5/commit/8b55d99e490ddf171372987ccfd06d63a2dae1ed Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp.c Log Message: ----------- make RC-stack-aware: unwrap pp_not() Remove the temporary wrapper from pp_not() and update it to directly work in a refcounted-stack environment. This is mainly a proof of concept to check that rpp_tryAMAGICun_MG() and its underlying Perl_try_amagic_un() work ok. Commit: b3c301bbeeed7c75d093f1e007b0b24fb0ce1ae6 https://github.com/Perl/perl5/commit/b3c301bbeeed7c75d093f1e007b0b24fb0ce1ae6 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M op.c Log Message: ----------- make RC-stack-aware: S_fold_constants() The folded SV it pops off the stack should be treated as reference counted. Commit: 810deb04af49a239ee7c55ff58b87ca3cf407cf7 https://github.com/Perl/perl5/commit/810deb04af49a239ee7c55ff58b87ca3cf407cf7 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M ext/Devel-Peek/Peek.pm M ext/Devel-Peek/Peek.xs Log Message: ----------- make RC-stack-aware: Devel::Peek::Dump() This function is implemented as a custom op rather than as a function call, so it wasn't getting wrapped for reference-count purposes by either pp_wrap() nor xs_wrap(). This commit makes it directly handle a reference-counted stack. This was a particularly confusing one, as a failing test in t/op/eval.t was reduced to use Devel::Peek; Dump($@); eval { die "boo\n"; }; Dump($@); which showed the ref count of $@ increasing after the eval {}. But it was actually Dump(), not eval{} which was messing with the ref count. Commit: 29ea9b8a5a35fcfde92f289a21765150c242850a https://github.com/Perl/perl5/commit/29ea9b8a5a35fcfde92f289a21765150c242850a Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M sv.c Log Message: ----------- make RC-stack-aware: perl_clone() It has some code that calls out to PACKAGE::CLONE(). Depending on how the interpreter is being cloned, the code which assembles the args on the stack for the CLONE() call may be operating on either a reference-counted stack, or a non-RC one. So handle both cases Commit: 064d2ad0fc89254fc8a61f7728bbd6335ee3a165 https://github.com/Perl/perl5/commit/064d2ad0fc89254fc8a61f7728bbd6335ee3a165 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M gv.c Log Message: ----------- make RC-stack-aware: Perl_try_amagic_un() & _bin() Note that these two functions may be called both from PP functions aware of a reference-counted stack and from wrapped (non-aware) functions, hence the RC manipulation is runtime conditional. Commit: 5c9aca77ec9a3545ccd5f71525f8419b5b0147fb https://github.com/Perl/perl5/commit/5c9aca77ec9a3545ccd5f71525f8419b5b0147fb Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M dist/threads/lib/threads.pm M dist/threads/threads.xs Log Message: ----------- make RC-stack-aware: threads.xs S_ithread_run() starts running a new top-level perl interpreter, which will have a ref-counted stack. So treat the stack as ref-counted. Commit: 600728531fc2e621795e5345e9010ac32c2df09d https://github.com/Perl/perl5/commit/600728531fc2e621795e5345e9010ac32c2df09d Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M t/cmd/for.t Log Message: ----------- make RC-stack-aware: t/cmd/for.t A test that expects to fail on a non reference-counted stack should no longer do so if the stack is fully reference-counted Commit: 982845e8e26f1f8cda840b2543d92e4bba735f6e https://github.com/Perl/perl5/commit/982845e8e26f1f8cda840b2543d92e4bba735f6e Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M op.c Log Message: ----------- make RC-stack-aware: invoking DB::postponed() Various places in core call DB::postponed(). This all happens in places where the stack will be reference-counted, so fix the callers. Commit: 28580fd1f45bed204d97ac9cb051fc6d6cc48294 https://github.com/Perl/perl5/commit/28580fd1f45bed204d97ac9cb051fc6d6cc48294 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M ext/XS-APItest/APItest.pm M ext/XS-APItest/APItest.xs M ext/XS-APItest/t/magic.t Log Message: ----------- make RC-stack-aware: ext/XS-APItest/ APItest.xs mainly consists of XS functions - which are not aware of a ref counted stack - but such functions are called via xs_wrap(), so that's ok. However, THX_pp_establish_cleanup() is actually a PP function attached to an op, rather than an XS function - so it doesn't get automatically wrapped. Update it to handle a reference-counted stack. Also, use the standard CALLRUNOPS() macro to call a runops loop rather than invoking directly; the former will allow a wrapper to be inserted. Finally, parameterise the expected ref count in t/magic.t, as it will change if the stack is ref counted. Commit: 6fa0a102361ddaeadfb6426fb4d7fd8907540cad https://github.com/Perl/perl5/commit/6fa0a102361ddaeadfb6426fb4d7fd8907540cad Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M class.c Log Message: ----------- make RC-stack-aware: class.c: pp_methstart etc Commit: 9f70b42ae33d4a5a770a0edf13fd2b9a67b38691 https://github.com/Perl/perl5/commit/9f70b42ae33d4a5a770a0edf13fd2b9a67b38691 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M av.c M inline.h M pp_hot.c Log Message: ----------- make shift(@a) leave old pointer Generally shifting is done by adjusting the start position of the array, so that AvALLOC() and AvARRAY() no longer align. Previously if the array was AvREAL(), perl NULL-ed out the old pointer at the same time, i.e. doing the equivalent of AvARRAY(av)[-1] = NULL. This commit changes it so that on PERL_RC_STACK builds, perl instead keeps the old pointer there on AvREAL() arrays too, even if it now points to a freed SV; and instead makes av_clear() and av_unshift() responsible for NULL-ing out the slot only when actually reclaiming any unused slots between AvALLOC() and AvARRAY(). The reason for this is because of the mechanism whereby @DB::args is sometimes populated by caller(), as used by Carp.pm to display function arguments in stack traces etc. For the common idiom of sub f { my $self = shift; ....; } then previously, since @_ wasn't AvREAL(), the old pointer in @_ to argument 0 was kept around and thus the hidden pointer value was still available for caller() to add to @DB::args. But since the next commit will change @_ to be AvREAL(), this will no longer work. Thus to keep the ability to debug stack traces etc (via the awful @DB::args hack), we make shift keep the pointer even for the AvREAL() case. Commit: 0b260301389aed57e613c263af117275cc74c6d8 https://github.com/Perl/perl5/commit/0b260301389aed57e613c263af117275cc74c6d8 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M cop.h M embed.fnc M embed.h M inline.h M pad.c M pp_ctl.c M pp_hot.c M pp_sort.c M proto.h M t/op/sub.t Log Message: ----------- make @_ AvREAL() on RC-stack Traditionally, the @_ AV has been marked as AvREAL_off(), but AvREIFY_on(). This commit changes it so that on PERL_RC_STACK builds, @_ is AvREAL(). This will mean that when the stack is ref-counted, the stack frame can be directly moved to @_ while maintaining the correct ref count for each argument (in the same way that !AvREAL() worked well with a non-ref-counted stack). Note that the stack will not actually be reference-counted for a few more commits yet. In the meantime, this commit allows for that by bumping up the reference count of SVs being transferred from the stack to @_. This extra step will soon be skipped. Commit: 2b95dbd6dcbab875282a6e33e36789fa6d0fa097 https://github.com/Perl/perl5/commit/2b95dbd6dcbab875282a6e33e36789fa6d0fa097 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M cop.h M embed.fnc M embed.h M inline.h M pp.h M pp_sort.c M proto.h Log Message: ----------- add switch_argstack(), push/pop_stackinfo() fns These new inline functions are supposed to be near-identical replacements for these macros: PUSHSTACKi(type) push_stackinfo(type) POPSTACK() pop_stackinfo() SWITCHSTACK(from,to) switch_argstack(to) // assumes (from == PL_curstack) except that they don't require dSP to be in scope (they operate on PL_stack_sp rather than sp) and their names more clearly specify what sort of stack they manipulate. The macros are now thin wrappers around the functions. I've kept most of their uses in core for now as they are still used in places with dSP present. Commit: bb3a68d1a6bb7459e0bc648351b02eb30edfdab8 https://github.com/Perl/perl5/commit/bb3a68d1a6bb7459e0bc648351b02eb30edfdab8 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M cop.h M embed.fnc M embed.h M inline.h M perl.c M pp_sort.c M proto.h M scope.c Log Message: ----------- allow argument stack to be AvREAL() Add a _flags() variant of new_stackinfo() which indicates whether the new stack AV should be created real or not. Modify the new push_stackinfo() function to have a similar flag. Then make the backcompat macros like PUSHSTACKi() still push a non-real stack, while functions which have been updated to use the new push_stackinfo() etc will be able get a real AV. The next commit makes use of that. This means that existing code (core and XS) which hasn't been updated to the new ref-counted stack regime can do stuff like: PUSHSTACKi(FOO); PUSHMARK(sp) XPUSHs(sv); call_sv(); where call_sv() (or rather, the runops loop it invokes) will be able to determine that it's been called from a non-RC environment and that the args on the stack aren't reference-counted. The next commit will update the runops loops etc to do exactly that. Commit: 0081067206aa2bcdf84505105659e7517d706029 https://github.com/Perl/perl5/commit/0081067206aa2bcdf84505105659e7517d706029 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M cop.h M embed.fnc M embed.h M inline.h M makedef.pl M perl.c M perl.h M pp_ctl.c M pp_hot.c M pp_pack.c M pp_sort.c M proto.h M run.c M scope.c M sv.c Log Message: ----------- Allow argument stacks to be reference-counted When built with PERL_RC_STACK, this substantial commit: * makes stacks AvREAL() by default. * adds the si_stack_nonrc_base field to the stackinfo structure. * adds the runops_wrap() wrapper It also adds rpp_obliterate_stack_to() for clearing the stack on exceptions Collectively, this allows each stack to be in one of three states: 1) AvREAL(PL_curstack) && PL_curstackinfo->si_stack_nonrc_base == 0 (the new default) all the SVs pointed to from the stack are reference-counted. 2) AvREAL(PL_curstack) && PL_curstackinfo->si_stack_nonrc_base > 0 items on the stack are reference-counted only below si_stack_nonrc_base. 3) !AvREAL(PL_curstack) (existing behaviour) no items on the stack are reference-counted. The way it generally works is that runops loops assume that all the PP functions they call out to are reference-count-aware. Where this isn't yet the case, the recently-added pp_wrap() and xs_wrap() functions assume a reference-counted stack, but use the si_stack_nonrc_base mechanism to call out to PP or XS functions which aren't aware of a reference-counted stack. Conversely, the runops_wrap() function added by this commit wraps calls to runops loops, since such loops work only with an RC stack. So if called with a non-RC or partially-RC stack, the wrapper temporarily fixes up the stack to be fully-RC, calls the real runops loop, then on return reverts the stack back to it's non-RC ways, mortalising any return values if necessary. This downgrading and upgrading by pp_wrap()/xs_wrap() and runops_wrap() allows handling of multiple nesting of when, for example, perl calls pp_entersub(), which calls XS code, which calls call_sv(), which calls a runops loop, which calls pp_entersub(), which calls XS code, and so on. For si_stack_nonrc_base, this index value marks the lowest position on the argument stack which is not reference-counted. The special (and normal) value of 0 indicates that *all* items on the stack are reference-counted. The new function rpp_obliterate_stack_to() is a bit like rpp_popfree_to(), except that it can handle stacks in all three of the states listed above. It's intended to be used when dying, throwing exceptions or exiting, since the stack could be in any state when that happens. Note that as of this commit, PERL_XXX_TMP_NORC is still defined, which means that even in the presence of AvREAL(PL_curstack) etc, the stack isn't yet actually reference counted. So with this commit, perl goes through all the motions, (calling wrappers like runops_wrap() etc), but skips actually manipulating any reference counts. There will be a few more commits, fixing up a few more things, before PERL_XXX_TMP_NORC will be removed. Commit: 45942fb750e80bbbd354aee26d409fe813955560 https://github.com/Perl/perl5/commit/45942fb750e80bbbd354aee26d409fe813955560 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- add stack extend to Perl_die_unwind() When invoked via a scalar-context eval, die_unwind() was pushing &PL_sv_undef without extending the stack first. It's been this way for many years. Spotted by visual inspection while adding rpp_() code. I could get a SEGV with the following code, but I haven't added it as a test as it's highly sensitive to exactly what size stack is initially allocated and the OS's malloc(). my $x; my @a = ($x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, $x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x,$x, scalar eval { die }, ); Commit: 04e44eb0e63a0673cfca2a9d5c85c041143fa261 https://github.com/Perl/perl5/commit/04e44eb0e63a0673cfca2a9d5c85c041143fa261 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M toke.c Log Message: ----------- make RC-stack-aware: Perl_filter_read() This function in toke.c calls out to a random C-level filter function for each source input chunk. Because that function may in turn invoke a Perl function via call_sv() - and may itself push stuff onto or off the stack for that call - present that function with a non-reference-counted stack environment for backwards compatibility. Commit: ae091657d3b5b15e20eb296e792a2230105370c6 https://github.com/Perl/perl5/commit/ae091657d3b5b15e20eb296e792a2230105370c6 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M inline.h Log Message: ----------- rpp_is_lone(): allow non-temp SV too This function indicates whether an SV on the stack is kept alive only by a single ref from the temps stack - which is often the case with stuff pushed onto the stack by PP functions. Often this means that the SV can be passed through rather then copied in places like returning from a subroutine. With a ref-counted stack, it's possible for a *non* SvTEMP SV to also be stealable, if it has a refcount of 1, since that SV is being kept alive purely by its (now refcounted) link from the stack. So this commit adds that condition too (and also simplifies the condition for efficiency). In summary: originally perl (and then this function) did: if (SvTEMP(sv) && SvREFCNT(sv) == 1) { ... steal it ... } Now this function's condition is the equivalent of #ifdef PERL_RC_STACK ( (SvREFCNT(sv) == 1) || (SvTEMP(sv) && SvREFCNT(sv) == 2)) #else (SvTEMP(sv) && SvREFCNT(sv) == 1) #endif Commit: e496c5dbc1053d38fdc9674ad18423834c9bc365 https://github.com/Perl/perl5/commit/e496c5dbc1053d38fdc9674ad18423834c9bc365 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M deb.c M embed.fnc M embed.h M proto.h Log Message: ----------- Perl_debstack(): indicate stack refcount status When displaying the stack with perl -Ds or -Dsv, put a vertical bar (|) at the boundary between the reference-counted part of the stack and the non-reference-counted part, if any. In addition with -Dsv, for each stack show its status; (real) all items on the stack are reference-counted (partial real) items below si->si_stack_nonrc_base are RCed blank no items are RCed This commit switches to invoking S_deb_stack_n() directly, rather than via the deb_stack_n() macro, as some compilers can't cope with the 'ifdef' amongst the calling args. Commit: 4f33d997195e90ec0e817bd04f2d62b0b0e9f3e4 https://github.com/Perl/perl5/commit/4f33d997195e90ec0e817bd04f2d62b0b0e9f3e4 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M perl.h Log Message: ----------- disable PERL_XXX_TMP_NORC This macro was just a temporary crutch that disabled actually adjusting the reference count of things being pushed and popped off the stack, even on PERL_RC_STACK builds. It allowed the core to be converted to support a reference-counted stack as a bunch of incremental changes. This commit turns it off, enabling the stack to become actually reference-counted on PERL_RC_STACK builds. In a few commits' time, this macro will be completely eliminated from the source code. Commit: dfd7ce929192fa66fa5d7d948ec2af1eaaad62fa https://github.com/Perl/perl5/commit/dfd7ce929192fa66fa5d7d948ec2af1eaaad62fa Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M inline.h M pp.h Log Message: ----------- stack push/pops: assert rpp_stack_is_rc() state On perls built with both -DPERL_RC_STACK and -DDEBUG_LEAKING_SCALARS, add asserts to the various old-style PUSHs() etc macros and new-style rpp_push_1() etc functions that they're operating on the right sort of stack. In particular: PUSHs() and POPs() should only be used on stacks where rpp_stack_is_rc() is false, since they don't modify the reference count of the SVs they are pushing or popping. Conversely, rpp_push_1(), rpp_popfree_1() etc should only be used on stacks where rpp_stack_is_rc() is true, since they modify the reference counts of the SVs they push and pop. On perls not compiled with PERL_RC_STACK, the rpp_ functions don't modify the reference counts, but on such builds the rpp_stack_is_rc() assertions in the rpp_ functions are disabled, so it all works out still. Commit: a1131a5068f5f0c5d71ab483f4fe794972afcd90 https://github.com/Perl/perl5/commit/a1131a5068f5f0c5d71ab483f4fe794972afcd90 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M dump.c Log Message: ----------- runops_debug(): add RC-stack assertions on PERL_RC_STACK builds, 1) assert that the stack is reference-counted (rpp_stack_is_rc()). 2) assert the any split point (si_stack_nonrc_base) isn't above the top of the stack. Commit: 8d47b4169dc4924debc5c47dba2e659aa2e3f3e8 https://github.com/Perl/perl5/commit/8d47b4169dc4924debc5c47dba2e659aa2e3f3e8 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_hot.c Log Message: ----------- pp_wrap(), xs_wrap(): extract some common code and put it in a static function, S_pp_xs_wrap_return(). Commit: 557d6e73b3ca7280871e3816d35d9eb0e8aae111 https://github.com/Perl/perl5/commit/557d6e73b3ca7280871e3816d35d9eb0e8aae111 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_hot.c Log Message: ----------- tweak pp_wrap() and xs_wrap() After bumping up the reference counts of the returned values, reset si_stack_nonrc_base earlier, so that if perl dies while decrementing the original args, all reference counts will be properly accounted for. Also in xs_wrap(), only bother doing PL_markstack_ptr[0] += nargs; in the branch where nargs > 0. Commit: c806e3fbac315913955acc0aff1c86b31ec2d918 https://github.com/Perl/perl5/commit/c806e3fbac315913955acc0aff1c86b31ec2d918 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M MANIFEST M ext/XS-APItest/APItest.xs A ext/XS-APItest/t/xsppwrap.t Log Message: ----------- add tests for XSPP_wrapped() macro This new macro allows an old-style non-reference-counted PP-style function to be used on a PERL_RC_STACK builds. It's intended for non-core PP functions which are typically in an XS file and are used to replace the PP function of an existing op, or to be attached to a custom op. It's virtually identical to the PP_wrapped() macro, except that it uses the name as-is rather than prepending Perl_ to the name Commit: 381c82c3d0c70fcbc39cf3a24302f9fd67b146c0 https://github.com/Perl/perl5/commit/381c82c3d0c70fcbc39cf3a24302f9fd67b146c0 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M ext/XS-APItest/APItest.xs Log Message: ----------- XS::APItest::clone_with_stack(); check SI RC state In this test function, push a couple of stackinfos, one with a ref-counted stack, and one without. After cloning, pop the cloned stackinfos and assert that the cloned argument stacks are also appropriately RC or non-RC. This should probably have been done properly as some new tests rather than a couple of asserts, but that would have involved a lot more effort. Commit: 7e2aa65ba111d98f8691bcfdcac89d1de3d2803c https://github.com/Perl/perl5/commit/7e2aa65ba111d98f8691bcfdcac89d1de3d2803c Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M dist/threads/threads.xs M ext/XS-APItest/t/magic.t M inline.h M op.c M perl.c M perl.h M pp_ctl.c M pp_hot.c M pp_pack.c M pp_sort.c M run.c M t/cmd/for.t M universal.c Log Message: ----------- Eliminate PERL_XXX_TMP_NORC This macro was disabled a few commits ago. Now remove all use of it from the core. It was just a temporary crutch that disabled actually adjusting the reference count of things being pushed and popped off the stack, even on PERL_RC_STACK builds. It allowed the core to be converted to support a reference-counted stack as a bunch of incremental changes. Commit: 353b2b328d8afedd393677296490915017d44f3a https://github.com/Perl/perl5/commit/353b2b328d8afedd393677296490915017d44f3a Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pod/perlguts.pod Log Message: ----------- add perlguts section on a reference-counted stack Explain what all the new rpp_foo() functions are for. Commit: 8cefaa3f325324e5c79a26c1f6f11ca5c98ff11f https://github.com/Perl/perl5/commit/8cefaa3f325324e5c79a26c1f6f11ca5c98ff11f Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- pp_goto: fix macro-in-macro compile error Turns out MSVC 1.42 doesn't like FOO(bar, #ifdef X 1 #else 0 #endif ); where FOO is a macro. Commit: a49733cd47d4cd0cc2e8a98bd04c473b24a32f94 https://github.com/Perl/perl5/commit/a49733cd47d4cd0cc2e8a98bd04c473b24a32f94 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M run.c M t/op/grep.t Log Message: ----------- Perl_runops_wrap((): don't mortalise NULLs This function was doing a delayed ref count decrement of all the SVs it had previously temporarily incremented, by mortalising each one. For efficiency it was just doing a Copy() of a block of SVs addresses from the argument stack to the TEMPs stack. However, the TEMPs stack can't cope with NULL pointers, while there are sometimes NULL pointers on the argument stack - in particular, while doing a map, any temporary holes in the stack are set to NULL on PERL_RC_STACK builds. The fix is simple - copy individual non-NULL addresses to the TEMPS stack rather than doing a block copy. Commit: 99440763f926c6f34cce86855cfb9e2c5572a346 https://github.com/Perl/perl5/commit/99440763f926c6f34cce86855cfb9e2c5572a346 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c Log Message: ----------- pp_caller(): set @DB::args earlier Copy @_ SV pointers to @DB::args earlier, before pp_caller() has the chance to create any new SVs (e.g. for the package name). This will give the next commit a chance to discover and skip any freed elements in @_. This commit makes no changes to the block of code that is moved; that's saved for the next commit. Commit: 3f16b5f6985c42a69ff68aaf8986d9ba01035bfb https://github.com/Perl/perl5/commit/3f16b5f6985c42a69ff68aaf8986d9ba01035bfb Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_ctl.c M t/op/caller.t Log Message: ----------- caller(): don't copy freed SV pointer to @DB::args When pp_caller() aliases the elements of @_ to @DB::args, it may include pointers to array elements which have been shifted away. So for example in sub f { my $self = shift; ....; croak() } it populates $DB::args[0] with the SV pointer one position below the AvARRAY() of @_. This means that the argument lists in the stack backtrace provided by croak() will include the original first arg, which is nice, but is very fragile. There is no guarantee that that SV hasn't been freed or reallocated in the meantime. This commit adds a check for each SV pointer being copied to @DB::args, which NULLs it if the SV has a refcount of 0. This means that in something like: package DB: () = caller($i); my @args = @DB::args; perl will no longer panic about copying a freed SV. Note that this commit does not address the issue of the SV having already been reallocated. In the example above, some elements of @args may contain a copy of some other random variable. However, the previous commit moved to earlier in the function, the block of code in pp_caller() which populates @DB::args, to before pp_caller() allocates new SVs for things like the package name. Which means the RC==0 test added by this commit has a better chance of spotting SVs which have been freed but not yet reallocated. In summary, @DB::args is still a horrible, fragile hack, but these two commits make it slightly less fragile in some circumstances. Commit: e96dd01c49bc9af280a3f7a0c9e9e4af289bbdf7 https://github.com/Perl/perl5/commit/e96dd01c49bc9af280a3f7a0c9e9e4af289bbdf7 Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M pp_hot.c M t/op/goto.t Log Message: ----------- when (re)creating @_, zero its elements When a sub's @_ gets abandoned (e.g. when 'goto &bar' "donates" the current @_ to the new sub), a new AV is created at pad[0] for the old sub. Perl alloc()'s the AvARRAY() buffer for the new AV, but didn't zero it. This didn't used to be a problem, because @_ AVs are created !AvREAL(), so the first thing perl did when someone tries to modify @_, is to call av_reify() on it. And it just so happens that av_reify() zeros out all the unused slots on the array first. So code like $_[1] = 1; do_something_with($_[0]) was fine. However, on PERL_RC_STACK builds, @_ is AvREAL() by default now, so av_reify() doesn't get called, so AvARRAY() can sometimes contain random wild pointers. The fix is simple: zero AvARRAY() when creating it. This was showing as a failure in Test::Trap Commit: 14148aac9cfc6e1e4c4ddb99a21ecd2ebf0934fc https://github.com/Perl/perl5/commit/14148aac9cfc6e1e4c4ddb99a21ecd2ebf0934fc Author: David Mitchell <da...@iabyn.com> Date: 2023-08-04 (Fri, 04 Aug 2023) Changed paths: M inline.h M pod/perlguts.pod M pp.h Log Message: ----------- RC_STACK asserts without DEBUG_LEAKING_SCALARS An earlier commit added some asserts to the old-style and new-style stack push/pop functions such as PUSHs() or rpp_popfree_1(). These assert that the stack is reference-counted or not as appropriate. These asserts were enabled only if perl was built with all of DEBUGGING && PERL_RC_STACK && DEBUG_LEAKING_SCALARS. This commit makes the asserts available under just DEBUGGING && PERL_RC_STACK Initially I was worried about the performance implications (you expect DEBUGGING builds to be slow, but not *too* slow), but they don't seem too bad. So by making them more likely to be enabled, they're more likely to help people spot code that needs fixing (e.g. code still doing POPs when the stack is reference counted and the function hasn't been wrapped). Compare: https://github.com/Perl/perl5/compare/f67f2243f29c...14148aac9cfc