Branch: refs/heads/yves/re_capture
Home: https://github.com/Perl/perl5
Commit: 77d4fd3fd3c9546e888262d67a07b2b904c47437
https://github.com/Perl/perl5/commit/77d4fd3fd3c9546e888262d67a07b2b904c47437
Author: Yves Orton <demer...@gmail.com>
Date: 2023-01-11 (Wed, 11 Jan 2023)
Changed paths:
M t/test.pl
Log Message:
-----------
test.pl - add support for rtriming fresh perl output
This makes it easier to do regexp debug tests, where we don't care
about trailing whitespace.
It also fixes the line number reporting for fresh_perl_is() and
fresh_perl_like() so that it shows the actual place where the line
number is located, and it changes the relevant code to work properly
with external $Level overrides.
Commit: b0e804579a3c3403623d77fa38910b2cab129c37
https://github.com/Perl/perl5/commit/b0e804579a3c3403623d77fa38910b2cab129c37
Author: Yves Orton <demer...@gmail.com>
Date: 2023-01-11 (Wed, 11 Jan 2023)
Changed paths:
M handy.h
Log Message:
-----------
handy.h - add NewCopy() macro to combine New and Copy.
Commit: f25e8abd64fcba61e27651d5ddb2f55e41249deb
https://github.com/Perl/perl5/commit/f25e8abd64fcba61e27651d5ddb2f55e41249deb
Author: Yves Orton <demer...@gmail.com>
Date: 2023-01-11 (Wed, 11 Jan 2023)
Changed paths:
M embed.fnc
M embed.h
M mg.c
M proto.h
M regcomp.c
M regcomp_debug.c
M regcomp_internal.h
M regexec.c
M regexp.h
M t/re/pat_advanced.t
M t/re/re_tests
Log Message:
-----------
regcomp.c etc - rework branch reset so it works properly
Branch reset was hacked in without much thought about how it might interact
with other features. Over time we added named capture and recursive patterns
with GOSUB, but I guess because branch reset is somewhat esoteric we didnt
notice the accumulating issues related to it.
The main problem was my original hack used a fairly simple device to give
multiple OPEN/CLOSE opcodes the same target buffer id. When it was introduced
this was fine. When GOSUB was added later however, we overlooked at that this
broke a key part of the book-keeping for GOSUB.
A GOSUB regop needs to know where to jump to, and which close paren to stop
at. However the structure of the regexp program can change from the time the
regop is created. This means we keep track of every OPEN/CLOSE regop we
encounter during parsing, and when something is inserted into the middle of
the program we make sure to move the offsets we store for the OPEN/CLOSE data.
This is essentially keyed and scaled to the number of parens we have seen.
When branch reset is used however the number of OPEN/CLOSE regops is more than
the number of logical buffers we have seen, and we only move one of the
OPEN/CLOSE buffers that is in the branch reset. Which of course breaks things.
Another issues with branch reset is that it creates weird artifacts like this:
/(?|(?<a>a)|(?<b>b))(?&a)(?&b)/ where the (?&b) actually maps to the (?<a>a)
capture buffer because they both have the same id. Another case is that you
cannot check if $+{b} matched and $+{a} did not, because conceptually they
were the same buffer under the hood.
These bugs are now fixed. The "aliasing" of capture buffers to each other is
now done virtually, and under the hood each capture buffer is distinct. We
introduce the concept of a "logical parno" which is the user visible capture
buffer id, and keep it distinct from the true capture buffer id. Most of the
internal logic uses the "true parno" for its business, so a bunch of problems
go away, and we keep maps from logical to physical parnos, and vice versa,
along with a map that gives use the "next physical parno with the same
logical parno". Thus we can quickly skip through the physical capture buffers
to find the one that matched. This means we also have to introduce a
logical_total_parens as well, to complement the already existing total_parens.
The latter refers to the true number of capture buffers. The former represents
the logical number visible to the user.
It is helpful to consider the following table:
Logical: $1 $2 $3 $2 $3 $4 $2 $5
Physical: 1 2 3 4 5 6 7 8
Next: 0 4 5 7 0 0 0 0
Pattern: /(pre)(?|(?<a>a)(?<b>b)|(?<c>c)(?<d>d)(?<e>e)|(?<f>))(post)/
The names are mapped to physical buffers. So $+{b} will show what is in
physical buffer 3. But $3 will show whichever of buffer 3 or 5 matched.
Similarly @{^CAPTURE} will contain 5 elements, not 8. But %+ will contain all
6 named buffers.
Since the need to map these values is rare, we only store these maps when they
are needed and branch reset has been used, when they are NULL it is assumed
that physical and logical buffers are identical.
Currently the way this change is implemented will likely break plug in regexp
engines because they will be missing the new logical_total_parens field at
the very least. Given that the perl internals code is somewhat poorly
abstracted from the regexp engine, with parts of the abstraction leaking out,
I think this is acceptable. If we want to make plug in regexp engines work
properly IMO we need to add some more hooks that they need to implement than
we currently do. For instance mg.c does more work than it should. Given there
are only a few plug in regexp engines and that it is specialized work, I
think this is acceptable. We can work with the authors to refine the API
properly later.
Compare: https://github.com/Perl/perl5/compare/6965a79d0000...f25e8abd64fc
