This and other RFCs are available on the web at
http://dev.perl.org/rfc/
=head1 TITLE
Regex: Support for incremental pattern matching
=head1 VERSION
Maintainer: Damian Conway <[EMAIL PROTECTED]>
Date: 11 August 2000
Last Modified: 24 August 2000
Version: 2
Mailing List: [EMAIL PROTECTED]
Number: 93
=head1 ABSTRACT
This RFC proposes that, in addition to strings, subroutine references may be
bound (with =~ or !~ or implicitly) to a regular expression.
=head1 CHANGES
Version 2 of this RFC redirects discussion of this topic to
[EMAIL PROTECTED]
=head1 DESCRIPTION
It is proposed that the Perl 6 regular expression engine be extended to
allow a regex to match against an incremental character source, rather than
only against a fixed string.
Specifically, it is proposed that a subroutine reference could be bound
to a regular expression:
sub {...} =~ /pattern/;
As the regular expression is matched, it would make calls to the subroutine
to request additional characters to match, or (after it has matched) to
return any unused characters.
When the regex engine requires additional characters to match, the
subroutine would be called with a single argument, and would be expected
to return a character string containing the extra characters. The single
argument would specify how many characters should be returned (typically
this would be 1, unless internal analysis by the regex engine can deduce
that more than one character will be required). Returning fewer than the
requested number of characters would typically indicate a premature
end-of-string and would probably trigger backtracking and/or failure to
match.
When the match is finished, the subroutine would be called one final time,
and passed two arguments: a string containing the "unused" characters (what
would be $' for a fixed string), and a flag set to 1. The subroutine
could use this call to push-back (or cache) unused data. In the case of
a failure to match (or success of the !~ operator), every character requested
during the match would be sent back.
A typical structure for a subroutine against which a regex was matched
would therefore be:
sub s {
if ($_[1]) { # "putback unused data" request
recache($_[0]);
}
else { # "send more data" request
return get_chars(max=>$_[0])
}
}
=head2 Examples
The most obvious example would be matching against an input stream:
sub { $_[1] ? $fh->pushback($_[0]) : $fh->getn($_[0]) } =~ /pat/;
which could also be written:
^1 ? $fh->pushback(^0) : $fh->getn(^0) =~ /pat/;
Of course, it would often be useful to have a subroutine that returns a
closure on a particular filehandle:
sub fhmatch { ^1 ? $_[0]->pushback(^0) : $_[0]->getn(^0) }
fhmatch($fh) =~ /pat/
fhmatch(\*STDIN) =~ /pat/
# etc.
In fact, this might be so commonly useful that matching against a
file handle should be made to work directly. That is:
$fh =~ /pat/
\*STDIN =~ /pat/
One could then do interactive lexing cleanly:
until (eof $fh) {
switch ($fh) {
/^\s*/; # skip leading whitespace
case /^(lexeme1)/ { push @tokens, $1=>LEX1 }
case /^(lexeme2)/ { interact_somehow }
case /^(lexeme3)/ { push @tokens, $1=>LEX3 }
# etc.
}
}
Note the use of the proposed PAIR data structure to store tokens
in the above example.
Because the character source is a subroutine, one could also match against
data coming out of a socket:
my $cache = "";
sub matching_socks {
if ($_[1]) { $cache .= $_[0]; return } # putback
if (length($cache) < $_[0]) { # not enough cached
my $extra; # so get some more
recv(SOCKET, $extra, $_[0]-length($cache));
$cache .= $extra;
}
return substr($cache,0,$_[0],"");
}
switch (\&matching_socks) {
case /pat1/ { action1() }
case /pat2/ { action1() }
case /pat3/ { action1() }
#etc.
}
or any other source:
sub mega_ape {
return join "", map {['a'..'z',(' ')x6]->[rand 32]} (1..$_[0])
unless $_[1]
}
\&mega_ape =~ /Now is the Winter of our discontent.../i;
print "Art after ", length($`), "chars\n";
=head1 IMPLEMENTATION
Dammit, Jim, I'm a doctor, not an magician!
Probably needs to be integrated with IO disciplines too.
=head1 REFERENCES
RFC 22 : Builtin switch statement
RFC 23 : Higher order functions
RFC 84 : Replace => (stringifying comma) with => (pair constructor)
