Author: larry
Date: Sat May 17 14:37:37 2008
New Revision: 14542
Modified:
doc/trunk/design/syn/S05.pod
Log:
Clarifications to how tied longest tokens are handled under LTM
Modified: doc/trunk/design/syn/S05.pod
==============================================================================
--- doc/trunk/design/syn/S05.pod (original)
+++ doc/trunk/design/syn/S05.pod Sat May 17 14:37:37 2008
@@ -14,9 +14,9 @@
Maintainer: Patrick Michaud <[EMAIL PROTECTED]> and
Larry Wall <[EMAIL PROTECTED]>
Date: 24 Jun 2002
- Last Modified: 7 May 2008
+ Last Modified: 18 May 2008
Number: 5
- Version: 78
+ Version: 79
This document summarizes Apocalypse 5, which is about the new regex
syntax. We now try to call them I<regex> rather than "regular
@@ -2094,8 +2094,14 @@
expressions). A logical alternation using C<|> then takes two or
more of these lists and dispatches to the alternative that matches
the longest token prefix. This may or may not be the alternative
-that comes first lexically. (However, in the case of a tie between
-alternatives, the textually earlier alternative does take precedence.)
+that comes first lexically.
+
+However, if two alternatives match at the same length, the tie is
+broken by one of two methods. If the alternatives are in different
+grammars, standard MRO (method resolution order) determines which
+one to try first. If the alternatives are in the same grammar, the
+textually earlier alternative takes precedence. (If a grammar's rules
+are defined in more than one file, the results are undefined.)
This longest token prefix corresponds roughly to the notion of "token"
in other parsing systems that use a lexer, but in the case of Perl
@@ -2150,6 +2156,11 @@
Greedy quantifiers and character classes do not terminate a token pattern.
Zero-width assertions such as word boundaries are also okay.
+Because such assertions can be part of the token, the lexer engine must
+be able to recover from the failure of such an assertion and backtrack
+to the next best token candidate, which might be the same length or shorter,
+but can never be longer than the current candidate.
+
For a pattern that starts with a positive lookahead assertion,
the assertion is assumed to be more specific than the subsequent
pattern, so the lookahead's pattern is treated as the longest token;
