I'm not going to modify the lexer to add the lexer attribute to tokens not processed by token functions. However, I'm pretty sure that I modified yacc to add this attribute when calling the p_error() function. I'll double check. Is there some other place besides p_error() that this is showing up?
-Dave On Feb 6, 2009, at 10:20 AM, Bart Whiteley wrote: > > The bug is still present in rc3. Did you decide against fixing it > for 3.0? > > On Tue, Feb 3, 2009 at 12:33 PM, David Beazley <[email protected]> > wrote: >> I'm going to look into this, but may not adopt the exact fix >> described. >> The lexer only sets the lexer attribute on tokens that get processed >> by functions in the lexer. Otherwise, it gets left off. This is a >> performance optimization. I'm pretty sure I can just add the >> lexer attribute onto the token in the yacc module right before the >> error >> function gets called. >> >> Cheers, >> Dave >> >> >> On Tue 03/02/09 1:39 PM , Bart Whiteley [email protected] sent: >>> >>> This bug is still present: >>> http://groups.google.com/group/ply-hack/browse_thread/thread/ed >>> 51d0c6c2a27d26/4bb10fe689313e78 >>> Other than that, 3.0rc1 looks good. >>> >>> Here is the updated patch: >>> >>> --- ply-3.0rc1/ply/lex.py.orig 2009-02-02 16:21:51.000000000 -0700 >>> +++ ply-3.0rc1/ply/lex.py 2009-02-03 11:36:02.000000000 -0700 >>> @@ -315,6 +315,7 @@ class Lexer: >>> >>> # Create a token for return >>> tok = LexToken() >>> + tok.lexer = self >>> tok.value = m.group() >>> tok.lineno = self.lineno >>> tok.lexpos = lexpos >>> @@ -335,7 +336,7 @@ class Lexer: >>> >>> # If token is processed by a function, call it >>> >>> - tok.lexer = self # Set additional attributes >>> useful in token rules >>> + # Set additional attributes useful in token rules >>> self.lexmatch = m >>> self.lexpos = lexpos >>> >>> @@ -359,6 +360,7 @@ class Lexer: >>> # No match, see if in literals >>> if lexdata[lexpos] in self.lexliterals: >>> tok = LexToken() >>> + tok.lexer = self >>> tok.value = lexdata[lexpos] >>> tok.lineno = self.lineno >>> tok.type = tok.value >>> >>> >>> On Mon, Feb 2, 2009 at 4:38 PM, David Beazley <[email protected] >>> om> wrote:> >>>> I've put up a first release candidate of PLY-3.0 >>> at:> >>>> http://www.dabeaz.com/ply/ply-3.0rc1.tar.gz> >>>> This version of PLY provides compatibility with >>> Python 2.2-3.0.> Except for the removal of a few undocumented >>> features, it> should be backwards compatible with prior >>> versions of PLY.> >>>> Why a new version number? A huge part of the >>> internal> implementation has been reorganized to >>> simplify> further maintenance, testing, and debugging. >>> These changes> were so significant a new version number seemed >>> in order.> >>>> I would appreciate feedback from current PLY >>> users. Please> check it out and let me know if it works with >>> your existing code> (or not). >>>> >>>> Cheers, >>>> Dave >>>> >>>> >>>> >>>> >>>>> >>>> >>> >>> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "ply-hack" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/ply-hack?hl=en -~----------~----~----~----~------~----~------~--~---
