How can a lexer decide a token to be get, IdentifierName or Identifier ?

Hi, everyone I noticed both “IdentifierName” and “Identifier” appeared in syntactical grammar of ES5. It seems a lexer will not be able to decide a token to be an IdentifierName or Identifier during lexing phase. A similar problem is “get”. get is not a keyword but is used like a keyword in

Re: How can a lexer decide a token to be get, IdentifierName or Identifier ?

SpiderMonkey, at least, uses feed-forward from parser to lexer, a one-bit mode flag: http://mxr.mozilla.org/mozilla-central/search?string=TSF_KEYWORD_IS_NAME /be 程劭非 wrote: Hi, everyone I noticed both “IdentifierName” and “Identifier” appeared in syntactical grammar of ES5. It seems a lexer

Re: How can a lexer decide a token to be get, IdentifierName or Identifier ?

If I was going to move something to the syntactic grammar it would probably be the current definition of Identifier Identifier : IdentifierName but not ReservedWord You might then lex all IdentifierNames (including ReservedWord) as IdentifierName tokens and treat all occurrences of