On Thu, 10 Aug 2000, Dan Sugalski wrote:
> With each box being replaceable, and the process being freezable between
> boxes. The lexer and parser probably ought to be separated, thinking about
> it.
Not necessarily. It might be dangerous to change what the lexer think
is a token. You might be able to get away with having the parser just
interpret multiple tokens as needed. (Which means the lexer, if it
doesn't do it already, needs to return some indicator of whitespace.
It could be that extraneous whitespace is itself a token, with the
standard resolution being ignoring it.
@foo = @bar >|< @baz;
The lexer would return something like.....
@
foo
{space}
=
{space}
@
bar
{space}
>
|
<
{space}
@
baz
;
Which the parser could then interpret:
@foo, okay.
Assignment, got it.
@bar, okay.
Greater than, okay.
Pipe? Hmmm, no space before it, back up one, and make it >|
Less than. After a >|? No space before it, so let's make it all >|<.
@baz. Okay, I see I have >|< in my user dictionary, so I'll allow this.
EOS.
It's ugly, but wouldn't require hooks into the lexer proper.
--
Bryan C. Warnock
([EMAIL PROTECTED])
