I’ve created a custom language with non-s-expression syntax, so I’ve
written a lexer and connected it to the DrRacket syntax highlighter.
This works pretty well, but I have a small problem: when modifying text
that would cause earlier tokens in the stream to lex differently, I am
unsure how to properly communicate that need to the highlighter.
Specifically, this language allows forward slashes within identifiers,
like foo/bar/baz, but they cannot be leading or trailing slashes.
Therefore, something like “/foo” on its own is a lexer error, so the “/”
gets highlighted in red.
Normally that’s ok, but when a user types a valid identifier, like
“foo/bar”, the highlighter treats “/bar” separately and attempts to lex
it in isolation, which produces an error. What I actually want is for
typing the forward slash in “foo/bar” to re-lex the entire symbol,
producing the correct token.
As far as I can tell, there are two mechanisms for this sort of
re-lexing, both using the 3-argument get-token function with
color:text<%>:
- Providing a backup distance, which causes re-lexing when a token
is internally modified.
- Returning a dont-stop struct as the mode, which prevents the
highlighter from being interrupted as long as get-token continues
to return dont-stop.
I don’t think the backup distance applies here because the token is not
being internally modified, it is being appended to. I thought that
returning dont-stop when lexing identifiers would solve my problem, but
it didn’t seem to affect anything, so I am likely misunderstanding how
it works.
What is the solution to this sort of problem? What mechanism do I need
to use to properly communicate the need to re-lex from the beginning of
each identifier when appending text to the end?
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
