I'm not quite following but I agree the dont-stop is not what you want. That is a form of concurrency control and not relevant here.
The lexing apparatus works best if your lexing has the invariant that editing a lexeme doesn't invalidate things before it. Is there another way to recast lexing for your language that has that property? If not, can you unpack the example a little bit more below? I don't get what you're saying. That, it seems like it should be okay when the editor contains: foobar and then when the user inserts a slash to produce this: foo/bar then that should continue to be okay, as the thing that calls your lexer will ask it to start over again at the "f" which should work, as long as it returns the entire "foo/bar" as a single lexeme. No? Robby On Sun, May 29, 2016 at 7:19 PM, Alexis King <lexi.lam...@gmail.com> wrote: > I’ve created a custom language with non-s-expression syntax, so I’ve > written a lexer and connected it to the DrRacket syntax highlighter. > This works pretty well, but I have a small problem: when modifying text > that would cause earlier tokens in the stream to lex differently, I am > unsure how to properly communicate that need to the highlighter. > > Specifically, this language allows forward slashes within identifiers, > like foo/bar/baz, but they cannot be leading or trailing slashes. > Therefore, something like “/foo” on its own is a lexer error, so the “/” > gets highlighted in red. > > Normally that’s ok, but when a user types a valid identifier, like > “foo/bar”, the highlighter treats “/bar” separately and attempts to lex > it in isolation, which produces an error. What I actually want is for > typing the forward slash in “foo/bar” to re-lex the entire symbol, > producing the correct token. > > As far as I can tell, there are two mechanisms for this sort of > re-lexing, both using the 3-argument get-token function with > color:text<%>: > > - Providing a backup distance, which causes re-lexing when a token > is internally modified. > > - Returning a dont-stop struct as the mode, which prevents the > highlighter from being interrupted as long as get-token continues > to return dont-stop. > > I don’t think the backup distance applies here because the token is not > being internally modified, it is being appended to. I thought that > returning dont-stop when lexing identifiers would solve my problem, but > it didn’t seem to affect anything, so I am likely misunderstanding how > it works. > > What is the solution to this sort of problem? What mechanism do I need > to use to properly communicate the need to re-lex from the beginning of > each identifier when appending text to the end? > > -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to racket-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
