I do understand why any-char wouldn't work. Thanks! I don't really understand why :seq didn't work, maybe because I want to require one or more chars (thus :+)? Anyway, it works now! Thanks a lot! You're a legend.
On Tuesday, 24 September 2019 00:57:17 UTC-5, Matthew Butterick wrote: > > > On 23 Sep 19, at 9:10 PM, Cistian Alejandro Alvarado Vázquez < > [email protected] <javascript:>> wrote: > > > Can anyone help me understand how my tokenizer should work? I'm truly > truly lost and reading the documentation has only added to my confusion! > > > What your parser parses is a sequence of tokens. If you don't pass the > parser all the tokens that the grammar expects, then the parse can never > succeed. > > For instance, the problem with this lexer: > > (lexer > ["select" lexeme] > [whitespace (token lexeme #:skip? #t)] > [any-char (next-token)])) > > Is that it only produces one token, "select". Since your parser uses more > than just the token "select": > > #lang brag > select : /"select" fields /"from" source joins* filters* > fields : @field (/"," @field)* > field : WORD > source : WORD > joins : join* > join : "join" source "on" "(" condition ")" > filters : "where" condition ("and" | "or" condition)* > condition : field | INTEGER "=" field | INTEGER > > The parse can never succeed. > > Likewise, your revised lexer: > > (lexer > [whitespace (token lexeme #:skip? #t)] > ["select" lexeme] > [(:seq alphabetic) (token 'WORD lexeme)])) > > Will only emit two kinds of tokens: "select" and a WORD token containing a > single letter as its lexeme. (Do you see why?) Also not what you want. > > I can't write your whole tokenizer. But as a start, you probably want to > match each of your reserved keywords as a whole token, e.g. — > > [(:or "select" "from" "join" "on" "where" "and") lexeme] > > If you want other sequences of characters to be captured as WORD tokens, > your pattern needs to be a quantified pattern: > > [(:+ alphabetic) (token 'WORD lexeme)] > > > > -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/a6487add-3104-4240-9989-7e2e5125470b%40googlegroups.com.
