On Monday, 22 February 2016 19:25:50 UTC-3, Jon Zeppieri wrote: > Have you looked at the parser-tools/lex library? > https://docs.racket-lang.org/parser-tools/Lexers.html > > > On Feb 22, 2016, at 4:46 PM, Federico Ramírez <fedekil...@gmail.com> wrote: > > > > Hello everyone! I'm new to Scheme, and I need some help wrapping my head > > around it. > > > > I want to write a simple tokenizer, which is just a function that takes a > > string as input and outputs a list of "tokens", which to keep thing simple > > is a tuple of NAME and VALUE. > > > > So, for example: > > > > (tokenize "foo = 1") => '((IDENTIFIER . "foo") (EQUALS . "=") (NUMBER . 1)) > > > > I didn't want to use global variables to keep things functional, so my > > solution doesn't use any, but I wouldn't mind much using them. This is some > > code for my current implementation, I omit some code for brevity as it's > > quite repetitive for now: > > > > (define (match-identifier input) > > (regexp-match #rx"^[a-zA-Z_]+")) > > > > (define (consume-identifier input) > > (let ([match (first (match-identifier input))]) > > (cons `(IDENTIFIER . ,match) (tokenize (substring input (string- > > length match)))))) > > > > ;; some more matchers and consumers, all pretty similar > > ;; ... > > > > (define (tokenize input) > > (cond > > ((match-identifier input) (consume-identifier input)) > > ((match-equals input) (consume-equals input)) > > ((match-number input) (consume-number input)) > > (else '()))) > > > > What bothers me is that it's calling the matchers twice for each token, > > which isn't very good for performance and it's not pretty :p > > > > I want to get rid of that double-check. I figured I could use some globals > > in order to do it but I'd rather not. > > > > Does anyone have a suggestion/fix on my implementation? > > > > Thank you for your time :) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Racket Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to racket-users+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout.
Yes, but I want to write it "by-hand" :p thanks though! -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
