josh kantor wrote: > I wrote a set of notes for myself when I was trying to figure out how > cython works.
I noticed one thing in it that's not quite right. You mentioned parsing list comprehensions as a possible use for states in the scanner. But that's not a good example -- it's a parser-level issue, not something the scanner would concern itself with. A better example is the way it currently handles quoted strings. I started out using regular expressions that match an entire string literal and return it as a single token. But I found I wanted to do RE-style processing on the contents of the string to handle translation of backslash sequences from Python to C conventions. What I do now is use an RE that matches the beginning of a string literal and produces a "begin string" token. Then it enters a special state where it looks for chunks of characters interspersed with backslash sequences, and returns them as separate tokens, until it finds the end of the string, whereupon it produces an "end string" token and returns to the default state. -- Greg _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
