Alan Manuel Gloria: > Monads: keeping readable Lisp obscure since the late 90's!
:-) :-) :-) :-) Probably more than a little true...! > >> So using a separate tokenizer is clearer IMO. > >> > >> The only drawback is that we now need to use SAME. ... > Mostly, it has to do with the reader having to consume only the data > it needs, and no more. I solved that problem in a different way, though either would work. I vaguely remember there being an annoying problem when I had a "SAME" token, but I can't remember now what that was. I then realized I didn't need it anyway, and never looked back :-). > Hmm, the code structure looks like the lexer calls "emit()", so it > looks as if the parser is CPS'ed inside a stateful emit() function. > Compare to bison where the lexer returns a token type and token value, > so that the lexer is the stateful function. Or are you using a > separate thread for the Java ANTLR lexer, with emit() being a channel > to the parser thread that calls a lexer function that just fetches > from the channel? Nothing that sophisticated. ANTLR runs the lexer to completion and saves every token in a CommonTokenStream (basically a buffer), and then calls the parser. That design makes it generally useless for interactive use, but I was using the tool to prototype and rigorously check the grammar, so was adequate for its purpose. > > I'm very leary of removing the hspace's from the parsing spec. > > SRFI-49 did that; the resulting BNF was certainly simpler, but it > > made it *much* more difficult to *correctly* implement the spec. > > If that all moves into the tokenizer, I'm concerned that > > it may not be obvious where it happens, > > especially for people implementing it using traditional > > recursive descent parsing approaches. > > I want people to be able to implement code that is "obviously correct"; > > if the spec is rigged so that implementation is mostly 1-to-1 it'll > > be easier to accept. > > hspace is significant in these cases: > > 1. Indentation > 2. After abbreviation sequences ' ` , ,@ and their Scheme-only syntax > variants. > 3. Must specifically be ignored after GROUP_SPLICE in order to handle > a top-level "foo bar \\ nitz kuu" sequence correctly. > 4. Must specifically be ignored after RESTART_BEGIN in order to > handle "let <* x v \\ y v2 *>" style. There's at least one other case: it's an error to NOT have hspace between n-expressions on the same line. Thus: (x)a is not legal. > The tokenizer can also be specced (and implemented!!) as a > recursive-descent tokenizer that calls an (emit x) function; this > gives significant clarity, since we don't have to mention an > indent-stack - the Scheme stack serves that purpose. In the > implementation, we just use call/cc in the emit function to suspend > execution. I don't want to spec it that way since the readable > project wants implementability across multiple Lisps, and most Lisps > don't have call/cc. Not even all "Scheme" implementations have a > call/cc, and many have an inefficient implementation (old Guile > versions for example). But that style can be done, and be equivalent > to the current specifications. I *really* don't want call/cc in the implementation (with perhaps an exception in the error handler). I hope to get people to use this in non-Scheme "Schemes" and even in completely different Lisps, which don't have such a thing. > Okay, I'll try that later. For now, I want to try my approach first, > which needs to simplify the BNF to remove some things I don't need and > replace comment_eol that isn't followed by an INDENT with SAME. Okay. Be sure it's separate; I'm already writing Scheme code to match the BNF!! > Incidentally, it might be more useful, in the error case, to have the > parser consume lines until it finds a completely empty line, or a line > without indentation. I agree, that sounds like a good approach. I think we should give people flexibility on how to handle errors, but we should at least strive to deal with errors in a reasonable-enough way in our sample implementation. ---- David A. Wheeler ------------------------------------------------------------------------------ Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnnow-d2d _______________________________________________ Readable-discuss mailing list Readable-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/readable-discuss
