Yeah I understand the purpose of maintaining comments; I've actually written a lot of source-to-source transformations myself. The reason I ask is that I always found it cumbersome to stick non-AST information on the AST itself. For instance, "for /* this is my loop */ (/* this is my initializer */ var ii = 0 /* there it is! */;;)". I tortured myself for a long time messing with maintaining comments and whitespace when I found that the solution is actually implicit in the token start/end positions.
If you're rendering the AST to string and you've got the source tokens attached to each node, you can just look at the original source string to find the comments and whitespace. Like I said, you should check out the fibers transformation in Streamline. It does a source-to-source transformation while maintaining *all* non-semantic source data (whitespace & comments). The other solution I've seen is to maintain a "lastDocBlock" variable in your tokenizer state. Then when the parser parses a function, you just attach `lastDocBlock` to the node that way. That's a pretty simple and easy to implement solution and it handles concerns about maintaining copyright information, but it won't maintain whitespace or esoteric comments which is desirable in many situations. On Wed, Aug 29, 2012 at 8:07 PM, Scott Taylor <sc...@railsnewbie.com> wrote: > Obviously it depends what you are trying to do. Usually compiled > programming languages strip them out in the tokenizer; for source to source > translations sometimes you want to keep them in (for instance a header with > copyright notice when compressing or *all* of them in a case like mine) > > Best > > Scott > > On Aug 29, 2012, at 2:27 PM, Marcel Laverdet <mar...@laverdet.com> wrote: > > Just curious why you need the comments in the AST at all? If you've got > the start position & length of every token in the AST (much easier to do) > you implicitly have the comments as well. The "fiber" engine in Streamline ( > https://github.com/Sage/streamlinejs/blob/master/lib/fibers/transform.js) > does this with really good results. > > On Wed, Aug 29, 2012 at 8:37 AM, Mihai Călin Bazon > <mihai.ba...@gmail.com>wrote: > >> Well, the code generator doesn't yet have an option to keep comments, >> but I can add it easily; the harder part was having them in the AST, >> and that's done. >> >> What exactly are you trying to achieve? My understanding is that you >> compile Lisp to JS (cool!), do you want to be able to do the reverse >> transform? If so, perhaps a better idea is to generate a source map. >> (not sure what you need to do though, just guessing) >> >> Cheers, >> -Mihai >> >> On Wed, Aug 29, 2012 at 1:52 PM, Scott Taylor <sc...@railsnewbie.com> >> wrote: >> > Wonderful! I've been working on a project that is sort of like >> parenscript >> > - but much more of a straight javascript in lisp/scheme clothes with a >> > define-syntax macro system. >> > >> > https://github.com/smtlaissezfaire/loop >> > >> > I've been hacking on the 1x source of uglify to translate javascript >> into a >> > lispy type system (and back) - but inline comments have been a cause of >> > concern. Where is the 2.x source at this point? >> > >> > Cheers, >> > >> > Scott >> > >> > On Aug 28, 2012, at 12:56 PM, Mihai Călin Bazon wrote: >> > >> > On Tue, Aug 28, 2012 at 5:33 AM, Scott Taylor <sc...@railsnewbie.com> >> wrote: >> > >> > Very cool. What comments in the AST are you going to preserve? >> > >> > >> > The new AST is able to store all comments, and the compressor and code >> > generator will be able to keep most of them. However, I suspect that >> > in general people will only need to store copyright notices, and those >> > usually start with some special marker like "/*!". It'll be easy to >> > add a configuration option to keep such comments, as long as they're >> > not in code that's going to be dropped (for example dead code, like, >> > code that follows a return, throw, break or continue statement). >> > >> > Cheers, >> > -- >> > Mihai Bazon, >> > http://mihai.bazon.net/blog >> > >> > -- >> > Job Board: http://jobs.nodejs.org/ >> > Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > You received this message because you are subscribed to the Google >> > Groups "nodejs" group. >> > To post to this group, send email to nodejs@googlegroups.com >> > To unsubscribe from this group, send email to >> > nodejs+unsubscr...@googlegroups.com >> > For more options, visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> > >> > >> > -- >> > Job Board: http://jobs.nodejs.org/ >> > Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > You received this message because you are subscribed to the Google >> > Groups "nodejs" group. >> > To post to this group, send email to nodejs@googlegroups.com >> > To unsubscribe from this group, send email to >> > nodejs+unsubscr...@googlegroups.com >> > For more options, visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> >> >> >> -- >> Mihai Bazon, >> http://mihai.bazon.net/blog >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
