Thanks Dan and Alex! I understand this a lot better now. Thanks for the explanation about how the automatic tracking of disappearing forms works too Dan, I was curious about that. I don't fully understand that but I'll sit down with the macro debugger and experiment, since I'm keen to see how the debugger works too.
That paper is also very interesting - I've skimmed it but I'll definitely come back and read it carefully. Unfortunately macroexpansion in Clojure is much more primitive than in Racket, so these sorts of things are not possible. It's basically a Common Lisp-style defmacro system, so there's no tracking of previous forms or anything like that. Most (but not all) Clojure forms *do* support metadata which could be used in a similar way to syntax properties, but all the bookkeeping is manual and Clojure macros are notorious for throwing it away. For example, a simple macro which just expands its body using syntax-quote will lose all the metadata that was present on the original form by default. The end result is that for any non-trivial macro to work with Cursive it would have to be written with that support in mind, and sadly Cursive is not as ubiquitous as DrRacket. I do have a couple of followup questions. Is performance or memory use ever a problem when doing this repeatedly in an interactive setting? When the user is editing a really large file (I assume Racket has some!), is it responsive enough? Macroexpansion can also result in a lot of code, especially since it seems like the Racket expander is essentially tracking all previous forms through the expansion process. Is memory use ever an issue? Also, does DrRacket do any caching, for example of data structures calculated from the expansions of modules? Say I use your `(require (for-syntax syntax/parse syntax/transformer))` in two separate files - is DrRacket able to re-use the analysis of the required modules across those two files? Oh, and re: the structural editing, one thing which is new in Cursive is the new version of parinfer, which infers parentheses based on indentation. It's a really nice solution for maintaining parens, especially for new users who don't want to learn 30 paredit commands. You can see a demo of it here <https://shaunlebron.github.io/parinfer/demo> - try indenting and outdenting forms to see how it works. It has some tricky corners, but mostly it works very intuitively. Thanks, Colin On 15 May 2018 at 17:04, Daniel Feltey <[email protected]> wrote: > Hi Colin, > > Cursive looks like a really cool project. DrRacket would definitely > benefit from better support for the sort of structural editing that > Cursive enables. > > > My understanding is that using macroexpansion in the way that > > DrRacket does requires a fairly deep integration between the > > macroexpander, the macros themselves and the IDE - is that a > > fair statement? > > I don't think they are as deeply integrated as they might appear > to be. DrRacket and the macro expander have certainly evolved > together over 25 years of development and the macro expander has > had to support more features to enable some of the tools that we > use everyday in DrRacket, but today the two are developed mostly > independently. > > DrRacket supports background expansion of Racket programs and > tools like Check Syntax by expanding programs on a separate > core (using places[1]) and by analyzing the resulting fully > expanded program. Check Syntax in particular looks at the > identifiers that show up in a fully expanded program and builds > data structures that associate the definitions and uses of > variables that will later be used to draw the binding arrows that > you see when you hover over identifiers in DrRacket. As Alex > mentioned, Check Syntax also uses the `mouse-over-tooltips` > syntax property in order to show tool tips in DrRacket. Check > syntax also uses variable binding structure for a a number of > other purposes including showing relevant documentation based on > the binding of an identifier, opening the file where a given > identifier is defined, and supporting simple refactorings such as > variable renaming to name few. > > > How well does all this work for macros which aren't written > > with DrRacket in mind? > > For the most part, things just work. That is if you implement a > macro that introduces a new identifier then DrRacket, usually, is > able to associate the uses of that identifier with the form that > introduces it. When this doesn't work, DrRacket provides hooks > for macro implementers to cooperate with Check Syntax via syntax > properties like `disappeared-use` and `disappeared-binding` which > Check Syntax uses to draw binding arrows for identifiers that may > not exist in the fully expanded program. As Alex explained, you > can also use the `sub-range-binders` syntax property to show the > connections between identifiers constructed only partially from > those that appear in a macro invocation as the `struct` macro > does. > > Here is a small macro that demonstrates that most of the time > things just work in DrRacket: > > ``` > #lang racket > > (require (for-syntax syntax/parse syntax/transformer)) > > (define-syntax (my-letrec stx) > (syntax-parse stx > [(my-letrec ([x expr]) body) > #`(let ([temp (box #f)]) > (let-syntax ([x (make-variable-like-transformer > #'(unbox temp) > #'(lambda (v) (set-box! temp v)))]) > (set-box! temp expr) > body))])) > > > (my-letrec ([fact (λ (x) (if (zero? x) 1 (* x (fact (sub1 x)))))]) > (fact 5)) > ``` > > When this program is fully expanded there are no more references > to the identifier `fact` in the program, but DrRacket is still > able to correctly draw arrows between the binding of fact and its > uses in the body of the `my-letrec` form. What happens in this > program is that the references to `fact` end up stored in the > `origin` syntax property wherever the original program referred > to `fact`. The macro expander stores the original syntax that a > piece of syntax expanded from in this field, and Check Syntax > uses this information to reconstruct the binding structure of the > original program without needing any help from the developed of > the macro. There are cases where this doesn't quite work out for > more complicated macros, and in those cases manually attaching > the `disappeared-use` and `disappeared-binding` properties allows > programmers to specify the correct binding structure in a way > that Check Syntax can use. > > > And finally, of course - is there some documentation about all > > this I could look at? Is this all implemented with syntax > > object properties? > > Check Syntax is described a little bit here[2] which describes > the set of syntax properties that Check Syntax uses to draw > arrows. Currently, syntax properties are the best tool we have > for communicating between programs and tools that analyze them, > and to the best of my knowledge most of this style of tool use > syntax properties to pass along static program information to be > analyzed. > > This strategy enables a lot of cool features, however, we wrote a > paper[3] a couple years ago that shows how syntax properties can > be used to implement simple refactorings for languages > implemented in Racket, and David Christiansen's very cool Todo > List[4] tool uses a similar technique. > > I hope this answers some of your questions, but feel free to > reach out if you have any other questions. > > Dan Feltey > > [1]: http://docs.racket-lang.org/reference/places.html?q=places > [2]: http://docs.racket-lang.org/tools/Check_Syntax.html?q=check%20syntax > [3]: http://eecs.northwestern.edu/u/daniel.feltey/papers/ > languages-the-racket-way.pdf > [4]: https://docs.racket-lang.org/todo-list/index.html > > On Mon, May 14, 2018 at 10:31 AM, Alex Knauth <[email protected]> > wrote: > >> >> >> On May 14, 2018, at 12:42 AM, Colin Fleming <[email protected]> >> wrote: >> >> Hi all, >> >> I work on Cursive <https://cursive-ide.com/>, which is an IDE for >> Clojure code based on IntelliJ. I've spoken to several of you at various >> conferences over the years. >> >> I'm interested in understanding better how the macroexpansion to support >> editor functionality in DrRacket works. By contrast, Cursive doesn't expand >> macros during editing and relies on an extension API to teach it about >> macro semantics. It's not ideal, but it does have some advantages - it's >> safe and it's fast, and you can add support for some really crazy things >> that macros do. However currently it requires me to add support for popular >> macros, although I do have plans to open source that part so that users >> will be able to add support for either third-party macros that they use, or >> macros that they've developed themselves. >> >> My understanding is that using macroexpansion in the way that DrRacket >> does requires a fairly deep integration between the macroexpander, the >> macros themselves and the IDE - is that a fair statement? >> >> I guess that things like source location for forms which are carried from >> the unexpanded source to the macroexpansion are more or less automatic. How >> does this work for synthesised forms which are either composed from forms >> in the original source, or don't exist in the original source at all? For >> example, playing around with DrRacket I can see that after (struct node >> ([left : Tree] [right : Tree])), hovering over node-left or node-right >> will indicate that the node part comes from the struct definition, and the >> left/right part will come from the fields. Are there annotations specifying >> partial ranges within symbols to allow this? >> >> >> Since `struct` creates a new name that wasn't in the source, but built >> from parts of the source, it uses the syntax property `'sub-range-binders` >> to communicate this to DrRacket [1]. >> >> However, if a macro doesn't "make up" names like struct does, it doesn't >> need to worry about this. >> >> I know there are features for tagging forms with information that will >> appear in tooltips and the like, e.g. Typed Racket showing type information >> in the editor. Are there other facilities for communicating with the user? >> >> >> These tooltips are controlled by the syntax property >> `'mouse-over-tooltips` [1]. >> >> How well does all this work for macros which aren't written with DrRacket >> in mind? >> >> >> It depends on whether they make up or introduce names that weren't >> originally there in the source. If a macro only produces definitions or >> expressions using identifiers it was given as an input, the macro generally >> doesn't need to be written with DrRacket in mind. >> >> And finally, of course - is there some documentation about all this I >> could look at? Is this all implemented with syntax object properties >> <https://docs.racket-lang.org/reference/stxprops.html>? >> >> >> Yes, the advanced features that work for macros that make up names or >> display tooltips are communicated to DrRacket through syntax properties >> like `'sub-range-binders` and `'mouse-over-tooltips`. These properties are >> documented here: >> >> [1]: https://docs.racket-lang.org/tools/Check_Syntax.html#(p >> art._.Syntax_.Properties_that_.Check_.Syntax_.Looks_.For) >> >> Thanks for any and all help! >> >> Cheers, >> Colin >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Racket Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/racket-dev/87D736B0-C013-48CD-9222-0AA2B41C1420%40knauth.org >> <https://groups.google.com/d/msgid/racket-dev/87D736B0-C013-48CD-9222-0AA2B41C1420%40knauth.org?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Racket Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/racket-dev/CA%2BnS5KaGWDfYJJZsG3OcUonJQCk5nU > 43m2wY%2BUxY3VnF%3DgbN1g%40mail.gmail.com > <https://groups.google.com/d/msgid/racket-dev/CA%2BnS5KaGWDfYJJZsG3OcUonJQCk5nU43m2wY%2BUxY3VnF%3DgbN1g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Racket Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CACK23AdQfn_MoPqomNYH6y_h4%2B%3DXtdL6QiGAKnPx2%2BGUv5EBmA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
