Re: Autocompilation/LilyPond
Hi David,
David Kastrup d...@gnu.org skribis:
l...@gnu.org (Ludovic Courtès) writes:
[...]
The order in which files get compiled does not matter; the semantics of
programs do not depend on whether code is being bytecode-interpreted or
just interpreted by (ice-9 eval).
Little things like
(define-public fancy-format
format)
(define-public (ergonomic-simple-format dest . rest)
Like ice-9's @code{format}, but without the memory consumption.
(if (string? dest)
(apply simple-format (cons #f (cons dest rest)))
(apply simple-format (cons dest rest
(define format
ergonomic-simple-format)
tend to make quite a difference depending on whether they are loaded or
not before compiling.
That one actually caused a lot of wasted effort on
URL:http://code.google.com/p/lilypond/issues/detail?id=1780
The ticket is a bit too dense for me. What’s the operational difference
between compiling and evaluating the above code?
The only reason you might want to compile files in topological order
is performance.
And macros. And redefinitions. And module hierarchy.
I think you’re talking about what should be *re*compiled when a module
has changed, right?
What I was mentioning above holds for one-shot compilation of a set of
modules, but you’re right that things are trickier when it comes to
incrementally changing part of that module set.
Thanks,
Ludo’.
Re: Autocompilation/LilyPond
Hi David, Mark, I am still around, I've not had much time for hacking lately as I've been getting sick again, and the meds tend to sap the higher brain functions. On 09/03/12 19:27, David Kastrup wrote: Mark H Weaver m...@netris.org writes: David Kastrup d...@gnu.org writes: In the long run, I think this is probably your best way forward, but admittedly it would require more work to make this transition. The main problem is that it requires such a large reorganisation of the LilyPond sources that the attempts to do it in that manner tended to consist of outdated nontrivially rearranged parts before getting through peer review successfully. LilyPond is quite a moving target. Ian Hulin has mostly worked on Guilev2 migration in that manner, and it has caused him to do lots of futile work, a major cause of frustration for him and of worry for others because the large reorganisations made the work hard to verify. So while this might be the if everything was written from scratch, it would make most sense to do it this way approach, it has proven less than fabulous as a migration strategy [...] Okay, understood. The other alternatives are workable, with the build script written in Scheme probably being the most future-proof of the remaining options. Sounds somewhat like it. It occurs to me that if the lilypond module has not already been set up, The module is '(lily). It's set up in the Lily initialization code in C++ thus: 1. main routine calls scm_boot_guile with callback to C++ routine main_with_guile. 2. main_with_guile 2.1 sets up scheme LOAD_PATH to prepend our local LilyPond root directory and the /scm subdirectory (this is so the modules declared in our kit's scheme files (like scm/display-lily.scm get autoloaded correctly). 2.2 calls ly_c_init_guile to define the '(lily) module in code and specify ly_init_ly_module as a callback. 2.2.1 ly_init_ly_callback 2.2.1.1 sets up some internal initialization functions, 2.2.1.2 dumps out some trace code, if required, about what it's doing 2.2.1.3 does the equivalent of (primitive_load_path lily.scm) from code. 2.2.2 (now we're back in ly_c_init_guile). scm_c_define_module has returned a handle to the new module, step 2.2.1.3 has loaded and evaluated all the stuff in lily.scm - including the load loop with all the component scheme file held in scm/*.scm with their problematic interactions. 2.2.3 do equivalent of (use_modules '(lily)) 2.3 (now we're back in main_with_guile). Set up local C++ constructors 2.4 set up fonts stuff 2.5 set up for multi-file compilation 2.5 process command-line options 2.6 set up handling to operate in server (jail) mode 2.7 look up and call the scheme entry-point in the '(lily) module to actually do the LilyPond compilations - there is no normal exit point from here, exiting the image is handled by scheme (exit) calls in the '(lily) code. So the lilypond module *is* normally set up when running with Guile 1.8, providing nothing breaks when lily.scm is loaded. I think it will be. then it will be completely empty and not suitable for compiling anything, so in that case you'll want to do something closer to this: (define lilypond-module (make-fresh-user-module)) (module-use! lilypond-module (resolve-interface '(guile))) (set-module-name! lilypond-module '(LILYPOND MODULE NAME)) (set-current-module lilypond-module) (for-each (lambda (base-name) (let ((scm-file-name (string-append base-name .scm)) (go-file-name (string-append base-name .go))) (compile-file scm-file-name #:output-file go-file-name #:env (current-module) #:opts %auto-compilation-options) (load scm-file-name))) LIST-OF-BASE-NAMES) BTW, if I wanted to take a look to try to get it working myself, what branch of the lilypond git repo should I look at? Believe it or not, master. There is nothing in the public repository that would have more progress regarding Guilev2 migration (it might make sense to ask Ian whether you can pull something from him). master is always kept in a state where it compiles, passes regtests, and builds the documentation, and should not trail the staging branch (where new changes get pushed) for more than a day. Unless staging does not pass the tests, in which case the automated push does not proceed. You can look for open issues regarding Guilev2 migration in URL:http://code.google.com/p/lilypond/issues/list?can=2q=guile Various issue carry patches. I think the one where Ian ran out of steam on last had been URL:http://code.google.com/p/lilypond/issues/detail?id=2026 Sorry David, this may be a bit of a Red Herring. URL:http://code.google.com/p/lilypond/issues/detail?id=1686 was what I was working on. 2026 involved trying to package the markup subsystem within the '(lily) module code so we could load it as a module. Problems are this is a) subject to the flakiness of the dynamic load order within lily.scm, *and* b) as it defines an internal interpreter,
Re: Autocompilation/LilyPond
Ian Hulin i...@hulin.org.uk writes: Hi David, Mark, I am still around, I've not had much time for hacking lately as I've been getting sick again, and the meds tend to sap the higher brain functions. I'll be taking a closer look in several days (my schedule does not really permit me doing much before the end of next week when I'll hold a talk about LilyPond). And I certainly wish you all the best for getting well. But the main points of what I wrote are not rendered invalid: you were put back repeatedly to square one by delays (whatever might have caused them), and if you had managed to do everything at once, the change would not have been easily reviewable, and would have made our change history very difficult to navigate, and would have rendered it very hard to track whether already made changes were lost. Just recently, we had a mishap with git and merging diverging branches that caused about two versions worth of work to disappear. I figured out a cure (probably not the optimal one), and about 35 already verified issues needed to get rechecked, by looking whether the changes had been properly reintroduced by my fix. This verification was achieved in several days due to the bug squad working overtime. And the patches that were checked only had to be checked to be present in the same place as before. If we would need to verify a large _reorganisation_ of the source to be up to date, this would be much much more intractable. We really need to be able to do this in smaller steps. And it is not your health that is causing a holdback, but rather the health of the project. I am glad for the information you gave here, and I'll try picking up on it after the conference. I wish you the best regarding your health, and with some luck, by the time you are interested in getting to work on LilyPond again, this rock of Sisyphos will already be on the other side, making room for more rewarding things. All the best -- David Kastrup
Re: Autocompilation/LilyPond
Hi David, Sorry for the late reply. David Kastrup d...@gnu.org skribis: Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. The order in which files get compiled does not matter; the semantics of programs do not depend on whether code is being bytecode-interpreted or just interpreted by (ice-9 eval). The only reason you might want to compile files in topological order is performance. Does that answer your question? Thanks, Ludo’.
Re: Autocompilation/LilyPond
David Kastrup d...@gnu.org writes: How is this supposed to work for compiling and installing a package? Basically, make all sudo make install The usual case will be that the user calling lilypond will not have write permission in the installed directories (and even if he did, like when calling lilypond as root, lilypond should not stomp over the installed files). So what would make all do to generate one or more .go files? I'm not very familiar with the build system, so it would be great if Ludovic or Andy could chime in here, but as I understand it, the way it's meant to work is as follows: Ideally, each file contains a single module, whose name matches its own pathname relative to an element of the GUILE_LOAD_PATH, with a (define-module ...) header at the top declaring its dependencies on other modules (using #:use-module) and its exports (using #:export and #:export-syntax). See guile-2.0.5/modules/* for many examples, and section 6.19.3 (Creating Guile Modules) for reference. Then, when you boot Lilypond, instead of using 'load' to load these files, you'd instead use 'use-modules', which would both load the .go files and import all of their exported definitions into the main Lilypond module. If you do this, then you don't have to worry about what order you use to compile or load things, and you can use the 'guild compile' command to compile each file to a .go file. See section 6.17.5 (Compiling Scheme Code) in the manual for more details. For example: $ guild compile -o foo.go foo.scm wrote `foo.go' In the long run, I think this is probably your best way forward, but admittedly it would require more work to make this transition. So now I will outline a couple of other options that require much less work. You could write a little Scheme script that gets run by the Lilypond build system to create the .go files. This script would first set the current module to whatever it will be when the Lilypond scheme files are loaded at runtime, and then compile and load the .scm files in the appropriate order, using something like this (untested) code: (set-current-module (resolve-module '(LILYPOND MODULE NAME))) (for-each (lambda (base-name) (let ((scm-file-name (string-append base-name .scm)) (go-file-name (string-append base-name .go))) (compile-file scm-file-name #:output-file go-file-name #:env (current-module) #:opts %auto-compilation-options) (load scm-file-name))) LIST-OF-BASE-NAMES) By compiling and loading each file in sequence, the macro definitions of the earlier files will be available to the later files. Alternatively, you could simply run Lilypond itself during the build process, with the XDG_CACHE_HOME environment set to something in the build directory so that the auto-compiled .go files will end up there. However, this solution seems a bit less robust to me, as I could imagine some day changing our policy of where the auto-compiled files are placed. Again, this is not my area of expertise, so hopefully Ludovic or Andy could take a look at what I've written here and let us know if I made any mistakes. Best, Mark
Re: Autocompilation/LilyPond
Mark H Weaver m...@netris.org writes: David Kastrup d...@gnu.org writes: How is this supposed to work for compiling and installing a package? Basically, make all sudo make install The usual case will be that the user calling lilypond will not have write permission in the installed directories (and even if he did, like when calling lilypond as root, lilypond should not stomp over the installed files). So what would make all do to generate one or more .go files? I'm not very familiar with the build system, so it would be great if Ludovic or Andy could chime in here, but as I understand it, the way it's meant to work is as follows: Ideally, each file contains a single module, whose name matches its own pathname relative to an element of the GUILE_LOAD_PATH, with a (define-module ...) header at the top declaring its dependencies on other modules (using #:use-module) and its exports (using #:export and #:export-syntax). See guile-2.0.5/modules/* for many examples, and section 6.19.3 (Creating Guile Modules) for reference. That's not really all that feasible: many of the dependencies are implied in the load order, and it would be rather awkward to put an explicit #:use-module for every part previous in the order. If you do this, then you don't have to worry about what order you use to compile or load things, and you can use the 'guild compile' command to compile each file to a .go file. See section 6.17.5 (Compiling Scheme Code) in the manual for more details. For example: $ guild compile -o foo.go foo.scm wrote `foo.go' In the long run, I think this is probably your best way forward, but admittedly it would require more work to make this transition. The main problem is that it requires such a large reorganisation of the LilyPond sources that the attempts to do it in that manner tended to consist of outdated nontrivially rearranged parts before getting through peer review successfully. LilyPond is quite a moving target. Ian Hulin has mostly worked on Guilev2 migration in that manner, and it has caused him to do lots of futile work, a major cause of frustration for him and of worry for others because the large reorganisations made the work hard to verify. So while this might be the if everything was written from scratch, it would make most sense to do it this way approach, it has proven less than fabulous as a migration strategy and is probably the major cause that Ian and myself probably don't have the most happy thoughts when thinking about each other. So an approach that does not require manual declaration of dependencies would certainly help. So now I will outline a couple of other options that require much less work. I'll have to look at them. -- David Kastrup
Re: Autocompilation/LilyPond
David Kastrup d...@gnu.org writes: In the long run, I think this is probably your best way forward, but admittedly it would require more work to make this transition. The main problem is that it requires such a large reorganisation of the LilyPond sources that the attempts to do it in that manner tended to consist of outdated nontrivially rearranged parts before getting through peer review successfully. LilyPond is quite a moving target. Ian Hulin has mostly worked on Guilev2 migration in that manner, and it has caused him to do lots of futile work, a major cause of frustration for him and of worry for others because the large reorganisations made the work hard to verify. So while this might be the if everything was written from scratch, it would make most sense to do it this way approach, it has proven less than fabulous as a migration strategy [...] Okay, understood. The other alternatives are workable, with the build script written in Scheme probably being the most future-proof of the remaining options. It occurs to me that if the lilypond module has not already been set up, then it will be completely empty and not suitable for compiling anything, so in that case you'll want to do something closer to this: (define lilypond-module (make-fresh-user-module)) (module-use! lilypond-module (resolve-interface '(guile))) (set-module-name! lilypond-module '(LILYPOND MODULE NAME)) (set-current-module lilypond-module) (for-each (lambda (base-name) (let ((scm-file-name (string-append base-name .scm)) (go-file-name (string-append base-name .go))) (compile-file scm-file-name #:output-file go-file-name #:env (current-module) #:opts %auto-compilation-options) (load scm-file-name))) LIST-OF-BASE-NAMES) BTW, if I wanted to take a look to try to get it working myself, what branch of the lilypond git repo should I look at? Thanks, Mark
Autocompilation/LilyPond
Hi, with the stable release 2.16 of LilyPond looming around the corner, it will become imminent soon to think about supporting Guile 2.0. Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. That would save us from trying to factor out macro dependencies into separate files (and since our markup system defines a macro for every markup function, and since the macros are needed when building markups in Scheme, this is actually rather hard to do). You might say that it is LilyPond's own fault that it has reverted a bit more to macro programming than feasible for its own good. I would not actually say that you are wrong. However, there is the problem of lifting a whole bunch of working code base under active development into the Guilev2 era, and if we could tackle design or maldesign questions mostly independently and in bite-sized chunks rather than humongous patches moving material around, this would help a lot in getting Guilev2 support on track. Suggestions? -- David Kastrup
Re: Autocompilation/LilyPond
David Kastrup d...@gnu.org writes: with the stable release 2.16 of LilyPond looming around the corner, it will become imminent soon to think about supporting Guile 2.0. Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. I'm not sure that would help much. There's a deeper problem that you should be aware of. In Guile 1.x, macro uses within procedures are not expanded until the procedure is evaluated. In Guile 2, macros are expanded as soon as the procedure is defined, even if compilation is turned off. This means that functions can only use macros that were previously defined. For example, the following works in Guile 1.8 but not in Guile 2: (define (foo x) (bar x)) (define-macro (bar x) `(quote ,x)) In Guile 2, you must put the definition of 'bar' before 'foo'. So you'll need to load your code in the right order so that macros always come before their uses. One clarification is in order. You might conclude from this that it is not possible to define mutually-recursive macros, but that's not true. For example, the following is fine: (define-macro (beep x) `(boop ,x)) (define-macro (boop x) `(quote ,x)) That's because 'beep' doesn't use 'boop', it merely produces a use of 'boop' in its expansion. 'boop' is merely part of a quoted literal within 'beep'. Regards, Mark
Re: Autocompilation/LilyPond
Mark H Weaver m...@netris.org writes: David Kastrup d...@gnu.org writes: with the stable release 2.16 of LilyPond looming around the corner, it will become imminent soon to think about supporting Guile 2.0. Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. I'm not sure that would help much. There's a deeper problem that you should be aware of. In Guile 1.x, macro uses within procedures are not expanded until the procedure is evaluated. In Guile 2, macros are expanded as soon as the procedure is defined, even if compilation is turned off. This means that functions can only use macros that were previously defined. I don't think that making this condition hold would be really hard. LilyPond has a rather carefully selected load order in several stages, so use-before-definition, whether in the context of macros or not, should be more the exception than the rule, and only require smaller rearrangements. One clarification is in order. You might conclude from this that it is not possible to define mutually-recursive macros, but that's not true. I don't think we use them, anyway. Most problems are more due to small oversights rather than maliciously clever overdesign. -- David Kastrup
Re: Autocompilation/LilyPond
David Kastrup d...@gnu.org writes: Mark H Weaver m...@netris.org writes: David Kastrup d...@gnu.org writes: with the stable release 2.16 of LilyPond looming around the corner, it will become imminent soon to think about supporting Guile 2.0. Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. I'm not sure that would help much. There's a deeper problem that you should be aware of. In Guile 1.x, macro uses within procedures are not expanded until the procedure is evaluated. In Guile 2, macros are expanded as soon as the procedure is defined, even if compilation is turned off. This means that functions can only use macros that were previously defined. I don't think that making this condition hold would be really hard. LilyPond has a rather carefully selected load order in several stages, so use-before-definition, whether in the context of macros or not, should be more the exception than the rule, and only require smaller rearrangements. Excellent! As long as you load everything in the right order, such that macros are defined before they are used, I don't see why there should be any other problems related to macros and compilation. Mark
Re: Autocompilation/LilyPond
Mark H Weaver m...@netris.org writes: David Kastrup d...@gnu.org writes: Mark H Weaver m...@netris.org writes: David Kastrup d...@gnu.org writes: with the stable release 2.16 of LilyPond looming around the corner, it will become imminent soon to think about supporting Guile 2.0. Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. I'm not sure that would help much. There's a deeper problem that you should be aware of. In Guile 1.x, macro uses within procedures are not expanded until the procedure is evaluated. In Guile 2, macros are expanded as soon as the procedure is defined, even if compilation is turned off. This means that functions can only use macros that were previously defined. I don't think that making this condition hold would be really hard. LilyPond has a rather carefully selected load order in several stages, so use-before-definition, whether in the context of macros or not, should be more the exception than the rule, and only require smaller rearrangements. Excellent! As long as you load everything in the right order, such that macros are defined before they are used, I don't see why there should be any other problems related to macros and compilation. Because the individual files are not independent from one another? That's why I wrote: Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. -- David Kastrup
Re: Autocompilation/LilyPond
David Kastrup d...@gnu.org writes: Mark H Weaver m...@netris.org writes: Excellent! As long as you load everything in the right order, such that macros are defined before they are used, I don't see why there should be any other problems related to macros and compilation. Because the individual files are not independent from one another? That's why I wrote: Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. There's no problem with them being dependent on one another. When you load a file, even with auto-compilation, the macro expander will make use of whatever macros are already bound in the current module. The rest of the compiler sees only the output of the macro expander. Try the following experiment: put (define-macro (bar x) `(quote ,x)) into foo1.scm, and (define (foo x) (bar x)) into foo2.scm, and then within a REPL type: (load foo1.scm) (load foo2.scm) and observe that everything works as it should. If you really want everything to be compiled as one unit, you can use 'include' (which acts essentially the same as #include in C), though beware that Guile is not yet smart enough to auto-recompile when one of the included files gets updated. I don't see any compelling benefit to compiling everything as one unit, but do as you wish :) Regards, Mark
