Ihor Radchenko <[email protected]> writes: > Morgan Smith <[email protected]> writes: > >> In our build system I often specify "compile-dirty" so that I don't >> trigger a world rebuild as happens with "compile". However, this is not >> strictly correct. If I where to modify a macro in a file and then >> "compile-dirty", then some of the "elc" files would have the older >> definition of the macro. > > FYI, Emacs build system struggles from the same problem. To the point > that we have a very special org--inhibit-version-check just because > Emacs developers say that it is "wontfix" and that broken compilation > for Emacs repo is considered ok and expected with stale .elc files. >
Not really in the mood at the moment to read 100+ emails from bug#62762. Maybe at some point. >> The correct way to handle this is to build out a dependency tree so that Make >> is aware of exactly what ".el" files should trigger which ".elc" >> regeneration. >> >> As far as I know, for compiling, the only thing we care about is top-level >> requires. So if we can pull out the top level requires from every file, we >> can >> build our dependency tree. I've done this using a simple regex and it seems >> to >> work great! With a two exceptions I've found. > > That's going to change. > We are going to reduce top-level requires, not increase. > The time to load Org is already too large, especially with debug Emacs builds. > Although, for the purposes for compilation, it should be possible to > leave top-level requires behind (eval-when-compile ...). > If you are going to accomplish this by splitting things into smaller files then it sounds like a dependency tree would be a great asset that could aid in parallel compilations. If you are going to do this by autoloading more things, then I believe my system should still work. Autoloads shouldn't be needed at compile time right? Only run time? >> 2. For reasons beyond my comprehension, `org-babel-do-load-languages' and >> `org-load-export-backends' manage to get run at compile time but I'm not >> really sure what to do about that. Not sure if `org-modules' gets used at >> compile time. I did not investigate. > > They are triggered by :set in defcustom. > Oh dear. I'll have to investigate this more. Thanks for pointing the way! >> However, just building a dependency tree doesn't solve all of our issues >> because of the loaddefs file. The loaddefs file depends on every single >> source >> file and all elc files depend on the loaddefs file so we are now back where >> we >> started with every change causing a world rebuild. > > And on top of that many files depend on each other to compile properly. > This is also the reason why you can't build Org in isolated processes. > (good luck trying to do something like make -j10 on Org) > >> The obvious solution is to simply not depend on loading the loaddefs file >> during compilation and only load it at runtime. This means the loaddefs file >> is still regenerated with every source change but that it doesn't cause a >> world >> rebuild. This seems to work just fine after adding in a few more `require's >> and a few `declare-function's. > > I do not buy "seems to work". We have struggled from circular > dependencies for years now. Nothing is simple in this area. > And do note that it is very deliberate that we emit a very loud warning > when loaddefs is not available. The fact that things work for you does > not mean that they will work universally. If anything, consider the case > when loaddefs is missing and Emacs will load org-loaddefs from built-in > Org. That will be a disaster. > I think I can solve this using Tup. Or at least make a lot of progress. Tup (https://gittup.org/tup/) creates a FUSE filesystem so it can detect the usage of all files. While I have successfully built and tested org using Tup and a strict dependency tree, there where some caveats. I'll have to send in a write up about this at a later date. But as far as I can tell, if we disable `org-babel-do-load-languages' and `org-load-export-backends', then there is no circular dependencies at all. Perhaps I need to do more research, but I believe we don't need to load the loaddefs file. >> While I did put a lot of work into this change, I still feel like there is >> much >> I didn't investigate. I very much expect there to be some glaring flaws in >> what I've done here. > > I do not mind improvement in this area, especially if it does not cost > too much maintenance, but I am skeptical that you can solve all the edge > cases. I'd say at least we need to make sure that out libraries do not > have circular dependencies. That alone is a very difficult ask. > Tup! >> +ifeq ($(SKIP_DEPS),) >> +include $(LISPF:%.el=./deps/%.d) >> +else >> +endif > > Note that these things are resolved when loading the makefile. So, > trying to set SKIP_DEPS as a part of the recipe is not goiing to work. > I should add a comment there explaining the situation. You are correct but I already thought of that! I only use SKIP_DEPS for recursive MAKE so the variable is set during loading. >> * lisp/Makefile (autoloads): Don't clean up these files when >> generating them. >> * mk/targets.mk (all, compile): Don't run clean in the lisp directory >> before a compilation. > > I would be ok to alter compile-dirty, but what you are doing here is > making make compile unreliable. Your patch is adding a useful > heuristics, which will fail on edge cases. I might take this path forward for now. Thank you for the suggestion!
