Re: Proposal to turn off AOT in clojure-build-system
Hello, >Correctly handling the ABI concerns — which Guix currently does >not do — would result in a combinatorial explosion of Clojure >packages should multiple versions of Clojure ever be available in >Guix at the same time. I think this is partly true and also a problem of other languages in Guix too but it is solvable IMO, one does need indeed to duplicate each and every package for the specific Clojure version used but that can be solved by writing a procedure with PACKAGE-MAPPING that changes `#:clojure' argument recursively for the package inputs, just like it is done with PACKAGE-WITH-PYTHON2, thus using one line of code to change a lot of packages. I don't know much of Clojure to have an opinion on it but just saying how that problem is and should be solved because it could also affect packages for a potential new Rust build system since ABI compatibility is a problem too.
Re: Proposal to turn off AOT in clojure-build-system
Hello,
I’ve been following along with this discussion, as well as a
discussion on Clojureverse, and thought it might be helpful to
pull together some threads and design decisions around Clojure’s
behavior.
Clojure is designed to ship libraries as source artifacts, not
bytecode ("pretty much all other Clojure libraries ... are all
source code by design[1]."; "Clojure is ... a source-first
language[2]"), and the view of the community is that shipping AOT
artifacts "is an anti-pattern[1]." Clojure library JARs are more
akin to source tarballs than binaries. The original design and
intent of Clojure’s AOT compiler is to compile "just a few
things... for the interop case" or "Everything... For the
'Application delivery', 'Syntax check', and 'reflection warnings'
cases[3]."
Clojure’s compiler is transitive and "does not support separate
compilation"[3], meaning when a namespace is compiled, anything it
uses is compiled and emitted with it. This is the crux of why
mixing AOT and non-AOT code is troublesome: it causes dependency
diamonds, where the AOT’d library contains a duplicate, older
version of code used elsewhere in the project.
The Clojure reference on compiling[4] gives some reasons you might
want to AOT: "To deliver your application without source," "To
speed up application startup," "To generate named classes for use
by Java," "To create an application that does not need runtime
bytecode generation and custom classloaders." Note that there’s
no mention of compiling libraries for any reason; only
applications.
When AOT is used "for the interop case," it’s typical to AOT only
those namespaces[5], not the entire library.
Shipping AOT-compiled Clojure libraries has caused real and very
weird and hard-to-debug problems in the past:
https://clojure.atlassian.net/browse/CLJ-1886?focusedCommentId=15290
https://github.com/clj-commons/byte-streams/issues/68 and
https://clojure.atlassian.net/browse/CLJ-1741
Clojure doesn’t have guarantees around ABI stability[6][7]. To
date, most ABI changes have been additive, but there are no
guarantees that the ABI will be compatible from any one version of
Clojure to any other. The understanding of the Clojure community
is that the design of the current compiler can’t offer a stable
ABI[8] at all. Because nobody in the Clojure community AOTs
intermediate (that is, library) code, this hasn’t been a problem
and is unlikely to change.
"Clojure tries very hard to provide source compatibility but not
bytecode compatibility across versions[9]."
Correctly handling the ABI concerns — which Guix currently does
not do — would result in a combinatorial explosion of Clojure
packages should multiple versions of Clojure ever be available in
Guix at the same time. For example, if someone wanted to package
Clojure 1.12.0-alpha9, you’d need to duplicate every package
taking Clojure as an input so they use the correct version. While
ABI breakage has been rare thus far, it seems likely that it’ll
occur at some point; perhaps if Clojure reaches version 2.0.0. If
Guix disables AOT for Clojure libraries, we have source
compatibility, and the AOT/ABI problems are moot.
Clojure’s compiler is non-deterministic[10]: the same compiler can
will produce different bytecode for the same input across multiple
runs. I’m not sure if this is a problem for Guix at this point in
time, but it seems out of line with Guix expectations for
compilation generally.
Opinions follow:
If we’re taking votes, mine is to *not* AOT Clojure libraries,
both for the technical reasons laid out in, and also for the
social reason of not violating the principle of least surprise. I
understand that Guix and Clojure have very different approaches,
and some balance must be struck. However, the lack of ABI
guarantees, the compiler’s behavior, the promise of source
compatibility, and matching the expectation of the audience these
tools are meant for all convince me that disabling AOT is the
right course here.
AOT’ing Clojure applications (which means, more or less, "the
Clojure tooling") is desirable, and should be maintained.
— Ian
[1]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/8
[2]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[3]: https://clojure.org/reference/compilation
[4]:
https://archive.clojure.org/design-wiki/display/design/Transitive%2BAOT%2BCompilation.html
[5]: https://clojure.org/guides/deps_and_cli#aot_compilation
[6]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[7]:
https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
[8]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/4
[9]:
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/18
Re: Proposal to turn off AOT in clojure-build-system
Steve George writes: > Hi, > > Guix's clojure-build-system turns on AOT compilation by default. I would like > to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, > so we should change the default. > > This has been discussed previously. In #56604 r0man noted that AOT > compilation should not be on by default [0], Reilly makes the same point in > #53765 [1]. Hello, we have both source and binary packages for Common Lisp (eg: cl-ppcre and sbcl-ppcre), the source packages are for development and binary packages for build applications. I think we can do the same for Clojure, and yes package them in source package (without AOT) by default, and add binary packages when needed by actual applications. I'm not using clojure myself, but the use cases for development and applications distributation are both important for Guix, and manage development environments better is a big sell point.
Re: Proposal to turn off AOT in clojure-build-system
Hi, I don't think I'm making any progress convincing you, and I'm not enjoying the interaction so I'm going to take a few days off from this thread. I've tried to ask the Clojure community for a definitive expression of what they think Linux distributions should do with byte-compiled libraries. We'll see if there's a response, this is a public forum so feel free to chime in and express clearly your perpective: https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/1 Thanks, Steve / Futurile On 22 Feb, Maxime Devos wrote: > >> [...] > >> > But, I would like to draw attention to this thread on Clojureverse as > >> > the best source I could find: > >> >Alex Miller is the main community manager for Clojure, and is a > >> >maintainer of the core libraries, so his perspective is key. He notes > >> >that, AOT code is tied to *specific versions of Clojure*: > >> > > >> > "AOT'ed code is that it is inherently the product of a particular > >> > version of tthe Clojure compiler ... I would recommend NOT AOT compiling > >> > libraries" [4] > >> > >> This reasoning does not follow – yes, it is tied to the Clojure version, > >> so what? Guix automatically rebuilds dependents when the dependency (in > >> this case, the Clojure compiler) changes. > (...) > > >I think this preceding sentence is the heart of different assumptions > >between us. > > >The Clojure packages we haave are for developers writing applications > >(libraries and tools). The ecosystem has very few end-user applications [0]. > >As a Clojure developer I can use the Clojure tools from Guix, and a few > >libraries. We (and all the other distributions) have a miniscule portion of > >the Clojure/Java library universe [1]. This leads to the following usage > >scenarios: > > >1. A developer installs Clojure from Guix, and uses libraries from outside > >Guix. > They can install the JVM/Clojure and some common tools (like clj-tools-cli). > They will use libraries from elsewhere, including their own. AOT compilation > is a problem because of the issue of mixed AOT and non-AOT. > > >2. A developer installs a Clojure from outside Guix, uses libraries from > >inside Guix > This will cause problems because the Guix Clojure libraries will have been > AOT'd by a different version of the compiler. It's also fairly common to > install/use parallel versions of Clojure/jvm due to different deployment > needs, this is likely to cause difficult to find bugs. > > My answer to (1) and (2) is: > > (a) About “install Clojure from outside Guix + use libraries inside Guix”: > > If these libraries are AOT: > > Don’t do that, then. If you mix-and-match binaries (in this case, .class > files) different distributions, you are free to do so, but when (not if, > when) things break, you get to keep the pieces. > > If these libraries are just the .clj files (not AOT) (which as I understand > it is the standard situation): doesn’t seem a problem to me (see point (b)). > > Adding source from other places is one thing (probably ok), adding binaries > is another (probably not ok, especially if unstable ABIs (see Clojure > compiler) and mismatched versions (see hypotheses of 2.) are involved. > > (b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source > on this? Besides Clojure supposedly, I haven’t ever heard of such problems > for any language – for example, there is no such issue with Guile and AFAIK > not for Python. I haven’t heard of any such issues for the Common Lisp > implementations either (though I haven’t checked), so this doesn’t seem like > a “Clojure doesn’t do hygienic macros” issue. > > (c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want > to assist in murdering the climate (or its a situation where total cost of > non-AOT is lower than AOT), they can unfortunately (*) simply do so applying > a recursive transformation that adds #:aot? #false flags everywhere or > whatever the exact argument is (**). > > Given that this transformation has some legitimate use cases, this > transformation could even be a pre-canned procedure + transformation included > in Guix itself > > (*) ‘unfortunately’ only applies to the first case. For the case in > parentheses, replace by ‘fortunately’. > (**) IIRC is wasn’t #:aot? but some other name, but that’s not really the > point here. > > (d) If a deployment need multiple versions of Clojure, then just fix your > deployment to make everything work with the latest version of Clojure. Or, if > you for some reason don’t do that, just use a (recursive) transformation to > change the version of the Clojure compiler used to match the Clojure that > will be used in the particular deployment. > > (e) You can simply add missing packages to Guix as the need arises. > > >I can see the sense of compiling to byte code if it's an end-user > >application. In that case we'd want to make the start-up
RE: Proposal to turn off AOT in clojure-build-system
Hey Maxime, On Thu, Feb 22 2024, Maxime Devos wrote: > familiarity with good manners Sometimes it's better to admit defeat. Maybe you find this apology [1] helpful, although it was imperfect. Kind regards Felix [1] https://lists.gnu.org/archive/html/guix-devel/2022-10/msg00141.html
Re: Proposal to turn off AOT in clojure-build-system
Hi Maxime! Maxime Devos writes: >>On Thu, Feb 22 2024, Andreas Enge wrote: >>> Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos: Yes. It appears you are unfamiliar with (...) It also appears you are unfamiliar with (...) >>> >>> May I suggest to not make assumptions about what other people are familiar >>> with or not? There is no point in claiming that others are less knowledge- >>> able than you; they may know as much or even more than you >> >>Asking questions can avoid any such conclusions. It's my favorite >>thing. Here, I might have asked: "Do you know about...?" > > OK. I don’t think I’ll do that. Your sharp replies came off as harsh, which is unfortunate because you seem to bring points of technical merits. Please smooth out your interactions/communication style, to ensure it remains pleasant for everyone involved, taking these code of conduct items into account: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences -- Thanks, Maxim
RE: Proposal to turn off AOT in clojure-build-system
>On Thu, Feb 22 2024, Andreas Enge wrote: >> Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos: >>> Yes. It appears you are unfamiliar with (...) >>> It also appears you are unfamiliar with (...) >> >> May I suggest to not make assumptions about what other people are familiar >> with or not? There is no point in claiming that others are less knowledge- >> able than you; they may know as much or even more than you > >Asking questions can avoid any such conclusions. It's my favorite >thing. Here, I might have asked: "Do you know about...?" OK. I don’t think I’ll do that. Best regards, Maxime Devos.
RE: Proposal to turn off AOT in clojure-build-system
>Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos: >> Yes. It appears you are unfamiliar with (...) >> It also appears you are unfamiliar with (...) >May I suggest to not make assumptions about what other people are familiar with or not? There is no point in claiming that others are less knowledge- able than you; they may know as much or even more than you, and still come to different conclusions. (And even if people were unfamiliar with something, I would object to this haughty tone and suggest a more pleasant way of making suggestions.) This is hypocritical, you are (somewhat implicitly) making assumptions on my (un)familiarity with good manners and (somewhat implicitly) claiming that I am less knowledges than you on good manners. Furthermore, you aren’t actually suggesting a more pleasant to formulate the message of the part you quoted. Also, I did not simply _assume_ that Steve was unfamiliar with transformation, I _concluded_ that Steve was likely unfamiliar by what they didn’t mention in their e-mails. Furthermore, the mere “likelihood” is included in the paragraph you quoted as part of the word “appears” – I did not claim that Steve is unfamiliar, I only claimed that it appeared to be the case, which is not the same thing. After all, perhaps Steve does know that such transformation exist but personally concluded them to not be a proper solution for some reason. In that case, now people know there is a disagreement on the role of transformations w.r.t. AOT and perhaps the source of the disagreement can be resolved, furthering knowledge and bring us closer to deciding on what the proper AOT default would be. You mention that other people might know more, but there is also a flipside to this – sometimes people know _less_. In this case, perhaps Steve simply did not know about transformations and the proposed use of transformations in combination with enabling AOT by default would be agreeable to Steve and as such perhaps an answer to the decision whether to enable AOT by default. As such, simply _assuming_ that Steve knew of transformations would be wrong, so I had to mention the option of transformations. I did not simply claim that Steve is less knowledgable than me, at most it could be said that I (implicitly) claimed that Steve is less knowledged than me on the relation between transformations and Clojure AOT problems, but even then, I included a qualifier “It appears that”, not “It is the case that”. The beginning “It appears you are unfamiliar with [...]” is simply a perfectly cromulent beginning of a sentence, not some assertion of superiority. >For instance concerning the topic at hand, knowing that users may transform packages as they wish to me seems to be independent of which default choice we should make for the distribution. It is not independent, see my previous e-mail where I explained how the existence of transformations turns some problems mentioned w.r.t. enabling AOT into non-problems. If you have a disagreement with that explanation, please actually say what the disagreement is instead of only saying that you disagree. The former might bring us closer to some collective decision on what the proper default behaviour is for Guix, the latter doesn’t, is almost useless and makes it look like you didn’t bother to read the ‘(...)’ part. While I suppose it is technically possible you have read the part ‘(...)’, given your response I simply don’t believe you did and hence I consider it fair to conclude that you are not familiar with the contents of the (...) part. Best regards, Maxime Devos
Re: Proposal to turn off AOT in clojure-build-system
Hey, On Thu, Feb 22 2024, Andreas Enge wrote: > Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos: >> Yes. It appears you are unfamiliar with (...) >> It also appears you are unfamiliar with (...) > > May I suggest to not make assumptions about what other people are familiar > with or not? There is no point in claiming that others are less knowledge- > able than you; they may know as much or even more than you Asking questions can avoid any such conclusions. It's my favorite thing. Here, I might have asked: "Do you know about...?" Goodwill toward all, Felix
Re: Proposal to turn off AOT in clojure-build-system
Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos: > Yes. It appears you are unfamiliar with (...) > It also appears you are unfamiliar with (...) May I suggest to not make assumptions about what other people are familiar with or not? There is no point in claiming that others are less knowledge- able than you; they may know as much or even more than you, and still come to different conclusions. (And even if people were unfamiliar with something, I would object to this haughty tone and suggest a more pleasant way of making suggestions.) For instance concerning the topic at hand, knowing that users may transform packages as they wish to me seems to be independent of which default choice we should make for the distribution. Andreas
RE: Proposal to turn off AOT in clojure-build-system
>> [...] >> > But, I would like to draw attention to this thread on Clojureverse as the >> > best source I could find: >> >Alex Miller is the main community manager for Clojure, and is a maintainer >> >of the core libraries, so his perspective is key. He notes that, AOT code >> >is tied to *specific versions of Clojure*: >> > >> > "AOT'ed code is that it is inherently the product of a particular version >> > of tthe Clojure compiler ... I would recommend NOT AOT compiling >> > libraries" [4] >> >> This reasoning does not follow – yes, it is tied to the Clojure version, so >> what? Guix automatically rebuilds dependents when the dependency (in this >> case, the Clojure compiler) changes. (...) >I think this preceding sentence is the heart of different assumptions between >us. >The Clojure packages we haave are for developers writing applications >(libraries and tools). The ecosystem has very few end-user applications [0]. >As a Clojure developer I can use the Clojure tools from Guix, and a few >libraries. We (and all the other distributions) have a miniscule portion of >the Clojure/Java library universe [1]. This leads to the following usage >scenarios: >1. A developer installs Clojure from Guix, and uses libraries from outside >Guix. They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT. >2. A developer installs a Clojure from outside Guix, uses libraries from >inside Guix This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs. My answer to (1) and (2) is: (a) About “install Clojure from outside Guix + use libraries inside Guix”: If these libraries are AOT: Don’t do that, then. If you mix-and-match binaries (in this case, .class files) different distributions, you are free to do so, but when (not if, when) things break, you get to keep the pieces. If these libraries are just the .clj files (not AOT) (which as I understand it is the standard situation): doesn’t seem a problem to me (see point (b)). Adding source from other places is one thing (probably ok), adding binaries is another (probably not ok, especially if unstable ABIs (see Clojure compiler) and mismatched versions (see hypotheses of 2.) are involved. (b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source on this? Besides Clojure supposedly, I haven’t ever heard of such problems for any language – for example, there is no such issue with Guile and AFAIK not for Python. I haven’t heard of any such issues for the Common Lisp implementations either (though I haven’t checked), so this doesn’t seem like a “Clojure doesn’t do hygienic macros” issue. (c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want to assist in murdering the climate (or its a situation where total cost of non-AOT is lower than AOT), they can unfortunately (*) simply do so applying a recursive transformation that adds #:aot? #false flags everywhere or whatever the exact argument is (**). Given that this transformation has some legitimate use cases, this transformation could even be a pre-canned procedure + transformation included in Guix itself (*) ‘unfortunately’ only applies to the first case. For the case in parentheses, replace by ‘fortunately’. (**) IIRC is wasn’t #:aot? but some other name, but that’s not really the point here. (d) If a deployment need multiple versions of Clojure, then just fix your deployment to make everything work with the latest version of Clojure. Or, if you for some reason don’t do that, just use a (recursive) transformation to change the version of the Clojure compiler used to match the Clojure that will be used in the particular deployment. (e) You can simply add missing packages to Guix as the need arises. >I can see the sense of compiling to byte code if it's an end-user application. >In that case we'd want to make the start-up as fast as possible. Your comments >seem to have this use-case in mind. >But, today there aren't any such end-user Clojure applications in Guix. That’s a shame. Hopefully that will change some day. Also, this is false, “clojure-tools” exists – developers are people too (I don’t care about the “_end_-user” distinction – surely developers want fast start-up as well). Also, I _don’t_ have that use case in mind – I have efficiency in general in mind, efficiency of the start-up is only a part of that. The point is: most likely you will want the application to have AOT code, and that library has dependencies. Furthermore, likely many of these dependencies are dependencies of other applications as well. Instead of redoing the compilation of N
Re: Proposal to turn off AOT in clojure-build-system
Hi Maxime, On 19 Feb, Maxime Devos wrote: (...) > > Consequently, there is no specific statement saying 'Distributors should > > not AOT libraries' that I can point to. > > In this bit about differences in perspective, I haven’t seen any mention of > AOT, hence the “Consequently” does not follow. The part that’s missing here > is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled > .class files in a superior Aryan JAR instead – the inferior UnderJARs you get > from the “guix install clj-whatever” equivalent would only contain > non-compiled .clj (and data files, whatever). > > > But, I would like to draw attention to this thread on Clojureverse as the > > best source I could find: > >Alex Miller is the main community manager for Clojure, and is a maintainer > >of the core libraries, so his perspective is key. He notes that, AOT code is > >tied to *specific versions of Clojure*: > > > > "AOT'ed code is that it is inherently the product of a particular version > > of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" > > [4] > > This reasoning does not follow – yes, it is tied to the Clojure version, so > what? Guix automatically rebuilds dependents when the dependency (in this > case, the Clojure compiler) changes. (...) I think this preceding sentence is the heart of different assumptions between us. The Clojure packages we haave are for developers writing applications (libraries and tools). The ecosystem has very few end-user applications [0]. As a Clojure developer I can use the Clojure tools from Guix, and a few libraries. We (and all the other distributions) have a miniscule portion of the Clojure/Java library universe [1]. This leads to the following usage scenarios: 1. A developer installs Clojure from Guix, and uses libraries from outside Guix. They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT. 2. A developer installs a Clojure from outside Guix, uses libraries from inside Guix This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs. I can see the sense of compiling to byte code if it's an end-user application. In that case we'd want to make the start-up as fast as possible. Your comments seem to have this use-case in mind. But, today there aren't any such end-user Clojure applications in Guix. > >I believe this means that with AOT code on, any user who installs a > >different version of Clojure from the one that we used to AOT the libraries > >*may* have problems. > > Unlike, say, Maven, this situation simply does not happen in Guix, because we > don’t just download binaries and call it a day (except for some bootstrapping > stuff, but that’s not relevant for Clojure AOT), because we have functioning > recompilation of dependents, because of shebang patching, because binaries > that are to be invoked should not rely on the ambient CLASSPATH / > LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they > are wrapped (see wrap-program) to set them (or, at least, they should be, you > might find some bugs in this department if you go looking). > > Even if they aren’t wrapped, then in that case the dependencies are > propagated-inputs, and you can only have a single version of a propagated > package in the same environment (barring stacking environment shenanigans, > but then you are looking for it and/or you can just report a bug about how > the binaries should be wrapped/classpath should be patched in/...). > In this paragraph you're assumption is that a Guix user is only using libraries from within Guix. Hopefully, I've shown why this assumption is unlikely above. You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries are not AOT'd on Debian, while applications are. From what I can find there are 130 packages in Debian with the word Clojure in them [3]. I looked at a selection and it seems true that Debian does not AOT libraries (and I can't find any Clojure 'apps'). For completeness I also checked what Clojars, the main distribution archive for Clojure developers, does: - https://sources.debian.org/src/core-match-clojure/1.0.0-1/project.clj/ - Lein-based project - has to have :aot keyword - distributes as source files - Clojure source files (.clj) in Debian - Clojure source files in Clojars - Byte compiled files in GUIX - installed and inspected with jar -tvf - https://sources.debian.org/src/data-csv-clojure/1.0.0-1/deps.edn/ - tools.deps project - has to have a specific aot alias - distributes as source files - Clojure source files (.clj) in the
Re: Proposal to turn off AOT in clojure-build-system
As someone who has worked as a professional Clojure programmer, I would like to add my voice in support of this: On Mon, Feb 19 2024, Ryan Sundberg wrote: > ... In my experience using AOT is the exception rather than the rule; > it is a nice optimization when practical for release engineering, but > typically the juice is not worth the squeeze. That is: the norm in the Clojure community is *not* using AOT, so there isn't any specific statement against it. It's just assumed that you won't use AOT unless you have a reason to do so. Reading the clojure.org page about compilation[1] with this in mind, we see four possible (although admittedly not exhaustive) reasons to provide AOT compilation: - to deliver your application without source - to speed up application startup - to generated named classes - to create an application that does not need runtime bytecode generation and custom classloaders I'm not convinced any of these apply across the board in Guix. Potentially for some libraries and applications it does, and so should be applied in those specific cases. The next statement on the page, after those reasons to use AOT is: "The Clojure compilation model preserves as much as possible the dynamic nature of Clojure, in spite of the code-reloading limitations of Java." Or, to read it another way: AOT compiled Clojure is not as dynamic as on-the-fly compiled Clojure. This isn't a bug in Clojure, this is how it is designed to function. It could be argued that it is a bug in the individual libraries to not work with AOT, but I think it's inappropriate for Guix to attempt to impose "AOT is the right way to do things" on Clojure libraries. That is not the norm in that community. As a personal datapoint: I spent three and a half years working on mission-critical Clojure libraries and applications, and I have never used AOT compilation in production. Carlo [1]: https://clojure.org/reference/compilation
Re: Proposal to turn off AOT in clojure-build-system
As a daily Clojure programmer using Guix the default 'off' makes sense. The only situation where AOT compilation is useful is in the final runnable application programs, and even then, its often incompatible with AOT compilation in subtle ways. Having libraries aot compiled not only adds incompatibility (including which java the library was built for, which creates a "least common denominator problem which JDK you can use for the final build) but also wastes time and energy. Identifying AOT compilation bugs can also be extremely frustrating to package maintainers as the underlying cause is usually not obvious. In my experience using AOT is the exception rather than the rule; it is a nice optimization when practical for release engineering, but typically the juice is not worth the squeeze. Sincerely, Ryan Sundberg Feb 19, 2024 3:48:54 AM Steve George : > Hi, > > Guix's clojure-build-system turns on AOT compilation by default. I would like > to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, > so we should change the default. > > This has been discussed previously. In #56604 r0man noted that AOT > compilation should not be on by default [0], Reilly makes the same point in > #53765 [1]. > > Maxime makes the point that where a compiler is available it should be used > [2] and that if it doesn't work it's a bug: > > "if a Clojure library misbehaves when AOT-compiled, without additional > context, that seems like a bug in the Clojure library to me (or the > AOT-compilation code). > > The perspective in the Clojure community is quite different from Guix's on a > number of fronts. There's not much discussion about offline builds, > reproducibility or code coming from Distributions. The internalised > perspective is that you use the build tools to download libraries directly > from Clojars (a Maven repo) and developers create a final uberjar for > production usage Consequently, there is no specific statement saying > 'Distributors should not AOT libraries' that I can point to. But, I would > like to draw attention to this thread on Clojureverse as the best source I > could find: > > Alex Miller is the main community manager for Clojure, and is a maintainer of > the core libraries, so his perspective is key. He notes that, AOT code is > tied to *specific versions of Clojure*: > > "AOT'ed code is that it is inherently the product of a particular version > of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" > [4] > > In the same thread thheller, who is the maintainer of the most popular > ClojureScript tooling, notes you cannot mix AOT and non-AOT libraries [5]: > > "you cannot just ship your library AOT compiles as it would also contain > clojure.core. Clojure AOT current ... can not load clj files from .class > files. So AOT produces the class files and will fail if one of the dependent > classes is missing although the .clj file is present" > > I believe this means that with AOT code on, any user who installs a different > version of Clojure from the one that we used to AOT the libraries *may* have > problems. And, that we can't have trees where some part is AOT'd but a > dependency is not. Finally, there is no expectation in the Clojure community > that this is a bug, consequently it will not be fixed. Therefore, we should > change to default to AOT off. > > What do people think, does this make sense? > > Thanks, > > Steve / Futurile > > [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5 > [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290 > [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293 > [4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6 > [5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3 > [5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
RE: Proposal to turn off AOT in clojure-build-system
((As I said about a month ago, in theory I should have access to a proper e-mail program again a week or two in the past.)) >Hi, >Guix's clojure-build-system turns on AOT compilation by default. I would like >to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, >so we should change the default. >This has been discussed previously. In #56604 r0man noted that AOT compilation >should not be on by default [0], Reilly makes the same point in #53765 [1]. >Maxime makes the point that where a compiler is available it should be used >[2] and that if it doesn't work it's a bug: >> "if a Clojure library misbehaves when AOT-compiled, without additional >> context, that seems like a bug in the Clojure library to me (or the >> AOT-compilation code). >The perspective in the Clojure community is quite different from Guix's on a >number of fronts. There's not much discussion about offline builds This (not the has nothing to do with AOT. > reproducibility If some libraries have irreproducible binaries, then just don’t AOT those libraries (and, if the irreproducibility is in macros (and maybe inlinable generated code, if that’s something the AOT does), the relevant dependents of those libraries as well). An AOT problem for a few libraries is not a reason to turn off AOT by default, it is a reason to turn off AOT for those few libraries in particular. > or code coming from Distributions. You are contradicting this by the next paragraph, in which you mention the distribution “Clojars”. A rather specialised and lacking distribution, but a distribution nonetheless. >The internalised perspective is that you use the build tools to download >libraries directly from Clojars (a Maven repo) That seems quite similar to Guix, where you use the build tools “guix build” / “guix install” / ... to download libraries (by default, when available) directly from ci.guix.gnu.org (I forgot the terminology, but you could compare it to Clojars I guess). > and developers create a final uberjar for production usage Except for the delusion of grandeur (AFAIK, in non-Java English, über only appears in the word Übermensch) (maybe whoever coined the term didn’t really mean it but eergh), this is also familiar, see “guix system vm” for large-scale things, and the command for creating relocatable packs (I forgot the exact name and command) on a smaller scale. > Consequently, there is no specific statement saying 'Distributors should not > AOT libraries' that I can point to. In this bit about differences in perspective, I haven’t seen any mention of AOT, hence the “Consequently” does not follow. The part that’s missing here is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .class files in a superior Aryan JAR instead – the inferior UnderJARs you get from the “guix install clj-whatever” equivalent would only contain non-compiled .clj (and data files, whatever). > But, I would like to draw attention to this thread on Clojureverse as the > best source I could find: >Alex Miller is the main community manager for Clojure, and is a maintainer of >the core libraries, so his perspective is key. He notes that, AOT code is tied >to *specific versions of Clojure*: > > "AOT'ed code is that it is inherently the product of a particular version of > tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4] This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes. I guess the maintainer has something like Maven etc. in mind, where many maintainers, each with different distribution, Java version, etc. upload binaries, but that’s not the case in Guix. (To a lesser extent, this is not the case in, say, Debian as well.) >In the same thread thheller, who is the maintainer of the most popular >ClojureScript tooling, notes you cannot mix AOT and non-AOT libraries [5]: "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present" This argument makes no sense at all. Of course it can’t load CLJ files from .class files – Clojure AOT is a compiler, not a decompiler, cf. how GCC can’t ‘load’ C files from .o / ELF / ..., and why would it in the first place? And of course you can’t run or compile an application if a part of the binaries / a part of the source code is missing. This is the same as with C & GCC, (Guile) Scheme & “guild compile”, etc.. Yet, GCC and Guile somehow have a functioning AOT (if we count Guile’s bytecode as AOT, which we can because we are counting .class files as “compiled”). (As a side-remark, I think a typo was made here: dependent classes -> dependency classes (i.e.
