Hi, I don't think I'm making any progress convincing you, and I'm not enjoying the interaction so I'm going to take a few days off from this thread.
I've tried to ask the Clojure community for a definitive expression of what they think Linux distributions should do with byte-compiled libraries. We'll see if there's a response, this is a public forum so feel free to chime in and express clearly your perpective: https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/1 Thanks, Steve / Futurile On 22 Feb, Maxime Devos wrote: > >> [...] > >> > But, I would like to draw attention to this thread on Clojureverse as > >> > the best source I could find: > >> >Alex Miller is the main community manager for Clojure, and is a > >> >maintainer of the core libraries, so his perspective is key. He notes > >> >that, AOT code is tied to *specific versions of Clojure*: > >> > > >> > "AOT'ed code is that it is inherently the product of a particular > >> > version of tthe Clojure compiler ... I would recommend NOT AOT compiling > >> > libraries" [4] > >> > >> This reasoning does not follow – yes, it is tied to the Clojure version, > >> so what? Guix automatically rebuilds dependents when the dependency (in > >> this case, the Clojure compiler) changes. > (...) > > >I think this preceding sentence is the heart of different assumptions > >between us. > > >The Clojure packages we haave are for developers writing applications > >(libraries and tools). The ecosystem has very few end-user applications [0]. > >As a Clojure developer I can use the Clojure tools from Guix, and a few > >libraries. We (and all the other distributions) have a miniscule portion of > >the Clojure/Java library universe [1]. This leads to the following usage > >scenarios: > > >1. A developer installs Clojure from Guix, and uses libraries from outside > >Guix. > They can install the JVM/Clojure and some common tools (like clj-tools-cli). > They will use libraries from elsewhere, including their own. AOT compilation > is a problem because of the issue of mixed AOT and non-AOT. > > >2. A developer installs a Clojure from outside Guix, uses libraries from > >inside Guix > This will cause problems because the Guix Clojure libraries will have been > AOT'd by a different version of the compiler. It's also fairly common to > install/use parallel versions of Clojure/jvm due to different deployment > needs, this is likely to cause difficult to find bugs. > > My answer to (1) and (2) is: > > (a) About “install Clojure from outside Guix + use libraries inside Guix”: > > If these libraries are AOT: > > Don’t do that, then. If you mix-and-match binaries (in this case, .class > files) different distributions, you are free to do so, but when (not if, > when) things break, you get to keep the pieces. > > If these libraries are just the .clj files (not AOT) (which as I understand > it is the standard situation): doesn’t seem a problem to me (see point (b)). > > Adding source from other places is one thing (probably ok), adding binaries > is another (probably not ok, especially if unstable ABIs (see Clojure > compiler) and mismatched versions (see hypotheses of 2.) are involved. > > (b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source > on this? Besides Clojure supposedly, I haven’t ever heard of such problems > for any language – for example, there is no such issue with Guile and AFAIK > not for Python. I haven’t heard of any such issues for the Common Lisp > implementations either (though I haven’t checked), so this doesn’t seem like > a “Clojure doesn’t do hygienic macros” issue. > > (c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want > to assist in murdering the climate (or its a situation where total cost of > non-AOT is lower than AOT), they can unfortunately (*) simply do so applying > a recursive transformation that adds #:aot? #false flags everywhere or > whatever the exact argument is (**). > > Given that this transformation has some legitimate use cases, this > transformation could even be a pre-canned procedure + transformation included > in Guix itself > > (*) ‘unfortunately’ only applies to the first case. For the case in > parentheses, replace by ‘fortunately’. > (**) IIRC is wasn’t #:aot? but some other name, but that’s not really the > point here. > > (d) If a deployment need multiple versions of Clojure, then just fix your > deployment to make everything work with the latest version of Clojure. Or, if > you for some reason don’t do that, just use a (recursive) transformation to > change the version of the Clojure compiler used to match the Clojure that > will be used in the particular deployment. > > (e) You can simply add missing packages to Guix as the need arises. > > >I can see the sense of compiling to byte code if it's an end-user > >application. In that case we'd want to make the start-up as fast as > >possible. Your comments seem to have this use-case in mind. > > >But, today there aren't any such end-user Clojure applications in Guix. > > That’s a shame. Hopefully that will change some day. > Also, this is false, “clojure-tools” exists – developers are people too (I > don’t care about the “_end_-user” distinction – surely developers want fast > start-up as well). > > Also, I _don’t_ have that use case in mind – I have efficiency in general in > mind, efficiency of the start-up is only a part of that. > > The point is: most likely you will want the application to have AOT code, and > that library has dependencies. Furthermore, likely many of these dependencies > are dependencies of other applications as well. > > Instead of redoing the compilation of N shared dependencies for each M > applications (total work: ~N*M), by not participating in these Clojure > conventions, the total work of compiling dependencies can be reduced to ~N). > > Let’s avoid joining the madness that is Go’s build system and Cargo(*). > > (*) Rust is fine for compilation, it’s cargo that is the problem (at least, > Cargo as used by Guix) – antioxidant > (https://notabug.org/maximed/cargoless-rust-experiments) avoids the sheer > inefficiency of Cargo. > > >> >I believe this means that with AOT code on, any user who installs a > >> >different version of Clojure from the one that we used to AOT the > >> >libraries *may* have problems. > >> > >> Unlike, say, Maven, this situation simply does not happen in Guix, because > >> we don’t just download binaries and call it a day (except for some > >> bootstrapping stuff, but that’s not relevant for Clojure AOT), because we > >> have functioning recompilation of dependents, because of shebang patching, > >> because binaries that are to be invoked should not rely on the ambient > >> CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do > >> rely on that, they are wrapped (see wrap-program) to set them (or, at > >> least, they should be, you might find some bugs in this department if you > >> go looking). > >> > >> Even if they aren’t wrapped, then in that case the dependencies are > >> propagated-inputs, and you can only have a single version of a propagated > >> package in the same environment (barring stacking environment shenanigans, > >> but then you are looking for it and/or you can just report a bug about how > >> the binaries should be wrapped/classpath should be patched in/...). > > > > >In this paragraph you're assumption is that a Guix user is only using > >libraries from within Guix. Hopefully, I've shown why this assumption is > >unlikely above. > > You haven’t. You have shown that other situations exist, not that this > situation is unlikely. To demonstrate likelihood / unlikelyhood, you need > some statistics (or a very thorough argument on why people wouldn’t just make > a package for missing dependencies – it’s very simple, especially if you > aren’t sending them for inclusion.) > > >You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries > >are not AOT'd on Debian, while applications are. From what I can find there > >are 130 packages in Debian with the word Clojure in them [3]. I looked at a > >selection and it seems true that Debian does not AOT libraries (and I can't > >find any Clojure 'apps'). For completeness I also checked what Clojars, the > >main distribution archive for Clojure developers, does: > > [...] > > OK ... and? Other people doing things differently doesn’t mean those > different approaches are better. If Debian mentions somewhere the reason > _why_ it doesn’t AOT libraries, that reason can be investigated and perhaps > carried over to Guix, but it’s the ‘why’ that’s important, not the ‘it > doesn’t’. > > > Does this help clarify why I'm asking to change the default? > > Yes. It appears you are unfamiliar with the “transformations” functionality > of Guix, which allows for easy rewriting of packages to whatever Clojure > version you want, eliminating compatibility concerns or even turning off AOT > completely. > > (This requires the relevant versions of Clojure to be packaged in Guix, but > that shouldn’t be too much of a problem.) > > It also appears you are unfamiliar with Guix importers, which easily make > package definitions (somewhat low quality w.r.t. descriptions, with lack of > verification of freeness, etc., but usually otherwise fully functional). I > don’t know if an importer for Clojure exists but it shouldn’t be too > difficult I think. > > Best regards, > Maxime Devos.
