Re: Proposal to turn off AOT in clojure-build-system
As someone who has worked as a professional Clojure programmer, I would like to add my voice in support of this: On Mon, Feb 19 2024, Ryan Sundberg wrote: > ... In my experience using AOT is the exception rather than the rule; > it is a nice optimization when practical for release engineering, but > typically the juice is not worth the squeeze. That is: the norm in the Clojure community is *not* using AOT, so there isn't any specific statement against it. It's just assumed that you won't use AOT unless you have a reason to do so. Reading the clojure.org page about compilation[1] with this in mind, we see four possible (although admittedly not exhaustive) reasons to provide AOT compilation: - to deliver your application without source - to speed up application startup - to generated named classes - to create an application that does not need runtime bytecode generation and custom classloaders I'm not convinced any of these apply across the board in Guix. Potentially for some libraries and applications it does, and so should be applied in those specific cases. The next statement on the page, after those reasons to use AOT is: "The Clojure compilation model preserves as much as possible the dynamic nature of Clojure, in spite of the code-reloading limitations of Java." Or, to read it another way: AOT compiled Clojure is not as dynamic as on-the-fly compiled Clojure. This isn't a bug in Clojure, this is how it is designed to function. It could be argued that it is a bug in the individual libraries to not work with AOT, but I think it's inappropriate for Guix to attempt to impose "AOT is the right way to do things" on Clojure libraries. That is not the norm in that community. As a personal datapoint: I spent three and a half years working on mission-critical Clojure libraries and applications, and I have never used AOT compilation in production. Carlo [1]: https://clojure.org/reference/compilation
“guix package -d” on empty profile downloads stuff
Hi Guix, I have a little i686-linux laptop to which I deployed the Sugar desktop. The only user account on that laptop (other than root) has no Guix profile. I switched to that user account and ran “guix package -d”. Here is what happened: --8<---cut here---start->8--- $ guix package -d substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% 25.8 MB will be downloaded glibc-2.35 8.5MiB 1.0MiB/s 00:08 ▕██▏ 100.0% bash-minimal-5.1.16 603KiB2.0MiB/s 00:00 ▕██▏ 100.0% gcc-11.3.0-lib 4.8MiB 977KiB/s 00:05 ▕██▏ 100.0% libffi-3.4.4 57KiB588KiB/s 00:00 ▕██▏ 100.0% libunistring-1.0 671KiB 1.1MiB/s 00:01 ▕██▏ 100.0% pkg-config-0.29.2 218KiB 1.1MiB/s 00:00 ▕██▏ 100.0% libgc-8.2.2 219KiB767KiB/s 00:00 ▕██▏ 100.0% guile-3.0.9 8.1MiB1.3MiB/s 00:06 ▕██▏ 100.0% guile-3.0.9-debug 7.5MiB 1.4MiB/s 00:05 ▕██▏ 100.0% substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% 0.4 MB will be downloaded glibc-utf8-locales-2.35 382KiB266KiB/s 00:01 ▕██▏ 100.0% guix package: error: profile '/var/guix/profiles/per-user/xxx/guix-profile' does not exist --8<---cut here---end--->8--- Why do we need to download 25.8 MB to decide that we can’t do anything here? My usual excuse when faced with questions of this kind what always that profile hooks may need to run, and they may require more recent versions of tools to be built first. Here we don’t seem to have a good reason for that though. -- Ricardo
Re: Proposal to turn off AOT in clojure-build-system
As a daily Clojure programmer using Guix the default 'off' makes sense. The only situation where AOT compilation is useful is in the final runnable application programs, and even then, its often incompatible with AOT compilation in subtle ways. Having libraries aot compiled not only adds incompatibility (including which java the library was built for, which creates a "least common denominator problem which JDK you can use for the final build) but also wastes time and energy. Identifying AOT compilation bugs can also be extremely frustrating to package maintainers as the underlying cause is usually not obvious. In my experience using AOT is the exception rather than the rule; it is a nice optimization when practical for release engineering, but typically the juice is not worth the squeeze. Sincerely, Ryan Sundberg Feb 19, 2024 3:48:54 AM Steve George : > Hi, > > Guix's clojure-build-system turns on AOT compilation by default. I would like > to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, > so we should change the default. > > This has been discussed previously. In #56604 r0man noted that AOT > compilation should not be on by default [0], Reilly makes the same point in > #53765 [1]. > > Maxime makes the point that where a compiler is available it should be used > [2] and that if it doesn't work it's a bug: > > "if a Clojure library misbehaves when AOT-compiled, without additional > context, that seems like a bug in the Clojure library to me (or the > AOT-compilation code). > > The perspective in the Clojure community is quite different from Guix's on a > number of fronts. There's not much discussion about offline builds, > reproducibility or code coming from Distributions. The internalised > perspective is that you use the build tools to download libraries directly > from Clojars (a Maven repo) and developers create a final uberjar for > production usage Consequently, there is no specific statement saying > 'Distributors should not AOT libraries' that I can point to. But, I would > like to draw attention to this thread on Clojureverse as the best source I > could find: > > Alex Miller is the main community manager for Clojure, and is a maintainer of > the core libraries, so his perspective is key. He notes that, AOT code is > tied to *specific versions of Clojure*: > > "AOT'ed code is that it is inherently the product of a particular version > of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" > [4] > > In the same thread thheller, who is the maintainer of the most popular > ClojureScript tooling, notes you cannot mix AOT and non-AOT libraries [5]: > > "you cannot just ship your library AOT compiles as it would also contain > clojure.core. Clojure AOT current ... can not load clj files from .class > files. So AOT produces the class files and will fail if one of the dependent > classes is missing although the .clj file is present" > > I believe this means that with AOT code on, any user who installs a different > version of Clojure from the one that we used to AOT the libraries *may* have > problems. And, that we can't have trees where some part is AOT'd but a > dependency is not. Finally, there is no expectation in the Clojure community > that this is a bug, consequently it will not be fixed. Therefore, we should > change to default to AOT off. > > What do people think, does this make sense? > > Thanks, > > Steve / Futurile > > [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5 > [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290 > [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293 > [4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6 > [5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3 > [5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
RE: Proposal to turn off AOT in clojure-build-system
((As I said about a month ago, in theory I should have access to a proper e-mail program again a week or two in the past.)) >Hi, >Guix's clojure-build-system turns on AOT compilation by default. I would like >to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, >so we should change the default. >This has been discussed previously. In #56604 r0man noted that AOT compilation >should not be on by default [0], Reilly makes the same point in #53765 [1]. >Maxime makes the point that where a compiler is available it should be used >[2] and that if it doesn't work it's a bug: >> "if a Clojure library misbehaves when AOT-compiled, without additional >> context, that seems like a bug in the Clojure library to me (or the >> AOT-compilation code). >The perspective in the Clojure community is quite different from Guix's on a >number of fronts. There's not much discussion about offline builds This (not the has nothing to do with AOT. > reproducibility If some libraries have irreproducible binaries, then just don’t AOT those libraries (and, if the irreproducibility is in macros (and maybe inlinable generated code, if that’s something the AOT does), the relevant dependents of those libraries as well). An AOT problem for a few libraries is not a reason to turn off AOT by default, it is a reason to turn off AOT for those few libraries in particular. > or code coming from Distributions. You are contradicting this by the next paragraph, in which you mention the distribution “Clojars”. A rather specialised and lacking distribution, but a distribution nonetheless. >The internalised perspective is that you use the build tools to download >libraries directly from Clojars (a Maven repo) That seems quite similar to Guix, where you use the build tools “guix build” / “guix install” / ... to download libraries (by default, when available) directly from ci.guix.gnu.org (I forgot the terminology, but you could compare it to Clojars I guess). > and developers create a final uberjar for production usage Except for the delusion of grandeur (AFAIK, in non-Java English, über only appears in the word Übermensch) (maybe whoever coined the term didn’t really mean it but eergh), this is also familiar, see “guix system vm” for large-scale things, and the command for creating relocatable packs (I forgot the exact name and command) on a smaller scale. > Consequently, there is no specific statement saying 'Distributors should not > AOT libraries' that I can point to. In this bit about differences in perspective, I haven’t seen any mention of AOT, hence the “Consequently” does not follow. The part that’s missing here is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .class files in a superior Aryan JAR instead – the inferior UnderJARs you get from the “guix install clj-whatever” equivalent would only contain non-compiled .clj (and data files, whatever). > But, I would like to draw attention to this thread on Clojureverse as the > best source I could find: >Alex Miller is the main community manager for Clojure, and is a maintainer of >the core libraries, so his perspective is key. He notes that, AOT code is tied >to *specific versions of Clojure*: > > "AOT'ed code is that it is inherently the product of a particular version of > tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4] This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes. I guess the maintainer has something like Maven etc. in mind, where many maintainers, each with different distribution, Java version, etc. upload binaries, but that’s not the case in Guix. (To a lesser extent, this is not the case in, say, Debian as well.) >In the same thread thheller, who is the maintainer of the most popular >ClojureScript tooling, notes you cannot mix AOT and non-AOT libraries [5]: "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present" This argument makes no sense at all. Of course it can’t load CLJ files from .class files – Clojure AOT is a compiler, not a decompiler, cf. how GCC can’t ‘load’ C files from .o / ELF / ..., and why would it in the first place? And of course you can’t run or compile an application if a part of the binaries / a part of the source code is missing. This is the same as with C & GCC, (Guile) Scheme & “guild compile”, etc.. Yet, GCC and Guile somehow have a functioning AOT (if we count Guile’s bytecode as AOT, which we can because we are counting .class files as “compiled”). (As a side-remark, I think a typo was made here: dependent classes -> dependency classes (i.e.
Proposal to turn off AOT in clojure-build-system
Hi, Guix's clojure-build-system turns on AOT compilation by default. I would like to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, so we should change the default. This has been discussed previously. In #56604 r0man noted that AOT compilation should not be on by default [0], Reilly makes the same point in #53765 [1]. Maxime makes the point that where a compiler is available it should be used [2] and that if it doesn't work it's a bug: "if a Clojure library misbehaves when AOT-compiled, without additional context, that seems like a bug in the Clojure library to me (or the AOT-compilation code). The perspective in the Clojure community is quite different from Guix's on a number of fronts. There's not much discussion about offline builds, reproducibility or code coming from Distributions. The internalised perspective is that you use the build tools to download libraries directly from Clojars (a Maven repo) and developers create a final uberjar for production usage Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to. But, I would like to draw attention to this thread on Clojureverse as the best source I could find: Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*: "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4] In the same thread thheller, who is the maintainer of the most popular ClojureScript tooling, notes you cannot mix AOT and non-AOT libraries [5]: "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present" I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems. And, that we can't have trees where some part is AOT'd but a dependency is not. Finally, there is no expectation in the Clojure community that this is a bug, consequently it will not be fixed. Therefore, we should change to default to AOT off. What do people think, does this make sense? Thanks, Steve / Futurile [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5 [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290 [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293 [4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6 [5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3 [5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
Re: [RFC] proposal for refactoring bootloaders
On Fri, Feb 16, 2024 at 02:33:02AM -0600, Lilah Tascheter wrote: > one more quick change that I've realized will be necessary: add a > bootloader-targets field to boot-parameters. some bootloaders would > need target info to know where to install config files, and > reinstall-bootloader doesn't have access to the operating-system > record. rollbacks to generations pre-field addition shouldn't be an > issue with this either, as existing bootloaders (sans depthcharge?) do > just fine without targets. > > all in all, bootloader-installer and bootloader-config-file-installer > would be procedures that return gexps, with the following signatures: > > * (installer entries #:key bootcfg mount-offset generation old-entries >store-crypto-devices store-directory-prefix locale) > * (config-file-installer entries #:key package targets mount-offset >generation old-entries store-crypto-devices store-directory-prefix >locale) > > mount-offset is the current target, and old-entries becomes an alist with > generation numbers. The current bootloader system started as grub only and then grew more capabilities for other things too. It should be possible to flash u-boot separately and then use grub-efi (or something like it) but it's not currently supported out of the box. I look forward to seeing how this might look. -- Efraim Flashner רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
