Re: Future plans for Autotools
> "Gavin" == Gavin Smith writes: Gavin> I remember somebody was Gavin> complaining about this page: Gavin> https://www.gnu.org/software/automake/manual/html_node/Program-and-Library-Variables.html Gavin> and asking what "maude" meant - it turned out it was the name of the Gavin> dog or something of the person who first wrote the program, who didn't Gavin> want it to change. Yeah, that was my dog's name. Feel free to change it if you think that's important. I don't recall objecting, but I wouldn't put it past me. Tom
Re: Future plans for Autotools
On 1/21/21 8:01 AM, Zack Weinberg wrote: I know that at least one person has tried to write a set of GNU Make library files intended to replace it altogether, but I've never seen anyone *finish* that project. I'd very much like to see someone give that another go. GNU Emacs never used Automake but its developers eventually decided to require GNU Make. This has not proved to be a problem in practice, as GNU Make is ubiquitous, and Automake features not directly supported by GNU Make don't seem to be needed by Emacs builds. One possible way forward is to have an Autoconf 2 that builds atop GNU Make, both as a partial replacement for Automake (which is what Emacs does already), and as a way to speed up and simplify configuration. If 'configure' were mostly a front end to a GNU Make invocation, it could run configuration probes in parallel which would certainly be a win for me. And perhaps configuration probes could be written in GNU Make rather than m4, which would also be a win because it'd be one less language to learn. (Of course we could continue to support existing m4-based probes, run sequentially, as well as letting Automake do its thing for people who prefer Automake.) Whatever way forward is chosen will surely need coordination with Gnulib, which has essentially taken over most of the low-level system-specific porting tasks that Autoconf used to have. When Emacs adopted Gnulib but did not want to use Automake, we had to hack on Gnulib to support that; bigger hacks to Gnulib will surely be needed to support any of the changes proposed here.
Re: Future plans for Autotools
My two cents: the competing build systems, cmake, meson, have in their "features" (and major motivation for their original development) of supporting Xcode and Microsoft Visual Studio. Supporting for these seem to become necessary for GNU Autotools to compete, even if these two may be outside of interests for people who have traditionally worked on autotools? And supporting such may be distraction for developer resources?
Re: Future plans for Autotools
Zack, On Thu, Jan 21, 2021 at 9:12 AM Zack Weinberg wrote: > On Wed, Jan 20, 2021 at 5:15 PM Zack Weinberg wrote: > > Now we've all had a while to recover from the long-awaited Autoconf > > 2.70 release, I'd like to start a conversation about where the > > Autotools in general might be going in the future. > > > Now we've all had a while to recover from the long-awaited Autoconf > > 2.70 release, I'd like to start a conversation about where the > > Autotools in general might be going in the future. > > Thanks for all the great thoughtful responses that came in overnight! > I see many people expressing similar sentiments, so rather than > respond to each of you individually I'm going to summarize and reply > to each strand in this one message. > > I like the way you think. But, as your response so clearly indicates, there's actually very little we can change in a radical way: 1. We shouldn't change the underlying language(s) and tools because one of the key strengths of the Autotools is (their products') ability to run everywhere without off-box dependencies. In my humble opinion, nothing the GNU project has ever done comes close to the value they created by designing the Autotools to work as seamlessly (for the end-user) as they do. Download a tarball, unpack, ./configure && make. How much simpler could it be? No additional tools to install - everything required is already there - for the end user. Of course, developers may need to install a few additional tools to build these tarballs from the base source repository, but developers are not end-users and those are the "customers" GNU was trying to appease. If you build from source repositories, you're effectively a developer, not an end-user - not always the case, but 98% of the time this is true. Additionally, making such changes would require long man-hours of unpaid effort - something the world is not willing to give. Gone are the days of university grad students who know enough of what they're talking about to effectively take on this task for the sake of their educational goals (this is the way these tools started, after all). In fact, as much as they'd hate to admit it, the GNU project has become much more corporate than educational today. But let's not get bent out of shape by it - the truth of the matter is that no one does anything for free (you yourself solicited funding for the 2.70 update - which did NOT ultimately come from philanthropic individuals. 2. The Autotools are actually more transparent than any other build tools out there. All these other tools' (cmake, maven, etc) - that purport to be so much simpler because they insulate the user from the underlying details of the build process - these tool's primary failure is that this very insulation keeps users from being able to make the changes they need to accomplish their unique project-specific build goals. Anyone who has nothing but good things to say about this aspect of cmake, maven, gradle, or whatever, has simply not worked on a project that requires them to move far enough away from the defaults. I've used them all and I've spent hours in frustration trying to determine how to work around the shortcomings of some "do-all" (except what I want) tool function. This is simply not an issue with the Autotools. As someone mentioned earlier in this thread, you can drop shell script into a configure.ac file, and make script into a Makefile.am file. That is the very definition of transparency. No other tool in existence allows this level of flexibility. The most interesting feedback I read in the responses was that the Autotools had an appalling lack of documentation which - ironically - is actually not true. At the risk of sounding self-serving, I'll say this: in the research I did for my book (Autotools, No Starch Press, 2019), I primarily (95%) used the GNU Autotools documentation as reference sources. The information is all there and actually very concise - too concise sometimes. The problem with it is two fold. First, the documentation for GNU projects is most often written by the author of the code - that's the open source way, after all. This brings with it all sorts of issues that most corporate documentation (inadvertently) bypasses by standard corporate siloing protocol - that is, they relegate doc writing to doc writers, who are often not experts in the code, so they have to spend the time researching the product from a user's perspective. This is exactly the perfect storm of events that (usually) generates pretty good documentation. Second, the documentation is written in a manner that is not conducive to fast learning. There are two types of instructive material. The first is reference material that, by its very nature, requires the user to read and absorb significant amounts of information until they hit a critical mass of understanding - an exponential learning curve that starts out long and slow - at which point they avalanche into the steeper portion of the curve.
Re: Future plans for Autotools
On Thu, Jan 21, 2021 at 11:01:34AM -0500, Zack Weinberg wrote:
> Having said that, switching to *anything else* would be a gigantic
> task -- multiple full-time person-years of effort just for the core --
> and would mean either porting or losing all of the third-party macro
> libraries. I don't think it makes sense to explore this possibility
> any further unless both a lot of funding and a lot of developers
> materialize.
It could be possible for autoconf to become one option of several for
use with automake. Then it would not be necessary to replicate the
entire functionality of autoconf in a new system. People could have
(part of) the configuration written in Perl or other languages.
For this to work, automake would need to change quite a lot, for
example not always using configure.ac as an input file. There would
be the question of information sharing between automake's own checks and
checks done in user configuration code. It would never be trivial
to add a new configuration language, but they could all have similar
characteristics, for example communicating their results as a list
of names and values (i.e. the "name=@name@" of autoconf), some of which
would be required for Automake rules (e.g. whatever configuration method
was used, it would have to yield a value for CC if C files were being
compiled).
Perhaps there could be an Automake variable CONFIGURATION_METHOD
that could select which system was to used ("autoconf" being the
current system). Other options could be to generate the configure
script completely automatically, or to access other programs that
allow some user code in the script.
This all depends on somebody creating a replacement for autoconf, though,
and integrating it with automake.
> A lesser possibility we could think about, would be dusting off GNU M4
> 2.0 and thinking about what we could change about the M4 language to
> make it friendlier for Autoconf's purposes. Again I would want
> funding and developers lined up, but we'd not need quite so much of
> either.
If you ask if the current design of the autotools is something that
should be preserved forever, when you look at what its use is (creating
build systems for software), you would answer that it shouldn't be. M4
is part of this.
> [Bob Friesenhahn]
> > There is a fork of Automake which was re-done to be based on GNU
> > Make. This assumes that makefiles written in GNU make can noticeably
> > improve things. I like your idea of supporting other underlying
> > build tools besides 'make'. Make's dependence on matching rules and
> > file time stamps is problematic and it does not scale.
>
> I haven't ever followed automake development closely. I know that
> at least one person has tried to write a set of GNU Make library files
> intended to replace it altogether, but I've never seen anyone *finish*
> that project. I'd very much like to see someone give that another go.
>
> The alternative to the Make family of tools, that I'm most familiar
> with, is Ninja, and it also depends on matching rules and file time
> stamps. What do you have in mind as a replacement for these things?
> Is there already an alternative tool that you like?
One issue here is that Makefile.am can contain Makefile rules, which
transfer cleanly to use by "make", but possibly not for other programs
like ninja. (I don't know anything about ninja, though.)
> I have a lot more sympathy with people who are annoyed with the GNU
> *project* and/or the FSF. I have any number of my own grievances with
> them. However, I would prefer not to get into that in this
> conversation, because it's liable to drag it off-topic and/or turn
> into a flame war. Let's declare the known obstacles imposed on
> Autotools by the FSF -- concretely, the requirement for copyright
> assignment paperwork and the requirement not to use proprietary CI and
> forge services -- immobile and out-of-scope for this discussion.
There could be a discussion to be had about project governance and how
to stimulate development in GNU packages: advertising for contributors,
promoting use of the package, encouraging exploratory development. I
don't have any concrete suggestions, though, and many changes are also
possible for the worse.
Re: Future plans for Autotools
On Wed, Jan 20, 2021 at 5:15 PM Zack Weinberg wrote: > Now we've all had a while to recover from the long-awaited Autoconf > 2.70 release, I'd like to start a conversation about where the > Autotools in general might be going in the future. > Now we've all had a while to recover from the long-awaited Autoconf > 2.70 release, I'd like to start a conversation about where the > Autotools in general might be going in the future. Thanks for all the great thoughtful responses that came in overnight! I see many people expressing similar sentiments, so rather than respond to each of you individually I'm going to summarize and reply to each strand in this one message. > continuous integration for Autotools This is one of my highest priorities for what volunteer time I have in the next few months. The biggest difficulty with CI, as I see it, is that Autotools needs to be tested on a very wide variety of operating systems, and ideally *versions* of operating systems, and unusual compilers, and so on, in order to confirm that it's still fulfilling its contract. Testing just on the "big three" operating systems (Linux, Windows, MacOS) typically supported by commercial CI providers ... would be better than nothing, but would not have helped all that much with the 2.70 release, if I'm honest. GNU's Hydra instance (https://www.gnu.org/software/devel.en.html#Hydra -- I didn't previously know about this, thank you Dan for pointing it out) is a strong contender just because it has FreeBSD and Solaris as well. > implementation languages - can't we do better than shell+perl+m4? I deeply sympathize with frustration with M4 and portable shell as implementation languages. Perl isn't so bad IMO, but being restricted to perl 5.6.0 with no CPAN modules is also a headache. Having said that, switching to *anything else* would be a gigantic task -- multiple full-time person-years of effort just for the core -- and would mean either porting or losing all of the third-party macro libraries. I don't think it makes sense to explore this possibility any further unless both a lot of funding and a lot of developers materialize. A lesser possibility we could think about, would be dusting off GNU M4 2.0 and thinking about what we could change about the M4 language to make it friendlier for Autoconf's purposes. Again I would want funding and developers lined up, but we'd not need quite so much of either. And we probably could bump up the minimum version requirement for Perl quite some ways, since it's only needed at autoreconf time. I'm not sure how much Autoconf would gain from this, though, since it's only incrementally changed since 5.6 and Autoconf doesn't do *that* much with it. Automake uses it much more heavily, though, perhaps people from that project would like to comment? [Nate Bargmann] > One strength of the Autotools that I think stands above the rest is > the fact that a user of a distributed package does not need to > install any of the Autotools packages. I definitely agree with this, and I'm putting it right next to the "implementation languages" note because this is a compelling reason to stick to shell as the language for generated configure scripts. The only alternative that would come anywhere near as close in terms of "just unpack and run" portability is Perl. Which is a tempting possibility -- it would wipe out dozens of internal headaches due to shell's limitations and portability quirks -- but again, multiple full-time person-years of effort, losing the third-party macro libraries, and we *would* be making life harder for people starting from scratch on OSes that don't bundle Perl. > Autotools are too difficult to work on Yes indeed. On the assumption that we are stuck with shell, M4, and Perl, however, what can we do about it? I have a few ideas, maybe you have more: - A linter for configure.ac and third-party macro libraries. This wouldn't be any fun to write, because it would have to parse both M4 and shell, *accurately*, but it would be valuable to every user of autoconf. - The parser for the linter could also be used as the basis for a reimplementation of autoupdate that *doesn't* rely on running configure.ac through M4 with most of the macros disabled. That implementation choice is responsible for most of the bugs in autoupdate, and for most of its limitations (e.g. not being able to fix quotation errors). - Audit existing configure.ac's looking for pain points: where are people having trouble doing the checks they need, where are people not realizing that they don't need to work so hard, where are people not realizing that they don't need to check for that at all... - Audit the core for workarounds that are no longer necessary now we can rely on shell functions (I'm looking at you, AC_LANG_*). - Better documentation and better debugging tools. Speaking of which, here's an interesting bit from Russell: [Russell Shaw] > The biggest problem i had that wasted t
Re: Future plans for Autotools
On 21/1/21 9:15 am, Zack Weinberg wrote: Now we've all had a while to recover from the long-awaited Autoconf 2.70 release, I'd like to start a conversation about where the Autotools in general might be going in the future. Clearly any future development depends on finding people who will do the work, but before we worry about that I think we might want to figure out what work we _want_ to do. ... Agree with all that. The biggest problem i had that wasted the most time is that none of the documentation gave a good overview of the sequence of steps the main perl script did to process the files and produce the output, and intermediate files involved. Using a language like C++ where a step-by-step debugger like gdb can be used would have been good. Having a nuts and bolts low level detail of how it all works makes one much more productive. I'm talking of real software engineering ppl that understand compilers and not programmers that are really artist dabbling users. It is the real engineers that are more motivated to understand and be contributors. As a result, i only gradually got proficient over a few years, and having to comprehend a large manual on ancient shell variants and study M4 didn't help. M4 is easy if there's a decent tutorial readily available. Currently one needs to download other manuals and tutorials to get a good idea of it, and it's a big step for starting programmers. I understand mostly how things work, but got bogged down in understanding the perl script. If i mastered it all, i'd think more about what can be changed or improved. I did one early iteration of a tool that did the functions of 'make' using an input description of the files one wanted in a project. It also had its own shell implementation. Users would need a copy of that shell installed. An idea was to have the users script automatically build and install the shell in a location the user or system dictated. Another option was an automatic download from a trusted place. I could get more done if i had another go at it, but i'm too busy on a different tool. If GNU paperwork gets in the way, just bypass it and release new stuff as BSD. Then it's truly free.
