Hi, On Mon, 18 Mar 2024 at 18:25, Christophe Lyon <christophe.l...@linaro.org> wrote: > > On Sat, 16 Mar 2024 at 18:16, Simon Marchi <sim...@simark.ca> wrote: > > > > > > > > On 2024-03-15 04:50, Christophe Lyon via Gdb wrote: > > > On Thu, 14 Mar 2024 at 19:10, Simon Marchi <sim...@simark.ca> wrote: > > >> My first thought it: why is it a Makefile target, instead of some script > > >> on the side (like autoregen.sh). It would be nice / useful to be > > >> able to it without configuring / building anything. For instance, the > > >> autoregen buildbot job could run it without configuring anything. > > >> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the > > >> side, it would just use whatever is in the repo. > > > > > > Firstly because of what you mention later: some regeneration steps > > > require building host tools first, like the XXX-gen in opcodes. > > > > "build" and not "host", I think? > > yes, sorry > > > > Since the existing Makefiles already contain the rules to autoregen > > > all these files, it seemed natural to me to reuse them, to avoid > > > reinventing the wheel with the risk of introducing new bugs. > > > > I understand. Although one advantage of moving the actual code out of > > the Makefile (even if there's still a Makefile rule calling the external > > script), is that it's much easier to maintain. Editors are much more > > useful when editing a standalone shell script than editing shell code in > > a Makefile target. It doesn't have to be this big one liner if you want > > to use variables, you don't need to escape $, you can run it through > > linters, you can call it by hand, etc. This is what I did here, for > > instance: > > > > https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2 > > > > So I think there's value in any case of moving the regeneration logic > > out of the Makefiles per se. > > > In this case, the generation rules look simple enough indeed. > But as mentioned elsewhere in the thread, there are more complex > cases, which involve building helper tools, which have dependencies on > bfd and libiberty for instance. I'm not sure that's easily/naturally > scriptable? > There's also 'chew' in bfd/ > > > > This involves changes in places where I've never looked at before, so > > > I'd rather reuse as much existing support as possible. > > > > > > For instance, there are the generators in opcodes/, but also things in > > > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also > > > something "unusual" in fixincludes/ and libgfortran/ > > > > > > In fact, I considered also including 'configure', 'Makefile.in', > > > etc... in the 'regenerate' target, it does not seem natural to me to > > > invoke a script on the side, where you have to replicate the behaviour > > > of existing Makefiles, possibly getting out-of-sync when someone > > > forgets to update either Makefile or autoregen.py. > > > > I'm not sure I follow. Are you referring to the rules that automake > > automatically puts to re-generate Makefile.in and others when > > Makefile.am has changed? Your regenerate target would depend on those > > builtin rules? > Yes, "regenerate" would include "configure, Makeifile.in, configh" > (as/if needed) in its list of dependencies. > > > > > Let's say my generate-autostuff.sh script does: > > > > aclocal --some-flags > > automake --some-other-flags > > autoconf --some-other-other-flags > > > > And the checked-in Makefile.in is regenerated based on that. Wouldn't > > the built-in rules just call aclocal/automake/autoconf with those same > > flags? I don't see why they would get out of sync. > Well the rule to regenerate Makefile.in (eg in in opcodes/) is a bit > more complex > than just calling automake. IIUC it calls automake --foreign it any of > *.m4 file from $(am__configure_deps) that is newer than Makefile.in > (with an early exit in the loop), does nothing if Makefile.am or > doc/local.mk are newer than Makefile.in, and then calls 'automake > --foreign Makefile' > > I've never looked closely at that rule (I suppose he does what it's > intended to do ;-) ), but why not call automake once in $srcdir then > once in $top_srcdir? > TBH I'd rather not spend ages figuring out all this magic :-) > > But yeah, maybe some careful looking at these rules might lead to a > couple of simple shell lines. >
I looked a bit more closely at gcc, and noticed that ACLOCAL_AMFLAGS is given different values at various parts of the source tree: -I $(top_srcdir) -I $(top_srcdir)/config -I ../config -I ../config -I .. -I ./config -I ../config -I .. -I ../../config -I .. -I ../config -I ../.. -I ../../config -I . -I .. -I ../config -I m4 not sure if the current autoregen.py is in sync with that? Also... I discovered the existence of an automake rule: am--refresh which IIUC is intended to automake the update of Makefile and its dependencies. I'm by no means an autotool expert :-) Christophe > > > > > > What is currently > > > missing is a way to easily regenerate files without having to run a > > > full 'make all' (which currently takes care of calling autoconf & > > > friends to update configure/Makefile.in). > > > > > > But yeah, having to configure before being able to regenerate files is > > > a bit awkward too :-) > > > > I understand the constraints your are working with, and I guess that > > doing: > > > > ./configure && make regenerate > > > > is not too bad. The buildbot could probably do that... except that > > it would need a way to force regenerate everything, ignoring the > > timestamps. Perhaps this option of GNU make would work? > > > > -B, --always-make > > Unconditionally make all targets. > I noticed that option when writing my previous message, maybe that would work. > > > >> Looking at the rule to re-generate copying.c in gdb for instance: > > >> > > >> # Make copying.c from COPYING > > >> $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 > > >> $(srcdir)/copying.awk > > >> awk -f $(srcdir)/copying.awk \ > > >> < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp > > >> mv $(srcdir)/copying.tmp $(srcdir)/copying.c > > >> > > >> There is nothing in this code that requires having configured the source > > >> tree. This code could for instance be moved to some > > >> generate-copying-c.sh script. generate-copying-c.sh could be called by > > >> an hypothetical autoregen.sh script, as well as the copying.c Makefile > > >> target, if we want to continue supporting the maintainer mode. > > > Wouldn't it be more obscure than now? Currently such build rules are > > > all in the relevant Makefile. You'd have to open several scripts to > > > discover what's involved with updating copying.c > > > > Maybe, that's subjective :). The logic to regenerate would be in one > > script, and yes that script could be invoked from different places. At > > least the paper trail would be relatively easy to follow. > > > > >> Much like your regenerate targets, an autoregen.sh script in a given > > >> directory would be responsible to re-generate all the files in this > > >> directory that are generated and checked in git. It would also be > > >> responsible to call any autoregen.sh file in subdirectories. > > > Makefiles already have all that in place :-) > > > Except if you consider that you'd want to ignore timestamps and always > > > regenerate things? > > > > Yeah, for the buildbot autoregen job, since its job it to verify that > > everything has been generated correctly, we would like to regenerate > > everything, regardless of the timestamps. > The bot I want to put in place would regenerate things as they are > supposed to be, then build and run the testsuite to make sure that > what is supposed to be committed would work (if the committer > regenerates everything correctly) > > > > > Speaking of "already have all that in place", maintaining a script in > > the buildbot to re-implement all the re-generation logic is the worst of > > all, so I would love to avoid having to reimplemement some of that logic > > out of the repo :). > agreed! > autoregen.py should probably be moved from the bot to the binutils-gdb > and gcc repos (but we'd have to remember to keep them in sync, like > many other files already...) > > > >> compiled. When experimenting with maintainer mode the other day, I > > >> stumbled on the opcodes/i386-gen, for instance. I don't have a good > > >> solution to that, except to rewrite these tools in a scripting language > > >> like Python. > > > > > > So for opcodes, it currently means rewriting such programs for i386, > > > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c > > > generator. > > > Not sure how to find volunteers? > > > > I would love to do that if I had infinite time :). Of course, the > > current state of things + finite amount of resources are some contraints > > you are working with, and I completely undertand that. > > :-) > > Thanks, > > Christophe > > > > > Simon