Re: Help needed with maintainer-mode

[Richard Earnshaw (lists) via Gcc]

On 06/03/2024 15:04, Andrew Carlotti via Gcc wrote:
> On Thu, Feb 29, 2024 at 06:39:54PM +0100, Christophe Lyon via Gcc wrote:
>> On Thu, 29 Feb 2024 at 12:00, Mark Wielaard  wrote:
>>>
>>> Hi Christophe,
>>>
>>> On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
 I've noticed that sourceware's buildbot has a small script
 "autoregen.py" which does not use the project's build system, but
 rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
 Should we replicate that?
>>>
>>> That python script works across gcc/binutils/gdb:
>>> https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
>>>
>>> It is installed into a container file that has the exact autoconf and
>>> automake version needed to regenerate the autotool files:
>>> https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
>>>
>>> And it was indeed done this way because that way the files are
>>> regenerated in a reproducible way. Which wasn't the case when using 
>>> --enable-maintainer-mode (and autoreconfig also doesn't work).
>>
>> I see. So it is possibly incomplete, in the sense that it may lack
>> some of the steps that maintainer-mode would perform?
>> For instance, gas for aarch64 has some *opcodes*.c files that need
>> regenerating before committing. The regeneration step is enabled in
>> maintainer-mode, so I guess the autoregen bots on Sourceware would
>> miss a problem with these files?
>>
>> Thanks,
>>
>> Christophe
> 
> Speaking of opcodes/aarch64-{asm|dis|opc}-2.c - why are these in the source
> directory in the first place?  For a similar situation in GCC (gimple-match,
> generic-match, insn-emit, etc.) we write the generated files to the build
> directory, and generation is always enabled.  I see no reason not to do the
> same thing for aarch64-{asm|dis|opc}-2.c.
> 

GCC supports building a canadian cross, but binutils doesn't.  To put them in 
the build area would require setting up host compiler as well and I don't think 
there's currently enough appetite for doing that extra work.

But every do has its day; maybe the time has come...

R.

> Andrew
> 
>>>
>>> It is run on all commits and warns if it detects a change in the
>>> (checked in) generated files.
>>> https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
>>> https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
>>>
>>> Cheers,
>>>
>>> Mark

Re: Help needed with maintainer-mode

[Andrew Carlotti via Gcc]

On Thu, Feb 29, 2024 at 06:39:54PM +0100, Christophe Lyon via Gcc wrote:
> On Thu, 29 Feb 2024 at 12:00, Mark Wielaard  wrote:
> >
> > Hi Christophe,
> >
> > On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
> > > I've noticed that sourceware's buildbot has a small script
> > > "autoregen.py" which does not use the project's build system, but
> > > rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> > > Should we replicate that?
> >
> > That python script works across gcc/binutils/gdb:
> > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> >
> > It is installed into a container file that has the exact autoconf and
> > automake version needed to regenerate the autotool files:
> > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> >
> > And it was indeed done this way because that way the files are
> > regenerated in a reproducible way. Which wasn't the case when using 
> > --enable-maintainer-mode (and autoreconfig also doesn't work).
> 
> I see. So it is possibly incomplete, in the sense that it may lack
> some of the steps that maintainer-mode would perform?
> For instance, gas for aarch64 has some *opcodes*.c files that need
> regenerating before committing. The regeneration step is enabled in
> maintainer-mode, so I guess the autoregen bots on Sourceware would
> miss a problem with these files?
> 
> Thanks,
> 
> Christophe

Speaking of opcodes/aarch64-{asm|dis|opc}-2.c - why are these in the source
directory in the first place?  For a similar situation in GCC (gimple-match,
generic-match, insn-emit, etc.) we write the generated files to the build
directory, and generation is always enabled.  I see no reason not to do the
same thing for aarch64-{asm|dis|opc}-2.c.

Andrew

> >
> > It is run on all commits and warns if it detects a change in the
> > (checked in) generated files.
> > https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
> > https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
> >
> > Cheers,
> >
> > Mark

Re: Help needed with maintainer-mode

[Richard Earnshaw via Gcc]

On 05/03/2024 14:26, Richard Earnshaw (lists) wrote:
> On 04/03/2024 20:04, Jonathan Wakely wrote:
>> On Mon, 4 Mar 2024 at 19:27, Vladimir Mezentsev
>>  wrote:
>>>
>>>
>>>
>>> On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
 Tools like git (and svn before it) don't try to maintain time-stamps on 
 patches, the tool just modifies the file and the timestamp comes from the 
 time of the modification.  That's fine if there is nothing regenerated 
 within the repository (it's pure original source), but will cause problems 
 if there are generated files as their time stamps aren't necessarily 
 correct.  `gcc_update --touch` addresses that by ensuring all the 
 generated files are retouched when needed.
>>>
>>> Why do we save generated files in the source tree?
>>> What will be the problem if we remove Makefile.in and configure from
>>> source tree and will run `autoreconf -i -f` before building ?
>> 
>> Having the exact correct versions of autoconf and automake increases
>> the barrier for new contributors to start work. And to regenerate
>> everything, they also need autogen, mkinfo, etc.
> 
> It's worse than that.  They might need multiple versions of those tools 
> because different subtrees are built with different, subtly incompatible, 
> versions of those tools.
> 
> R.
> 

And I've just remembered another reason as well, which is that some people want 
to store their sources in a read-only environment; having the tools write to 
the source area during a build can cause problems (eg if building multiple 
configurations of the compiler in parallel).

R.

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists) via Gcc]

On 04/03/2024 20:04, Jonathan Wakely wrote:
> On Mon, 4 Mar 2024 at 19:27, Vladimir Mezentsev
>  wrote:
>>
>>
>>
>> On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
>>> Tools like git (and svn before it) don't try to maintain time-stamps on 
>>> patches, the tool just modifies the file and the timestamp comes from the 
>>> time of the modification.  That's fine if there is nothing regenerated 
>>> within the repository (it's pure original source), but will cause problems 
>>> if there are generated files as their time stamps aren't necessarily 
>>> correct.  `gcc_update --touch` addresses that by ensuring all the generated 
>>> files are retouched when needed.
>>
>> Why do we save generated files in the source tree?
>> What will be the problem if we remove Makefile.in and configure from
>> source tree and will run `autoreconf -i -f` before building ?
> 
> Having the exact correct versions of autoconf and automake increases
> the barrier for new contributors to start work. And to regenerate
> everything, they also need autogen, mkinfo, etc.

It's worse than that.  They might need multiple versions of those tools because 
different subtrees are built with different, subtly incompatible, versions of 
those tools.

R.

Re: Help needed with maintainer-mode

[Jonathan Wakely via Gcc]

On Mon, 4 Mar 2024 at 19:27, Vladimir Mezentsev
 wrote:
>
>
>
> On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
> > Tools like git (and svn before it) don't try to maintain time-stamps on 
> > patches, the tool just modifies the file and the timestamp comes from the 
> > time of the modification.  That's fine if there is nothing regenerated 
> > within the repository (it's pure original source), but will cause problems 
> > if there are generated files as their time stamps aren't necessarily 
> > correct.  `gcc_update --touch` addresses that by ensuring all the generated 
> > files are retouched when needed.
>
> Why do we save generated files in the source tree?
> What will be the problem if we remove Makefile.in and configure from
> source tree and will run `autoreconf -i -f` before building ?

Having the exact correct versions of autoconf and automake increases
the barrier for new contributors to start work. And to regenerate
everything, they also need autogen, mkinfo, etc.

Re: Help needed with maintainer-mode

[Vladimir Mezentsev via Gcc]

On 3/4/24 09:38, Richard Earnshaw (lists) wrote:

Tools like git (and svn before it) don't try to maintain time-stamps on 
patches, the tool just modifies the file and the timestamp comes from the time 
of the modification.  That's fine if there is nothing regenerated within the 
repository (it's pure original source), but will cause problems if there are 
generated files as their time stamps aren't necessarily correct.  `gcc_update 
--touch` addresses that by ensuring all the generated files are retouched when 
needed.


Why do we save generated files in the source tree?
What will be the problem if we remove Makefile.in and configure from 
source tree and will run `autoreconf -i -f` before building ?

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists) via Gcc]

On 04/03/2024 16:42, Christophe Lyon wrote:
> On Mon, 4 Mar 2024 at 16:41, Richard Earnshaw  
> wrote:
>>
>>
>>
>> On 04/03/2024 15:36, Richard Earnshaw (lists) wrote:
>> > On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
>> >> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely  
>> >> wrote:
>> >>>
>> >>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc  
>> >>> wrote:
>> 
>>  Hi!
>> 
>>  On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  
>>  wrote:
>> >
>> > Hi!
>> >
>> > On 2024-03-04T00:30:05+, Sam James  wrote:
>> >> Mark Wielaard  writes:
>> >>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>>  [...], I read
>>  https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration 
>>   
>>  >  >
>>  which basically says "run autoreconf in every dir where there is a
>>  configure script"
>>  but this is not exactly what autoregen.py is doing. IIRC it is based
>>  on a script from Martin Liska, do you know/remember why it follows a
>>  different process?
>> >>>
>> >>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>> >>> might remember more details. We wanted a script that worked for both
>> >>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>> >>> work in all directories. We also needed to skip some directories that
>> >>> did contain a configure script, but that were imported (gotools,
>> >>> readline, minizip).
>> >>
>> >> What we really need to do is, for a start, land tschwinge/aoliva's 
>> >> patches [0]
>> >> for AC_CONFIG_SUBDIRS.
>> >
>> > Let me allocate some time this week to get that one completed.
>> >
>> >> Right now, the current situation is janky and it's nowhere near
>> >> idiomatic autotools usage. It is not a comfortable experience
>> >> interacting with it even as someone who is familiar and happy with 
>> >> using
>> >> autotools otherwise.
>> >>
>> >> I didn't yet play with maintainer-mode myself but I also didn't see 
>> >> much
>> >> point given I knew of more fundamental problems like this.
>> >>
>> >> [0] 
>> >> https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/
>> >>  
>> >> 
>> >>  
>> >> > >>  
>> >> >
>> >
>> 
>>  Thanks for the background. I didn't follow that discussion at that time 
>>  :-)
>> 
>>  So... I was confused because I noticed many warnings when doing a simple
>>  find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
>>  autoreconf -f $d && echo $d; done
>>  as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration 
>>   
>>  >  >
>> 
>>  Then I tried with autoregen.py, and saw the same and now just
>>  checked Sourceware's bot logs and saw the same numerous warnings at
>>  least in GCC (didn't check binutils yet). Looks like this is
>>  "expected" 
>> 
>>  I started looking at auto-regenerating these files in our CI a couple
>>  of weeks ago, after we received several "complaints" from contributors
>>  saying that our precommit CI was useless / bothering since it didn't
>>  regenerate files, leading to false alarms.
>>  But now I'm wondering how such contributors regenerate the files
>>  impacted by their patches before committing, they probably just
>>  regenerate things in their subdir of interest, not noticing the whole
>>  picture :-(
>> 
>>  As a first step, we can probably use autoregen.py too, and declare
>>  maintainer-mode broken. However, I do notice that besides the rules
>>  about regenerating configure/Makefile.in/..., maintainer-mode is also
>>  used to update some files.
>>  In gcc:
>>  fixincludes: fixincl.x
>>  libffi: doc/version.texi
>>  libgfortran: some stuff :-)
>>  libiberty: functions.texi
>> >>>
>> >>> My recently proposed patch adds the first of those to gcc_update, the
>> >>> other should be done too.
>> >>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html 
>> >>>  
>> >>> > >>> 

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Mon, 4 Mar 2024 at 16:41, Richard Earnshaw  wrote:
>
>
>
> On 04/03/2024 15:36, Richard Earnshaw (lists) wrote:
> > On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
> >> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely  wrote:
> >>>
> >>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc  
> >>> wrote:
> 
>  Hi!
> 
>  On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  
>  wrote:
> >
> > Hi!
> >
> > On 2024-03-04T00:30:05+, Sam James  wrote:
> >> Mark Wielaard  writes:
> >>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>  [...], I read
>  https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration 
>  
>  which basically says "run autoreconf in every dir where there is a
>  configure script"
>  but this is not exactly what autoregen.py is doing. IIRC it is based
>  on a script from Martin Liska, do you know/remember why it follows a
>  different process?
> >>>
> >>> CCing Sam and Arsen who helped refine the autoregen.py script, who
> >>> might remember more details. We wanted a script that worked for both
> >>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> >>> work in all directories. We also needed to skip some directories that
> >>> did contain a configure script, but that were imported (gotools,
> >>> readline, minizip).
> >>
> >> What we really need to do is, for a start, land tschwinge/aoliva's 
> >> patches [0]
> >> for AC_CONFIG_SUBDIRS.
> >
> > Let me allocate some time this week to get that one completed.
> >
> >> Right now, the current situation is janky and it's nowhere near
> >> idiomatic autotools usage. It is not a comfortable experience
> >> interacting with it even as someone who is familiar and happy with 
> >> using
> >> autotools otherwise.
> >>
> >> I didn't yet play with maintainer-mode myself but I also didn't see 
> >> much
> >> point given I knew of more fundamental problems like this.
> >>
> >> [0] 
> >> https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/
> >>  
> >> 
> >
> 
>  Thanks for the background. I didn't follow that discussion at that time 
>  :-)
> 
>  So... I was confused because I noticed many warnings when doing a simple
>  find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
>  autoreconf -f $d && echo $d; done
>  as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration 
>  
> 
>  Then I tried with autoregen.py, and saw the same and now just
>  checked Sourceware's bot logs and saw the same numerous warnings at
>  least in GCC (didn't check binutils yet). Looks like this is
>  "expected" 
> 
>  I started looking at auto-regenerating these files in our CI a couple
>  of weeks ago, after we received several "complaints" from contributors
>  saying that our precommit CI was useless / bothering since it didn't
>  regenerate files, leading to false alarms.
>  But now I'm wondering how such contributors regenerate the files
>  impacted by their patches before committing, they probably just
>  regenerate things in their subdir of interest, not noticing the whole
>  picture :-(
> 
>  As a first step, we can probably use autoregen.py too, and declare
>  maintainer-mode broken. However, I do notice that besides the rules
>  about regenerating configure/Makefile.in/..., maintainer-mode is also
>  used to update some files.
>  In gcc:
>  fixincludes: fixincl.x
>  libffi: doc/version.texi
>  libgfortran: some stuff :-)
>  libiberty: functions.texi
> >>>
> >>> My recently proposed patch adds the first of those to gcc_update, the
> >>> other should be done too.
> >>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html 
> >>> 
> >>>
> >>
> >> This script touches files such that they appear more recent than their
> >> dependencies,
> >> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
> >> For auto* files, this is "fine" as we can run autoreconf or
> >> autoregen.py before starting configure+build, but what about other
> >> files?
> >> For instance, if we have to test a patch which implies changes to
> >> fixincludes/fixincl.x, how should we proceed?
> >> 1- git checkout (with possibly "wrong" timestamps)
> >> 2- apply patch-to-test
> >> 3- contrib/gcc_update -t
> >> 4- configure --enable-maintainer-mode
> >
> > If you ran
> >
> > git reset --hard master // restore state to 'master'
> > contrib/gcc_update // pull latest code
> >
> > then 

Re: Help needed with maintainer-mode

[Richard Earnshaw via Gcc]

On 04/03/2024 15:36, Richard Earnshaw (lists) wrote:
> On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
>> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely  wrote:
>>>
>>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc  
>>> wrote:

 Hi!

 On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  
 wrote:
>
> Hi!
>
> On 2024-03-04T00:30:05+, Sam James  wrote:
>> Mark Wielaard  writes:
>>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
 [...], I read
 https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration 
 
 which basically says "run autoreconf in every dir where there is a
 configure script"
 but this is not exactly what autoregen.py is doing. IIRC it is based
 on a script from Martin Liska, do you know/remember why it follows a
 different process?
>>>
>>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>>> might remember more details. We wanted a script that worked for both
>>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>>> work in all directories. We also needed to skip some directories that
>>> did contain a configure script, but that were imported (gotools,
>>> readline, minizip).
>>
>> What we really need to do is, for a start, land tschwinge/aoliva's 
>> patches [0]
>> for AC_CONFIG_SUBDIRS.
>
> Let me allocate some time this week to get that one completed.
>
>> Right now, the current situation is janky and it's nowhere near
>> idiomatic autotools usage. It is not a comfortable experience
>> interacting with it even as someone who is familiar and happy with using
>> autotools otherwise.
>>
>> I didn't yet play with maintainer-mode myself but I also didn't see much
>> point given I knew of more fundamental problems like this.
>>
>> [0] 
>> https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/
>>  
>> 
>

 Thanks for the background. I didn't follow that discussion at that time :-)

 So... I was confused because I noticed many warnings when doing a simple
 find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
 autoreconf -f $d && echo $d; done
 as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration 
 

 Then I tried with autoregen.py, and saw the same and now just
 checked Sourceware's bot logs and saw the same numerous warnings at
 least in GCC (didn't check binutils yet). Looks like this is
 "expected" 

 I started looking at auto-regenerating these files in our CI a couple
 of weeks ago, after we received several "complaints" from contributors
 saying that our precommit CI was useless / bothering since it didn't
 regenerate files, leading to false alarms.
 But now I'm wondering how such contributors regenerate the files
 impacted by their patches before committing, they probably just
 regenerate things in their subdir of interest, not noticing the whole
 picture :-(

 As a first step, we can probably use autoregen.py too, and declare
 maintainer-mode broken. However, I do notice that besides the rules
 about regenerating configure/Makefile.in/..., maintainer-mode is also
 used to update some files.
 In gcc:
 fixincludes: fixincl.x
 libffi: doc/version.texi
 libgfortran: some stuff :-)
 libiberty: functions.texi
>>>
>>> My recently proposed patch adds the first of those to gcc_update, the
>>> other should be done too.
>>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html 
>>> 
>>>
>> 
>> This script touches files such that they appear more recent than their
>> dependencies,
>> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
>> For auto* files, this is "fine" as we can run autoreconf or
>> autoregen.py before starting configure+build, but what about other
>> files?
>> For instance, if we have to test a patch which implies changes to
>> fixincludes/fixincl.x, how should we proceed?
>> 1- git checkout (with possibly "wrong" timestamps)
>> 2- apply patch-to-test
>> 3- contrib/gcc_update -t
>> 4- configure --enable-maintainer-mode
> 
> If you ran
> 
> git reset --hard master // restore state to 'master'
> contrib/gcc_update // pull latest code
> 
> then anything coming from upstream will be touched automatically.  You really 
> don't want to re-touch the files after patching unless you're sure they've 
> all been patched correctly, it will break if there's anything regenerated 
> that's missing.
> 
> R.

Alternatively, if you did 

git reset --hard 

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists) via Gcc]

On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely  wrote:
>>
>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc  wrote:
>>>
>>> Hi!
>>>
>>> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  wrote:

 Hi!

 On 2024-03-04T00:30:05+, Sam James  wrote:
> Mark Wielaard  writes:
>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>>> [...], I read
>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>>> which basically says "run autoreconf in every dir where there is a
>>> configure script"
>>> but this is not exactly what autoregen.py is doing. IIRC it is based
>>> on a script from Martin Liska, do you know/remember why it follows a
>>> different process?
>>
>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>> might remember more details. We wanted a script that worked for both
>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>> work in all directories. We also needed to skip some directories that
>> did contain a configure script, but that were imported (gotools,
>> readline, minizip).
>
> What we really need to do is, for a start, land tschwinge/aoliva's 
> patches [0]
> for AC_CONFIG_SUBDIRS.

 Let me allocate some time this week to get that one completed.

> Right now, the current situation is janky and it's nowhere near
> idiomatic autotools usage. It is not a comfortable experience
> interacting with it even as someone who is familiar and happy with using
> autotools otherwise.
>
> I didn't yet play with maintainer-mode myself but I also didn't see much
> point given I knew of more fundamental problems like this.
>
> [0] 
> https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/

>>>
>>> Thanks for the background. I didn't follow that discussion at that time :-)
>>>
>>> So... I was confused because I noticed many warnings when doing a simple
>>> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
>>> autoreconf -f $d && echo $d; done
>>> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>>>
>>> Then I tried with autoregen.py, and saw the same and now just
>>> checked Sourceware's bot logs and saw the same numerous warnings at
>>> least in GCC (didn't check binutils yet). Looks like this is
>>> "expected" 
>>>
>>> I started looking at auto-regenerating these files in our CI a couple
>>> of weeks ago, after we received several "complaints" from contributors
>>> saying that our precommit CI was useless / bothering since it didn't
>>> regenerate files, leading to false alarms.
>>> But now I'm wondering how such contributors regenerate the files
>>> impacted by their patches before committing, they probably just
>>> regenerate things in their subdir of interest, not noticing the whole
>>> picture :-(
>>>
>>> As a first step, we can probably use autoregen.py too, and declare
>>> maintainer-mode broken. However, I do notice that besides the rules
>>> about regenerating configure/Makefile.in/..., maintainer-mode is also
>>> used to update some files.
>>> In gcc:
>>> fixincludes: fixincl.x
>>> libffi: doc/version.texi
>>> libgfortran: some stuff :-)
>>> libiberty: functions.texi
>>
>> My recently proposed patch adds the first of those to gcc_update, the
>> other should be done too.
>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html
>>
> 
> This script touches files such that they appear more recent than their
> dependencies,
> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
> For auto* files, this is "fine" as we can run autoreconf or
> autoregen.py before starting configure+build, but what about other
> files?
> For instance, if we have to test a patch which implies changes to
> fixincludes/fixincl.x, how should we proceed?
> 1- git checkout (with possibly "wrong" timestamps)
> 2- apply patch-to-test
> 3- contrib/gcc_update -t
> 4- configure --enable-maintainer-mode

If you ran

git reset --hard master // restore state to 'master'
contrib/gcc_update // pull latest code

then anything coming from upstream will be touched automatically.  You really 
don't want to re-touch the files after patching unless you're sure they've all 
been patched correctly, it will break if there's anything regenerated that's 
missing.

R.

> 
> I guess --enable-maintainer-mode would be largely (if not completely)
> disabled by step 3 above?
> And we should probably swap steps 2 and 3, so that the effects of
> patch-to-test are handled by make?
> 
> Thanks,
> 
> Christophe
> 
>>
>>
>>>
>>> in binutils/bfd:
>>> gdb/sim
>>> bfd/po/SRC-POTFILES.in
>>> bfd/po/BLD-POTFILES.in
>>> bfd/bfd-in2.h
>>> bfd/libbfd.h
>>> bfd/libcoff.h
>>> binutils/po/POTFILES.in
>>> gas/po/POTFILES.in
>>> opcodes/i386*.h
>>> gdb/copying.c
>>> gdb/data-directory/*.xml
>>> gold/po/POTFILES.in

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely  wrote:
>
> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc  wrote:
> >
> > Hi!
> >
> > On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  wrote:
> > >
> > > Hi!
> > >
> > > On 2024-03-04T00:30:05+, Sam James  wrote:
> > > > Mark Wielaard  writes:
> > > >> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> > > >>> [...], I read
> > > >>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> > > >>> which basically says "run autoreconf in every dir where there is a
> > > >>> configure script"
> > > >>> but this is not exactly what autoregen.py is doing. IIRC it is based
> > > >>> on a script from Martin Liska, do you know/remember why it follows a
> > > >>> different process?
> > > >>
> > > >> CCing Sam and Arsen who helped refine the autoregen.py script, who
> > > >> might remember more details. We wanted a script that worked for both
> > > >> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> > > >> work in all directories. We also needed to skip some directories that
> > > >> did contain a configure script, but that were imported (gotools,
> > > >> readline, minizip).
> > > >
> > > > What we really need to do is, for a start, land tschwinge/aoliva's 
> > > > patches [0]
> > > > for AC_CONFIG_SUBDIRS.
> > >
> > > Let me allocate some time this week to get that one completed.
> > >
> > > > Right now, the current situation is janky and it's nowhere near
> > > > idiomatic autotools usage. It is not a comfortable experience
> > > > interacting with it even as someone who is familiar and happy with using
> > > > autotools otherwise.
> > > >
> > > > I didn't yet play with maintainer-mode myself but I also didn't see much
> > > > point given I knew of more fundamental problems like this.
> > > >
> > > > [0] 
> > > > https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/
> > >
> >
> > Thanks for the background. I didn't follow that discussion at that time :-)
> >
> > So... I was confused because I noticed many warnings when doing a simple
> > find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
> > autoreconf -f $d && echo $d; done
> > as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> >
> > Then I tried with autoregen.py, and saw the same and now just
> > checked Sourceware's bot logs and saw the same numerous warnings at
> > least in GCC (didn't check binutils yet). Looks like this is
> > "expected" 
> >
> > I started looking at auto-regenerating these files in our CI a couple
> > of weeks ago, after we received several "complaints" from contributors
> > saying that our precommit CI was useless / bothering since it didn't
> > regenerate files, leading to false alarms.
> > But now I'm wondering how such contributors regenerate the files
> > impacted by their patches before committing, they probably just
> > regenerate things in their subdir of interest, not noticing the whole
> > picture :-(
> >
> > As a first step, we can probably use autoregen.py too, and declare
> > maintainer-mode broken. However, I do notice that besides the rules
> > about regenerating configure/Makefile.in/..., maintainer-mode is also
> > used to update some files.
> > In gcc:
> > fixincludes: fixincl.x
> > libffi: doc/version.texi
> > libgfortran: some stuff :-)
> > libiberty: functions.texi
>
> My recently proposed patch adds the first of those to gcc_update, the
> other should be done too.
> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html
>

This script touches files such that they appear more recent than their
dependencies,
so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
For auto* files, this is "fine" as we can run autoreconf or
autoregen.py before starting configure+build, but what about other
files?
For instance, if we have to test a patch which implies changes to
fixincludes/fixincl.x, how should we proceed?
1- git checkout (with possibly "wrong" timestamps)
2- apply patch-to-test
3- contrib/gcc_update -t
4- configure --enable-maintainer-mode

I guess --enable-maintainer-mode would be largely (if not completely)
disabled by step 3 above?
And we should probably swap steps 2 and 3, so that the effects of
patch-to-test are handled by make?

Thanks,

Christophe

>
>
> >
> > in binutils/bfd:
> > gdb/sim
> > bfd/po/SRC-POTFILES.in
> > bfd/po/BLD-POTFILES.in
> > bfd/bfd-in2.h
> > bfd/libbfd.h
> > bfd/libcoff.h
> > binutils/po/POTFILES.in
> > gas/po/POTFILES.in
> > opcodes/i386*.h
> > gdb/copying.c
> > gdb/data-directory/*.xml
> > gold/po/POTFILES.in
> > gprof/po/POTFILES.in
> > gfprofng/doc/version.texi
> > ld/po/SRC-POTFILES.in
> > ld/po/BLD-POTFILES.in
> > ld: ldgram/ldlex... and all emulation sources
> > libiberty/functions.texi
> > opcodes/po/POTFILES.in
> > opcodes/aarch64-{asm,dis,opc}-2.c
> > opcodes/ia64 msp430 rl78 rx z8k decoders
> >
> > How are these files "normally" expected to be updated? Do people just
> > manually uncomment the 

Re: Help needed with maintainer-mode

[Jonathan Wakely via Gcc]

On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc  wrote:
>
> Hi!
>
> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  wrote:
> >
> > Hi!
> >
> > On 2024-03-04T00:30:05+, Sam James  wrote:
> > > Mark Wielaard  writes:
> > >> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> > >>> [...], I read
> > >>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> > >>> which basically says "run autoreconf in every dir where there is a
> > >>> configure script"
> > >>> but this is not exactly what autoregen.py is doing. IIRC it is based
> > >>> on a script from Martin Liska, do you know/remember why it follows a
> > >>> different process?
> > >>
> > >> CCing Sam and Arsen who helped refine the autoregen.py script, who
> > >> might remember more details. We wanted a script that worked for both
> > >> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> > >> work in all directories. We also needed to skip some directories that
> > >> did contain a configure script, but that were imported (gotools,
> > >> readline, minizip).
> > >
> > > What we really need to do is, for a start, land tschwinge/aoliva's 
> > > patches [0]
> > > for AC_CONFIG_SUBDIRS.
> >
> > Let me allocate some time this week to get that one completed.
> >
> > > Right now, the current situation is janky and it's nowhere near
> > > idiomatic autotools usage. It is not a comfortable experience
> > > interacting with it even as someone who is familiar and happy with using
> > > autotools otherwise.
> > >
> > > I didn't yet play with maintainer-mode myself but I also didn't see much
> > > point given I knew of more fundamental problems like this.
> > >
> > > [0] 
> > > https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/
> >
>
> Thanks for the background. I didn't follow that discussion at that time :-)
>
> So... I was confused because I noticed many warnings when doing a simple
> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
> autoreconf -f $d && echo $d; done
> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>
> Then I tried with autoregen.py, and saw the same and now just
> checked Sourceware's bot logs and saw the same numerous warnings at
> least in GCC (didn't check binutils yet). Looks like this is
> "expected" 
>
> I started looking at auto-regenerating these files in our CI a couple
> of weeks ago, after we received several "complaints" from contributors
> saying that our precommit CI was useless / bothering since it didn't
> regenerate files, leading to false alarms.
> But now I'm wondering how such contributors regenerate the files
> impacted by their patches before committing, they probably just
> regenerate things in their subdir of interest, not noticing the whole
> picture :-(
>
> As a first step, we can probably use autoregen.py too, and declare
> maintainer-mode broken. However, I do notice that besides the rules
> about regenerating configure/Makefile.in/..., maintainer-mode is also
> used to update some files.
> In gcc:
> fixincludes: fixincl.x
> libffi: doc/version.texi
> libgfortran: some stuff :-)
> libiberty: functions.texi

My recently proposed patch adds the first of those to gcc_update, the
other should be done too.
https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html



>
> in binutils/bfd:
> gdb/sim
> bfd/po/SRC-POTFILES.in
> bfd/po/BLD-POTFILES.in
> bfd/bfd-in2.h
> bfd/libbfd.h
> bfd/libcoff.h
> binutils/po/POTFILES.in
> gas/po/POTFILES.in
> opcodes/i386*.h
> gdb/copying.c
> gdb/data-directory/*.xml
> gold/po/POTFILES.in
> gprof/po/POTFILES.in
> gfprofng/doc/version.texi
> ld/po/SRC-POTFILES.in
> ld/po/BLD-POTFILES.in
> ld: ldgram/ldlex... and all emulation sources
> libiberty/functions.texi
> opcodes/po/POTFILES.in
> opcodes/aarch64-{asm,dis,opc}-2.c
> opcodes/ia64 msp430 rl78 rx z8k decoders
>
> How are these files "normally" expected to be updated? Do people just
> manually uncomment the corresponding maintainer lines in the Makefiles
> and update manually?   In particular we hit issues several times with
> files under opcodes, that we don't regenerate currently. Maybe we
> could build binutils in maintainer-mode at -j1 but, well
>
> README-maintainer-mode in binutils/gdb only mentions a problem with
> 'make distclean' and maintainer mode
> binutils/README-how-to-make-a-release indicates to use
> --enable-maintainer-mode, and the sample 'make' invocations do not
> include any -j flag, is that an indication that only -j1 is supposed
> to work?
> Similarly, the src-release.sh script does not use -j.
>
> Thanks,
>
> Christophe
>
> >
> > Grüße
> >  Thomas

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

Hi!

On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge  wrote:
>
> Hi!
>
> On 2024-03-04T00:30:05+, Sam James  wrote:
> > Mark Wielaard  writes:
> >> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> >>> [...], I read
> >>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> >>> which basically says "run autoreconf in every dir where there is a
> >>> configure script"
> >>> but this is not exactly what autoregen.py is doing. IIRC it is based
> >>> on a script from Martin Liska, do you know/remember why it follows a
> >>> different process?
> >>
> >> CCing Sam and Arsen who helped refine the autoregen.py script, who
> >> might remember more details. We wanted a script that worked for both
> >> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> >> work in all directories. We also needed to skip some directories that
> >> did contain a configure script, but that were imported (gotools,
> >> readline, minizip).
> >
> > What we really need to do is, for a start, land tschwinge/aoliva's patches 
> > [0]
> > for AC_CONFIG_SUBDIRS.
>
> Let me allocate some time this week to get that one completed.
>
> > Right now, the current situation is janky and it's nowhere near
> > idiomatic autotools usage. It is not a comfortable experience
> > interacting with it even as someone who is familiar and happy with using
> > autotools otherwise.
> >
> > I didn't yet play with maintainer-mode myself but I also didn't see much
> > point given I knew of more fundamental problems like this.
> >
> > [0] 
> > https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/
>

Thanks for the background. I didn't follow that discussion at that time :-)

So... I was confused because I noticed many warnings when doing a simple
find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
autoreconf -f $d && echo $d; done
as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration

Then I tried with autoregen.py, and saw the same and now just
checked Sourceware's bot logs and saw the same numerous warnings at
least in GCC (didn't check binutils yet). Looks like this is
"expected" 

I started looking at auto-regenerating these files in our CI a couple
of weeks ago, after we received several "complaints" from contributors
saying that our precommit CI was useless / bothering since it didn't
regenerate files, leading to false alarms.
But now I'm wondering how such contributors regenerate the files
impacted by their patches before committing, they probably just
regenerate things in their subdir of interest, not noticing the whole
picture :-(

As a first step, we can probably use autoregen.py too, and declare
maintainer-mode broken. However, I do notice that besides the rules
about regenerating configure/Makefile.in/..., maintainer-mode is also
used to update some files.
In gcc:
fixincludes: fixincl.x
libffi: doc/version.texi
libgfortran: some stuff :-)
libiberty: functions.texi

in binutils/bfd:
gdb/sim
bfd/po/SRC-POTFILES.in
bfd/po/BLD-POTFILES.in
bfd/bfd-in2.h
bfd/libbfd.h
bfd/libcoff.h
binutils/po/POTFILES.in
gas/po/POTFILES.in
opcodes/i386*.h
gdb/copying.c
gdb/data-directory/*.xml
gold/po/POTFILES.in
gprof/po/POTFILES.in
gfprofng/doc/version.texi
ld/po/SRC-POTFILES.in
ld/po/BLD-POTFILES.in
ld: ldgram/ldlex... and all emulation sources
libiberty/functions.texi
opcodes/po/POTFILES.in
opcodes/aarch64-{asm,dis,opc}-2.c
opcodes/ia64 msp430 rl78 rx z8k decoders

How are these files "normally" expected to be updated? Do people just
manually uncomment the corresponding maintainer lines in the Makefiles
and update manually?   In particular we hit issues several times with
files under opcodes, that we don't regenerate currently. Maybe we
could build binutils in maintainer-mode at -j1 but, well

README-maintainer-mode in binutils/gdb only mentions a problem with
'make distclean' and maintainer mode
binutils/README-how-to-make-a-release indicates to use
--enable-maintainer-mode, and the sample 'make' invocations do not
include any -j flag, is that an indication that only -j1 is supposed
to work?
Similarly, the src-release.sh script does not use -j.

Thanks,

Christophe

>
> Grüße
>  Thomas

Re: Help needed with maintainer-mode

[Thomas Schwinge]

Hi!

On 2024-03-04T00:30:05+, Sam James  wrote:
> Mark Wielaard  writes:
>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>>> [...], I read
>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>>> which basically says "run autoreconf in every dir where there is a
>>> configure script"
>>> but this is not exactly what autoregen.py is doing. IIRC it is based
>>> on a script from Martin Liska, do you know/remember why it follows a
>>> different process?
>>
>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>> might remember more details. We wanted a script that worked for both
>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>> work in all directories. We also needed to skip some directories that
>> did contain a configure script, but that were imported (gotools,
>> readline, minizip).
>
> What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
> for AC_CONFIG_SUBDIRS.

Let me allocate some time this week to get that one completed.

> Right now, the current situation is janky and it's nowhere near
> idiomatic autotools usage. It is not a comfortable experience
> interacting with it even as someone who is familiar and happy with using
> autotools otherwise.
>
> I didn't yet play with maintainer-mode myself but I also didn't see much
> point given I knew of more fundamental problems like this.
>
> [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/


Grüße
 Thomas

Re: Help needed with maintainer-mode

[Sam James via Gcc]

Mark Wielaard  writes:

> Hi Christophe,

Hi Mark, Christophe, et. al,

>
> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>> > > > And it was indeed done this way because that way the files are
>> > > > regenerated in a reproducible way. Which wasn't the case when using 
>> > > > --enable-maintainer-mode (and autoreconfig also doesn't work).
>> > >
>> > > I see. So it is possibly incomplete, in the sense that it may lack
>> > > some of the steps that maintainer-mode would perform?
>> > > For instance, gas for aarch64 has some *opcodes*.c files that need
>> > > regenerating before committing. The regeneration step is enabled in
>> > > maintainer-mode, so I guess the autoregen bots on Sourceware would
>> > > miss a problem with these files?
>> >
>> > Yes, it is certainly incomplete. But it is done this way because it is
>> > my understanding that even the gcc release maintainers do the
>> > automake/autoconf invocations by hand instead of running with configure
>> > --enable-maintainer-mode.
>> 
>> After a discussion on IRC, I read
>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>> which basically says "run autoreconf in every dir where there is a
>> configure script"
>> but this is not exactly what autoregen.py is doing. IIRC it is based
>> on a script from Martin Liska, do you know/remember why it follows a
>> different process?
>
> CCing Sam and Arsen who helped refine the autoregen.py script, who
> might remember more details. We wanted a script that worked for both
> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> work in all directories. We also needed to skip some directories that
> did contain a configure script, but that were imported (gotools,
> readline, minizip).

What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
for AC_CONFIG_SUBDIRS.

Right now, the current situation is janky and it's nowhere near
idiomatic autotools usage. It is not a comfortable experience
interacting with it even as someone who is familiar and happy with using
autotools otherwise.

I didn't yet play with maintainer-mode myself but I also didn't see much
point given I knew of more fundamental problems like this.

[0] https://inbox.sourceware.org/gcc-patches/oril72c4yh@lxoliva.fsfla.org/

>
> Cheers,
>
> Mark

best,
sam


signature.asc
Description: PGP signature

Re: Help needed with maintainer-mode

[Mark Wielaard]

Hi Christophe,

On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> > > > And it was indeed done this way because that way the files are
> > > > regenerated in a reproducible way. Which wasn't the case when using 
> > > > --enable-maintainer-mode (and autoreconfig also doesn't work).
> > >
> > > I see. So it is possibly incomplete, in the sense that it may lack
> > > some of the steps that maintainer-mode would perform?
> > > For instance, gas for aarch64 has some *opcodes*.c files that need
> > > regenerating before committing. The regeneration step is enabled in
> > > maintainer-mode, so I guess the autoregen bots on Sourceware would
> > > miss a problem with these files?
> >
> > Yes, it is certainly incomplete. But it is done this way because it is
> > my understanding that even the gcc release maintainers do the
> > automake/autoconf invocations by hand instead of running with configure
> > --enable-maintainer-mode.
> 
> After a discussion on IRC, I read
> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> which basically says "run autoreconf in every dir where there is a
> configure script"
> but this is not exactly what autoregen.py is doing. IIRC it is based
> on a script from Martin Liska, do you know/remember why it follows a
> different process?

CCing Sam and Arsen who helped refine the autoregen.py script, who
might remember more details. We wanted a script that worked for both
gcc and binutils-gdb. And as far as I know autoreconf simply didn't
work in all directories. We also needed to skip some directories that
did contain a configure script, but that were imported (gotools,
readline, minizip).

Cheers,

Mark

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Fri, 1 Mar 2024 at 14:08, Mark Wielaard  wrote:
>
> Hi Christophe,
>
> On Thu, 2024-02-29 at 18:39 +0100, Christophe Lyon wrote:
> > On Thu, 29 Feb 2024 at 12:00, Mark Wielaard  wrote:
> > > That python script works across gcc/binutils/gdb:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> > >
> > > It is installed into a container file that has the exact autoconf and
> > > automake version needed to regenerate the autotool files:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> > >
> > > And it was indeed done this way because that way the files are
> > > regenerated in a reproducible way. Which wasn't the case when using 
> > > --enable-maintainer-mode (and autoreconfig also doesn't work).
> >
> > I see. So it is possibly incomplete, in the sense that it may lack
> > some of the steps that maintainer-mode would perform?
> > For instance, gas for aarch64 has some *opcodes*.c files that need
> > regenerating before committing. The regeneration step is enabled in
> > maintainer-mode, so I guess the autoregen bots on Sourceware would
> > miss a problem with these files?
>
> Yes, it is certainly incomplete. But it is done this way because it is
> my understanding that even the gcc release maintainers do the
> automake/autoconf invocations by hand instead of running with configure
> --enable-maintainer-mode.

After a discussion on IRC, I read
https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
which basically says "run autoreconf in every dir where there is a
configure script"
but this is not exactly what autoregen.py is doing. IIRC it is based
on a script from Martin Liska, do you know/remember why it follows a
different process?

Thanks,

Christophe

>
> Note that another part that isn't caught at the moment are the
> regeneration of the opt.urls files. There is a patch for that pending:
> https://inbox.sourceware.org/buildbot/20231215005908.gc12...@gnu.wildebeest.org/
>
> But that is waiting for the actual opt.urls to be regenerated correctly
> first:
> https://inbox.sourceware.org/gcc-patches/20240224174258.gd1...@gnu.wildebeest.org/
> Ping David?
>
> It would be nice to have all these "regeneration targets" in one script
> that could be used by both the pre-commit and post-commit checkers.
>
> Cheers,
>
> Mark

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Fri, 1 Mar 2024 at 14:08, Mark Wielaard  wrote:
>
> Hi Christophe,
>
> On Thu, 2024-02-29 at 18:39 +0100, Christophe Lyon wrote:
> > On Thu, 29 Feb 2024 at 12:00, Mark Wielaard  wrote:
> > > That python script works across gcc/binutils/gdb:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> > >
> > > It is installed into a container file that has the exact autoconf and
> > > automake version needed to regenerate the autotool files:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> > >
> > > And it was indeed done this way because that way the files are
> > > regenerated in a reproducible way. Which wasn't the case when using 
> > > --enable-maintainer-mode (and autoreconfig also doesn't work).
> >
> > I see. So it is possibly incomplete, in the sense that it may lack
> > some of the steps that maintainer-mode would perform?
> > For instance, gas for aarch64 has some *opcodes*.c files that need
> > regenerating before committing. The regeneration step is enabled in
> > maintainer-mode, so I guess the autoregen bots on Sourceware would
> > miss a problem with these files?
>
> Yes, it is certainly incomplete. But it is done this way because it is
> my understanding that even the gcc release maintainers do the
> automake/autoconf invocations by hand instead of running with configure
> --enable-maintainer-mode.

Indeed, I've just discovered that earlier today :-)

>
> Note that another part that isn't caught at the moment are the
> regeneration of the opt.urls files. There is a patch for that pending:
Indeed. I hadn't thought of it either. And just noticed it requires
the D frontend, which we don't build in CI.

> https://inbox.sourceware.org/buildbot/20231215005908.gc12...@gnu.wildebeest.org/
>
> But that is waiting for the actual opt.urls to be regenerated correctly
> first:
> https://inbox.sourceware.org/gcc-patches/20240224174258.gd1...@gnu.wildebeest.org/
> Ping David?
>
> It would be nice to have all these "regeneration targets" in one script
> that could be used by both the pre-commit and post-commit checkers.
>
Agreed.

> Cheers,
>
> Mark

Re: Help needed with maintainer-mode

[Mark Wielaard]

Hi Christophe,

On Thu, 2024-02-29 at 18:39 +0100, Christophe Lyon wrote:
> On Thu, 29 Feb 2024 at 12:00, Mark Wielaard  wrote:
> > That python script works across gcc/binutils/gdb:
> > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> > 
> > It is installed into a container file that has the exact autoconf and
> > automake version needed to regenerate the autotool files:
> > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> > 
> > And it was indeed done this way because that way the files are
> > regenerated in a reproducible way. Which wasn't the case when using 
> > --enable-maintainer-mode (and autoreconfig also doesn't work).
> 
> I see. So it is possibly incomplete, in the sense that it may lack
> some of the steps that maintainer-mode would perform?
> For instance, gas for aarch64 has some *opcodes*.c files that need
> regenerating before committing. The regeneration step is enabled in
> maintainer-mode, so I guess the autoregen bots on Sourceware would
> miss a problem with these files?

Yes, it is certainly incomplete. But it is done this way because it is
my understanding that even the gcc release maintainers do the
automake/autoconf invocations by hand instead of running with configure
--enable-maintainer-mode.

Note that another part that isn't caught at the moment are the
regeneration of the opt.urls files. There is a patch for that pending:
https://inbox.sourceware.org/buildbot/20231215005908.gc12...@gnu.wildebeest.org/

But that is waiting for the actual opt.urls to be regenerated correctly
first:
https://inbox.sourceware.org/gcc-patches/20240224174258.gd1...@gnu.wildebeest.org/
Ping David?

It would be nice to have all these "regeneration targets" in one script
that could be used by both the pre-commit and post-commit checkers.

Cheers,

Mark

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Thu, 29 Feb 2024 at 20:49, Thiago Jung Bauermann
 wrote:
>
>
> Hello,
>
> Christophe Lyon  writes:
>
> > I hoped improving this would be as simple as adding
> > --enable-maintainer-mode when configuring, after making sure
> > autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> > libtool and gettext seems OK).
> >
> > However, doing so triggered several problems, which look like race
> > conditions in the build system (we build at -j160):
> > - random build errors in binutils / gdb with messages like "No rule to
> > make target 'po/BLD-POTFILES.in". I managed to reproduce something
> > similar manually once, I noticed an empty Makefile; the build logs are
> > of course difficult to read, so I couldn't figure out yet what could
> > cause this.
> >
> > - random build failures in gcc in fixincludes. I think this is a race
> > condition because fixincludes is updated concurrently both from
> > /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> > Makefiles.
> >
> > - I've seen other errors when building gcc like
> > configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> > from libquadmath. I haven't investigated this yet.
>
> I don't know about the last one, but regarding the race conditions, one
> workaround might be to define a make target that regenerates all files
> (if one doesn't exist already, I don't know) and make the CI call it
> with -j1 to avoid concurrency, and then do the regular build step with
> -j160.
>

Yes, that's what I meant below with "magic" make target ;-)

Thanks,

Christophe

> --
> Thiago

Re: Help needed with maintainer-mode

[Thiago Jung Bauermann via Gcc]

Hello,

Christophe Lyon  writes:

> I hoped improving this would be as simple as adding
> --enable-maintainer-mode when configuring, after making sure
> autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> libtool and gettext seems OK).
>
> However, doing so triggered several problems, which look like race
> conditions in the build system (we build at -j160):
> - random build errors in binutils / gdb with messages like "No rule to
> make target 'po/BLD-POTFILES.in". I managed to reproduce something
> similar manually once, I noticed an empty Makefile; the build logs are
> of course difficult to read, so I couldn't figure out yet what could
> cause this.
>
> - random build failures in gcc in fixincludes. I think this is a race
> condition because fixincludes is updated concurrently both from
> /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> Makefiles.
>
> - I've seen other errors when building gcc like
> configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> from libquadmath. I haven't investigated this yet.

I don't know about the last one, but regarding the race conditions, one
workaround might be to define a make target that regenerates all files
(if one doesn't exist already, I don't know) and make the CI call it
with -j1 to avoid concurrency, and then do the regular build step with
-j160.

-- 
Thiago

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Thu, 29 Feb 2024 at 12:00, Mark Wielaard  wrote:
>
> Hi Christophe,
>
> On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
> > I've noticed that sourceware's buildbot has a small script
> > "autoregen.py" which does not use the project's build system, but
> > rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> > Should we replicate that?
>
> That python script works across gcc/binutils/gdb:
> https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
>
> It is installed into a container file that has the exact autoconf and
> automake version needed to regenerate the autotool files:
> https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
>
> And it was indeed done this way because that way the files are
> regenerated in a reproducible way. Which wasn't the case when using 
> --enable-maintainer-mode (and autoreconfig also doesn't work).

I see. So it is possibly incomplete, in the sense that it may lack
some of the steps that maintainer-mode would perform?
For instance, gas for aarch64 has some *opcodes*.c files that need
regenerating before committing. The regeneration step is enabled in
maintainer-mode, so I guess the autoregen bots on Sourceware would
miss a problem with these files?

Thanks,

Christophe

>
> It is run on all commits and warns if it detects a change in the
> (checked in) generated files.
> https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
> https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
>
> Cheers,
>
> Mark

Re: Help needed with maintainer-mode

[Christophe Lyon via Gcc]

On Thu, 29 Feb 2024 at 11:41, Richard Earnshaw (lists)
 wrote:
>
> On 29/02/2024 10:22, Christophe Lyon via Gcc wrote:
> > Hi!
> >
> > Sorry for cross-posting, but I'm not sure the rules/guidelines are the
> > same in gcc vs binutils/gdb.
> >
> > TL;DR: are there some guidelines about how to use/enable maintainer-mode?
> >
> > In the context of the Linaro CI, I've been looking at enabling
> > maintainer-mode at configure time in our configurations where we test
> > patches before they are committed (aka "precommit CI", which relies on
> > patchwork).
> >
> > Indeed, auto-generated files are not part of patch submissions, and
> > when a patch implies regenerating some files before building, we
> > currently report wrong failures because we don't perform such updates.
> >
> > I hoped improving this would be as simple as adding
> > --enable-maintainer-mode when configuring, after making sure
> > autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> > libtool and gettext seems OK).
> >
> > However, doing so triggered several problems, which look like race
> > conditions in the build system (we build at -j160):
> > - random build errors in binutils / gdb with messages like "No rule to
> > make target 'po/BLD-POTFILES.in". I managed to reproduce something
> > similar manually once, I noticed an empty Makefile; the build logs are
> > of course difficult to read, so I couldn't figure out yet what could
> > cause this.
> >
> > - random build failures in gcc in fixincludes. I think this is a race
> > condition because fixincludes is updated concurrently both from
> > /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> > Makefiles.
> >
> > - I've seen other errors when building gcc like
> > configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> > from libquadmath. I haven't investigated this yet.
> >
> > I've read binutils' README-maintainer-mode, which contains a warning
> > about distclean, but we don't use this: we start our builds from a
> > scratch directory.
> >
> > So... I'm wondering if there are some "official" guidelines about how
> > to regenerate files, and/or use maintainer-mode?  Maybe I missed a
> > "magic" make target (eg 'make autoreconf-all') that should be executed
> > after configure and before 'make all'?
> >
> > I've noticed that sourceware's buildbot has a small script
> > "autoregen.py" which does not use the project's build system, but
> > rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> > Should we replicate that?
> >
> > Thanks,
> >
> > Christophe
>
> There are other potential gotchas as well, such as the manual copying of the 
> generated tm.texi back into the source repo due to relicensing.  Perhaps we 
> haven't encountered that one because patches generally contain that 
> duplicated output.
>
It did happen a few weeks ago, with a patch that was updating the
target hooks IIRC.

> If we want a CI to work reliably, then perhaps we should reconsider our 
> policy of stripping out regenerated code.  We have a number of developer 
> practices, such as replying to an existing patch with an updated version that 
> the CI can't handle easily (especially if the patch is part of a series), so 
> there may be space for a discussion on how to work smarter.
>
Sure, there are many things we can improve in the current workflow to
make it more CI friendly ;-)
But I was only asking how maintainer-mode is supposed to be used, so
that I can replicate the process in CI.
I couldn't find any documentation :-)

Thanks,

Christophe

> My calendar says we have a toolchain office hours meeting today, perhaps this 
> would be worth bringing up.
>
> R.
>

Re: Help needed with maintainer-mode

[Mark Wielaard]

Hi Christophe,

On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
> I've noticed that sourceware's buildbot has a small script
> "autoregen.py" which does not use the project's build system, but
> rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> Should we replicate that?

That python script works across gcc/binutils/gdb:
https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py

It is installed into a container file that has the exact autoconf and
automake version needed to regenerate the autotool files:
https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools

And it was indeed done this way because that way the files are
regenerated in a reproducible way. Which wasn't the case when using 
--enable-maintainer-mode (and autoreconfig also doesn't work).

It is run on all commits and warns if it detects a change in the
(checked in) generated files.
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen

Cheers,

Mark

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists) via Gcc]

On 29/02/2024 10:22, Christophe Lyon via Gcc wrote:
> Hi!
> 
> Sorry for cross-posting, but I'm not sure the rules/guidelines are the
> same in gcc vs binutils/gdb.
> 
> TL;DR: are there some guidelines about how to use/enable maintainer-mode?
> 
> In the context of the Linaro CI, I've been looking at enabling
> maintainer-mode at configure time in our configurations where we test
> patches before they are committed (aka "precommit CI", which relies on
> patchwork).
> 
> Indeed, auto-generated files are not part of patch submissions, and
> when a patch implies regenerating some files before building, we
> currently report wrong failures because we don't perform such updates.
> 
> I hoped improving this would be as simple as adding
> --enable-maintainer-mode when configuring, after making sure
> autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> libtool and gettext seems OK).
> 
> However, doing so triggered several problems, which look like race
> conditions in the build system (we build at -j160):
> - random build errors in binutils / gdb with messages like "No rule to
> make target 'po/BLD-POTFILES.in". I managed to reproduce something
> similar manually once, I noticed an empty Makefile; the build logs are
> of course difficult to read, so I couldn't figure out yet what could
> cause this.
> 
> - random build failures in gcc in fixincludes. I think this is a race
> condition because fixincludes is updated concurrently both from
> /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> Makefiles.
> 
> - I've seen other errors when building gcc like
> configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> from libquadmath. I haven't investigated this yet.
> 
> I've read binutils' README-maintainer-mode, which contains a warning
> about distclean, but we don't use this: we start our builds from a
> scratch directory.
> 
> So... I'm wondering if there are some "official" guidelines about how
> to regenerate files, and/or use maintainer-mode?  Maybe I missed a
> "magic" make target (eg 'make autoreconf-all') that should be executed
> after configure and before 'make all'?
> 
> I've noticed that sourceware's buildbot has a small script
> "autoregen.py" which does not use the project's build system, but
> rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> Should we replicate that?
> 
> Thanks,
> 
> Christophe

There are other potential gotchas as well, such as the manual copying of the 
generated tm.texi back into the source repo due to relicensing.  Perhaps we 
haven't encountered that one because patches generally contain that duplicated 
output.

If we want a CI to work reliably, then perhaps we should reconsider our policy 
of stripping out regenerated code.  We have a number of developer practices, 
such as replying to an existing patch with an updated version that the CI can't 
handle easily (especially if the patch is part of a series), so there may be 
space for a discussion on how to work smarter.

My calendar says we have a toolchain office hours meeting today, perhaps this 
would be worth bringing up.

R.

Help needed with maintainer-mode

[Christophe Lyon via Gcc]

Hi!

Sorry for cross-posting, but I'm not sure the rules/guidelines are the
same in gcc vs binutils/gdb.

TL;DR: are there some guidelines about how to use/enable maintainer-mode?

In the context of the Linaro CI, I've been looking at enabling
maintainer-mode at configure time in our configurations where we test
patches before they are committed (aka "precommit CI", which relies on
patchwork).

Indeed, auto-generated files are not part of patch submissions, and
when a patch implies regenerating some files before building, we
currently report wrong failures because we don't perform such updates.

I hoped improving this would be as simple as adding
--enable-maintainer-mode when configuring, after making sure
autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
libtool and gettext seems OK).

However, doing so triggered several problems, which look like race
conditions in the build system (we build at -j160):
- random build errors in binutils / gdb with messages like "No rule to
make target 'po/BLD-POTFILES.in". I managed to reproduce something
similar manually once, I noticed an empty Makefile; the build logs are
of course difficult to read, so I couldn't figure out yet what could
cause this.

- random build failures in gcc in fixincludes. I think this is a race
condition because fixincludes is updated concurrently both from
/fixincludes and $buillddir/fixincludes. Probably fixable in gcc
Makefiles.

- I've seen other errors when building gcc like
configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
from libquadmath. I haven't investigated this yet.

I've read binutils' README-maintainer-mode, which contains a warning
about distclean, but we don't use this: we start our builds from a
scratch directory.

So... I'm wondering if there are some "official" guidelines about how
to regenerate files, and/or use maintainer-mode?  Maybe I missed a
"magic" make target (eg 'make autoreconf-all') that should be executed
after configure and before 'make all'?

I've noticed that sourceware's buildbot has a small script
"autoregen.py" which does not use the project's build system, but
rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
Should we replicate that?

Thanks,

Christophe