Re: MIPS parallel build breakage
On Thu, Mar 01, 2018 at 12:07:29PM +0100, Sascha Hauer wrote: > Hi Sam and Antony, > > (Antony because you introduced this and Sam because you have intimate > knowledge with our build system ;) > > I am trying to solve a problem that annoys me for some years now. The > problem is that the MIPS defconfigs regularly fail in my autobuilder. > I usually help myself with trying it again and most of the time it > then works, but doing this for years increases the pressure to do > something. > > What happens is that the MIPS builds use include/generated/compile.h: > > > arch/mips/include/asm/pbl_macros.h:28:#include > > This often ends in: > > > In file included from arch/mips/boot/start.S:20:0: > > arch/mips/include/asm/pbl_macros.h:28:31: > > fatal error: generated/compile.h: No such file or directory > > compilation terminated. > > include/generated/compile.h is generated in common/Makefile: > > > include/generated/compile.h: FORCE > > @$($(quiet)chk_compile.h) > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > Other users of this file add an explicit dependency on it: > > > # dependencies on generated files need to be listed explicitly > > $(obj)/version.o: include/generated/compile.h > > This only seems to work though when the dependent file is in the same > directory, but not when the file is in arch/mips/boot/. > > I tried doing the include/generated/compile.h in Makefile instead of > common/Makefile, but then the version counting gets confused and causes > unnecessary rebuilds. > > Any ideas how to solve this? We still have no solution for this. I dropped MIPS from my autobuilder, so we won't catch any build errors on MIPS anymore :( Sascha -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
On Fri, Mar 02, 2018 at 04:37:31PM +0300, Antony Pavlov wrote: > On Fri, 2 Mar 2018 13:34:01 +0100 > Sascha Hauer wrote: > > > On Fri, Mar 02, 2018 at 03:17:35PM +0300, Antony Pavlov wrote: > > > On Thu, 1 Mar 2018 12:07:29 +0100 > > > Sascha Hauer wrote: > > > > > > > Hi Sam and Antony, > > > > > > > > (Antony because you introduced this and Sam because you have intimate > > > > knowledge with our build system ;) > > > > > > > > I am trying to solve a problem that annoys me for some years now. The > > > > problem is that the MIPS defconfigs regularly fail in my autobuilder. > > > > I usually help myself with trying it again and most of the time it > > > > then works, but doing this for years increases the pressure to do > > > > something. > > > > > > > > What happens is that the MIPS builds use include/generated/compile.h: > > > > > > > > > arch/mips/include/asm/pbl_macros.h:28:#include > > > > > > > > This often ends in: > > > > > > > > > In file included from arch/mips/boot/start.S:20:0: > > > > > arch/mips/include/asm/pbl_macros.h:28:31: > > > > > fatal error: generated/compile.h: No such file or directory > > > > > compilation terminated. > > > > > > > > include/generated/compile.h is generated in common/Makefile: > > > > > > > > > include/generated/compile.h: FORCE > > > > > @$($(quiet)chk_compile.h) > > > > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > > > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > > > > > > > Other users of this file add an explicit dependency on it: > > > > > > > > > # dependencies on generated files need to be listed explicitly > > > > > $(obj)/version.o: include/generated/compile.h > > > > > > > > This only seems to work though when the dependent file is in the same > > > > directory, but not when the file is in arch/mips/boot/. > > > > > > > > I tried doing the include/generated/compile.h in Makefile instead of > > > > common/Makefile, but then the version counting gets confused and causes > > > > unnecessary rebuilds. > > > > > > > > Any ideas how to solve this? > > > > > > I can't reproduce parallel build breakage. > > > > The failure starts here when using 6 threads or more and only when > > compiling the black-swift_defconfig. After one successful build I can > > reproduce this reliably here with: > > > > rm -f include/generated/compile.h && make -j 6 > > > > > But I see another problem: > > > the version string inside mips pbl binary is differ from the version > > > string > > > inside main barebox binary. > > > > > > I suppose that the problem is that current build system handles mips asm > > > files > > > (which are actually use generated compile.h) and c-files in a different > > > way. > > > > Ups. How do the strings differ? > > insert generated version into pbl by using mips_barebox_10h: > > barebox$ git diff > diff --git a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > index 5c1537262e..6e85a7009a 100644 > --- a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > +++ b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h > @@ -43,18 +43,8 @@ > .setpush > .setnoreorder > > - b __start > -nop > + mips_barebox_10h > > - /* > - On MIPS Technologies boards > - 0x1fc00010 address is reserved for BoardID > - */ > - .org0x10 > - .word 0x > - .word 0x > - > -__start: > mips_disable_interrupts > > /* cpu specific setup ... */ > barebox$ make mrproper > ... > barebox$ export ARCH=mips > barebox$ export CROSS_COMPILE=mips-linux-gnu- > barebox$ make qemu-malta_defconfig > ... > barebox$ make -j3 -s > ... > barebox$ xxd barebox-flash-image | head -n 6 > : 1000 0017 > 0010: 6261 7265 626f 7820 3230 3138 2e30 322e barebox 2018.02. > 0020: 302d 3030 3037 392d 6731 6564 6462 3064 0-00079-g1eddb0d > 0030: 3338 322d 6469 7274 7920 2330 2046 7269 382-dirty #0 Fri >^^^ > 0040: 204d 6172 2032 2031 363a 3032 3a32 3920 Mar 2 16:02:29 > 0050: 4d53 4b20 3230 3138 MSK 2018 > > #0, 16:02:29 > > barebox$ qemu-system-mips -nodefaults -M malta -m 256 \ > > -nographic -serial stdio -monitor null \ > > -bios barebox-flash-image > > > barebox 2018.02.0-00079-g1eddb0d382-dirty #1 Fri Mar 2 16:02:38 MSK 2018 > ^^^ > > #1, 16:02:38 I can imagine this also goes down to the missing dependency. UTS_VERSION contains the build number (the number behind the #) and the build date. Now if the PBL macros lack the dependency and generated/compile.h is included *before* it is re-generated whereas the files in common do have
Re: MIPS parallel build breakage
On Fri, 2 Mar 2018 13:34:01 +0100 Sascha Hauer wrote: > On Fri, Mar 02, 2018 at 03:17:35PM +0300, Antony Pavlov wrote: > > On Thu, 1 Mar 2018 12:07:29 +0100 > > Sascha Hauer wrote: > > > > > Hi Sam and Antony, > > > > > > (Antony because you introduced this and Sam because you have intimate > > > knowledge with our build system ;) > > > > > > I am trying to solve a problem that annoys me for some years now. The > > > problem is that the MIPS defconfigs regularly fail in my autobuilder. > > > I usually help myself with trying it again and most of the time it > > > then works, but doing this for years increases the pressure to do > > > something. > > > > > > What happens is that the MIPS builds use include/generated/compile.h: > > > > > > > arch/mips/include/asm/pbl_macros.h:28:#include > > > > > > This often ends in: > > > > > > > In file included from arch/mips/boot/start.S:20:0: > > > > arch/mips/include/asm/pbl_macros.h:28:31: > > > > fatal error: generated/compile.h: No such file or directory > > > > compilation terminated. > > > > > > include/generated/compile.h is generated in common/Makefile: > > > > > > > include/generated/compile.h: FORCE > > > > @$($(quiet)chk_compile.h) > > > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > > > > > Other users of this file add an explicit dependency on it: > > > > > > > # dependencies on generated files need to be listed explicitly > > > > $(obj)/version.o: include/generated/compile.h > > > > > > This only seems to work though when the dependent file is in the same > > > directory, but not when the file is in arch/mips/boot/. > > > > > > I tried doing the include/generated/compile.h in Makefile instead of > > > common/Makefile, but then the version counting gets confused and causes > > > unnecessary rebuilds. > > > > > > Any ideas how to solve this? > > > > I can't reproduce parallel build breakage. > > The failure starts here when using 6 threads or more and only when > compiling the black-swift_defconfig. After one successful build I can > reproduce this reliably here with: > > rm -f include/generated/compile.h && make -j 6 > > > But I see another problem: > > the version string inside mips pbl binary is differ from the version string > > inside main barebox binary. > > > > I suppose that the problem is that current build system handles mips asm > > files > > (which are actually use generated compile.h) and c-files in a different way. > > Ups. How do the strings differ? insert generated version into pbl by using mips_barebox_10h: barebox$ git diff diff --git a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h index 5c1537262e..6e85a7009a 100644 --- a/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h +++ b/arch/mips/boards/qemu-malta/include/board/board_pbl_start.h @@ -43,18 +43,8 @@ .setpush .setnoreorder - b __start -nop + mips_barebox_10h - /* - On MIPS Technologies boards - 0x1fc00010 address is reserved for BoardID - */ - .org0x10 - .word 0x - .word 0x - -__start: mips_disable_interrupts /* cpu specific setup ... */ barebox$ make mrproper ... barebox$ export ARCH=mips barebox$ export CROSS_COMPILE=mips-linux-gnu- barebox$ make qemu-malta_defconfig ... barebox$ make -j3 -s ... barebox$ xxd barebox-flash-image | head -n 6 : 1000 0017 0010: 6261 7265 626f 7820 3230 3138 2e30 322e barebox 2018.02. 0020: 302d 3030 3037 392d 6731 6564 6462 3064 0-00079-g1eddb0d 0030: 3338 322d 6469 7274 7920 2330 2046 7269 382-dirty #0 Fri ^^^ 0040: 204d 6172 2032 2031 363a 3032 3a32 3920 Mar 2 16:02:29 0050: 4d53 4b20 3230 3138 MSK 2018 #0, 16:02:29 barebox$ qemu-system-mips -nodefaults -M malta -m 256 \ > -nographic -serial stdio -monitor null \ > -bios barebox-flash-image barebox 2018.02.0-00079-g1eddb0d382-dirty #1 Fri Mar 2 16:02:38 MSK 2018 ^^^ #1, 16:02:38 > Another obvious way to fix the issue is to remove UTS_RELEASE and > UTS_VERSION from the pbl headers. Would that be an option? It isn't a __good__ option. -- Best regards, Antony Pavlov ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
On Fri, Mar 02, 2018 at 03:17:35PM +0300, Antony Pavlov wrote: > On Thu, 1 Mar 2018 12:07:29 +0100 > Sascha Hauer wrote: > > > Hi Sam and Antony, > > > > (Antony because you introduced this and Sam because you have intimate > > knowledge with our build system ;) > > > > I am trying to solve a problem that annoys me for some years now. The > > problem is that the MIPS defconfigs regularly fail in my autobuilder. > > I usually help myself with trying it again and most of the time it > > then works, but doing this for years increases the pressure to do > > something. > > > > What happens is that the MIPS builds use include/generated/compile.h: > > > > > arch/mips/include/asm/pbl_macros.h:28:#include > > > > This often ends in: > > > > > In file included from arch/mips/boot/start.S:20:0: > > > arch/mips/include/asm/pbl_macros.h:28:31: > > > fatal error: generated/compile.h: No such file or directory > > > compilation terminated. > > > > include/generated/compile.h is generated in common/Makefile: > > > > > include/generated/compile.h: FORCE > > > @$($(quiet)chk_compile.h) > > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > > > Other users of this file add an explicit dependency on it: > > > > > # dependencies on generated files need to be listed explicitly > > > $(obj)/version.o: include/generated/compile.h > > > > This only seems to work though when the dependent file is in the same > > directory, but not when the file is in arch/mips/boot/. > > > > I tried doing the include/generated/compile.h in Makefile instead of > > common/Makefile, but then the version counting gets confused and causes > > unnecessary rebuilds. > > > > Any ideas how to solve this? > > I can't reproduce parallel build breakage. The failure starts here when using 6 threads or more and only when compiling the black-swift_defconfig. After one successful build I can reproduce this reliably here with: rm -f include/generated/compile.h && make -j 6 > But I see another problem: > the version string inside mips pbl binary is differ from the version string > inside main barebox binary. > > I suppose that the problem is that current build system handles mips asm files > (which are actually use generated compile.h) and c-files in a different way. Ups. How do the strings differ? Another obvious way to fix the issue is to remove UTS_RELEASE and UTS_VERSION from the pbl headers. Would that be an option? Sascha -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
On Thu, 1 Mar 2018 12:07:29 +0100 Sascha Hauer wrote: > Hi Sam and Antony, > > (Antony because you introduced this and Sam because you have intimate > knowledge with our build system ;) > > I am trying to solve a problem that annoys me for some years now. The > problem is that the MIPS defconfigs regularly fail in my autobuilder. > I usually help myself with trying it again and most of the time it > then works, but doing this for years increases the pressure to do > something. > > What happens is that the MIPS builds use include/generated/compile.h: > > > arch/mips/include/asm/pbl_macros.h:28:#include > > This often ends in: > > > In file included from arch/mips/boot/start.S:20:0: > > arch/mips/include/asm/pbl_macros.h:28:31: > > fatal error: generated/compile.h: No such file or directory > > compilation terminated. > > include/generated/compile.h is generated in common/Makefile: > > > include/generated/compile.h: FORCE > > @$($(quiet)chk_compile.h) > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > Other users of this file add an explicit dependency on it: > > > # dependencies on generated files need to be listed explicitly > > $(obj)/version.o: include/generated/compile.h > > This only seems to work though when the dependent file is in the same > directory, but not when the file is in arch/mips/boot/. > > I tried doing the include/generated/compile.h in Makefile instead of > common/Makefile, but then the version counting gets confused and causes > unnecessary rebuilds. > > Any ideas how to solve this? I can't reproduce parallel build breakage. But I see another problem: the version string inside mips pbl binary is differ from the version string inside main barebox binary. I suppose that the problem is that current build system handles mips asm files (which are actually use generated compile.h) and c-files in a different way. -- Best regards, Antony Pavlov ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
Re: MIPS parallel build breakage
Hi, On Thu, Mar 01, 2018 at 12:07:29PM +0100, Sascha Hauer wrote: > (Antony because you introduced this and Sam because you have intimate > knowledge with our build system ;) > > I am trying to solve a problem that annoys me for some years now. The > problem is that the MIPS defconfigs regularly fail in my autobuilder. > I usually help myself with trying it again and most of the time it > then works, but doing this for years increases the pressure to do > something. > > What happens is that the MIPS builds use include/generated/compile.h: > > > arch/mips/include/asm/pbl_macros.h:28:#include > > This often ends in: > > > In file included from arch/mips/boot/start.S:20:0: > > arch/mips/include/asm/pbl_macros.h:28:31: > > fatal error: generated/compile.h: No such file or directory > > compilation terminated. > > include/generated/compile.h is generated in common/Makefile: > > > include/generated/compile.h: FORCE > > @$($(quiet)chk_compile.h) > > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" > > Other users of this file add an explicit dependency on it: > > > # dependencies on generated files need to be listed explicitly > > $(obj)/version.o: include/generated/compile.h > > This only seems to work though when the dependent file is in the same > directory, but not when the file is in arch/mips/boot/. > > I tried doing the include/generated/compile.h in Makefile instead of > common/Makefile, but then the version counting gets confused and causes > unnecessary rebuilds. > > Any ideas how to solve this? arch/mips/boot/: common/ in arch/mips/Makefile or something like that? These two are both targets of the toplevel make call, so that should work. The same for $(BOARD), I think and maybe others that include generated/compile.h. Michael -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
MIPS parallel build breakage
Hi Sam and Antony, (Antony because you introduced this and Sam because you have intimate knowledge with our build system ;) I am trying to solve a problem that annoys me for some years now. The problem is that the MIPS defconfigs regularly fail in my autobuilder. I usually help myself with trying it again and most of the time it then works, but doing this for years increases the pressure to do something. What happens is that the MIPS builds use include/generated/compile.h: > arch/mips/include/asm/pbl_macros.h:28:#include This often ends in: > In file included from arch/mips/boot/start.S:20:0: > arch/mips/include/asm/pbl_macros.h:28:31: > fatal error: generated/compile.h: No such file or directory > compilation terminated. include/generated/compile.h is generated in common/Makefile: > include/generated/compile.h: FORCE > @$($(quiet)chk_compile.h) > $(Q)$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \ > "$(UTS_MACHINE)" "$(CC) $(KBUILD_CFLAGS)" Other users of this file add an explicit dependency on it: > # dependencies on generated files need to be listed explicitly > $(obj)/version.o: include/generated/compile.h This only seems to work though when the dependent file is in the same directory, but not when the file is in arch/mips/boot/. I tried doing the include/generated/compile.h in Makefile instead of common/Makefile, but then the version counting gets confused and causes unnecessary rebuilds. Any ideas how to solve this? Sascha -- Pengutronix e.K. | | Industrial Linux Solutions | http://www.pengutronix.de/ | Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0| Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917- | ___ barebox mailing list barebox@lists.infradead.org http://lists.infradead.org/mailman/listinfo/barebox
