Re: btrfs-progs-v4.12: cross compiling
On 2017年08月16日 01:28, Eric Sandeen wrote: On 8/15/17 7:48 AM, David Sterba wrote: On Tue, Aug 15, 2017 at 02:44:07PM +0200, Hallo32 wrote: ... How the kernel deals with this kind of problem ? Looking at the source of btrfs Makefile, it is more simple to replace mktables: kernel-lib/mktables.c @echo "[CC] $@" $(Q)$(CC) $(CFLAGS) $< -o $@ with mktables: kernel-lib/mktables.c @echo "[HOSTCC] $@" $(Q)$(HOSTCC) $(CFLAGS) $< -o $@ where HOSTCC is defined as HOSTCC=gcc (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?) If using HOSTCC then I'm fine with it. CFLAGS needs also be replaced by something like HOSTCFLAGS, because if you use something like mips/architecture specific CFLAGS, they may be not applicably on the host system. Good point. Without a regular/automated cross-compilation build testing I think we could break it quite easily. I'm going to keep the pregenerated file in git. Isn't using the host compiler for some binaries during a cross-compile a very standard thing to do? The kernel manages it, as shown above. xfsprogs does it (see libxfs/Makefile for the crc32table.h and crc32selftest targets), e2fsprogs does it (see gen_crc32ctable target in lib/ext2fs/Makefile), etc. Only if I can find it earlier. Seems that checking in a generated file would be more prone to eventual errors, no? Also my concern. I guess it's harder to do in btrfs-progs since it's not using autotools... And your patch does do a better job, especially for BUILD_CFLAGS and build cc detection. Thanks, Qu -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 8/14/17 11:10 AM, David Sterba wrote: > On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote: >> On 2017年08月14日 22:03, David Sterba wrote: >>> On Mon, Aug 14, 2017 at 09:17:08PM +0800, Qu Wenruo wrote: On 2017年08月14日 21:06, David Sterba wrote: > On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote: >> Since versions 4.12 btrfs-progs is complicated to cross compile for >> other systems. >> The problem is, that this version includes mktables, which needs to be >> compiled for the host system and executed there for the creation of >> tables.c. >> >> Are there any changes planed for the next version of btrfs-progs to make >> the cross compiling as simple as in the past? A included tables.c for >> example? > > Yes, keeping the generated tables.c around is fine. There's no reason it > needs to be generated each time during build. I'll fix that in 4.12.1. But the number of lines and impossibility to review it makes it not suitable to be managed by git. >>> >>> I don't understand your concern. The file is generated from a set of >>> formulas, not intended to be updated directly. >> >> Yes, it should never be updated directly, so it's generated by a less >> than 400 lines program, instead of a whole 10K+ lines file managed by git. > > mktables.c is synced from kernel sources, taking updates from there is > easier than porting any changes to the proposed scripted implementation. > > The workflow is simple: > - copy kernel mktables.c changes to btrfs-progs mktables.c > - compile mktables > - run 'make kernel-lib/tables.c' Can't this happen as part of a make dist (that we don't do right now)? > - commit the changes to git ... and anyone using the git repo directly can sort out how to build it? -Jeff -- Jeff Mahoney SUSE Labs signature.asc Description: OpenPGP digital signature
Re: btrfs-progs-v4.12: cross compiling
On 8/15/17 12:28 PM, Eric Sandeen wrote: > I guess it's harder to do in btrfs-progs since it's not using autotools... Eh, I don't know why I thought that was still true :) patch sent. -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 8/15/17 7:48 AM, David Sterba wrote: > On Tue, Aug 15, 2017 at 02:44:07PM +0200, Hallo32 wrote: ... How the kernel deals with this kind of problem ? Looking at the source of btrfs Makefile, it is more simple to replace mktables: kernel-lib/mktables.c @echo "[CC] $@" $(Q)$(CC) $(CFLAGS) $< -o $@ with mktables: kernel-lib/mktables.c @echo "[HOSTCC] $@" $(Q)$(HOSTCC) $(CFLAGS) $< -o $@ where HOSTCC is defined as HOSTCC=gcc (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?) >>> >>> If using HOSTCC then I'm fine with it. >> >> CFLAGS needs also be replaced by something like HOSTCFLAGS, because if >> you use something like mips/architecture specific CFLAGS, they may be >> not applicably on the host system. > > Good point. Without a regular/automated cross-compilation build testing > I think we could break it quite easily. I'm going to keep the > pregenerated file in git. Isn't using the host compiler for some binaries during a cross-compile a very standard thing to do? The kernel manages it, as shown above. xfsprogs does it (see libxfs/Makefile for the crc32table.h and crc32selftest targets), e2fsprogs does it (see gen_crc32ctable target in lib/ext2fs/Makefile), etc. Seems that checking in a generated file would be more prone to eventual errors, no? I guess it's harder to do in btrfs-progs since it's not using autotools... -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On Tue, Aug 15, 2017 at 02:44:07PM +0200, Hallo32 wrote: > Am 15.08.2017 um 01:39 schrieb Qu Wenruo: > > On 2017年08月15日 02:57, Goffredo Baroncelli wrote: > >> On 08/14/2017 05:10 PM, David Sterba wrote: > >>> On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote: > >> [...] > >>> mktables.c is synced from kernel sources, taking updates from there is > >>> easier than porting any changes to the proposed scripted > >>> implementation. > >>> > >>> The workflow is simple: > >>> - copy kernel mktables.c changes to btrfs-progs mktables.c > >> > >> How the kernel deals with this kind of problem ? > >> Looking at the source of btrfs Makefile, it is more simple to replace > >> > >>mktables: kernel-lib/mktables.c > >> @echo "[CC] $@" > >> $(Q)$(CC) $(CFLAGS) $< -o $@ > >> > >> with > >> > >> > >>mktables: kernel-lib/mktables.c > >> @echo "[HOSTCC] $@" > >> $(Q)$(HOSTCC) $(CFLAGS) $< -o $@ > >> > >> where HOSTCC is defined as > >> > >> HOSTCC=gcc > >> > >> > >> (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?) > > > > If using HOSTCC then I'm fine with it. > > CFLAGS needs also be replaced by something like HOSTCFLAGS, because if > you use something like mips/architecture specific CFLAGS, they may be > not applicably on the host system. Good point. Without a regular/automated cross-compilation build testing I think we could break it quite easily. I'm going to keep the pregenerated file in git. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
Am 15.08.2017 um 01:39 schrieb Qu Wenruo: On 2017年08月15日 02:57, Goffredo Baroncelli wrote: On 08/14/2017 05:10 PM, David Sterba wrote: On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote: [...] mktables.c is synced from kernel sources, taking updates from there is easier than porting any changes to the proposed scripted implementation. The workflow is simple: - copy kernel mktables.c changes to btrfs-progs mktables.c How the kernel deals with this kind of problem ? Looking at the source of btrfs Makefile, it is more simple to replace mktables: kernel-lib/mktables.c @echo "[CC] $@" $(Q)$(CC) $(CFLAGS) $< -o $@ with mktables: kernel-lib/mktables.c @echo "[HOSTCC] $@" $(Q)$(HOSTCC) $(CFLAGS) $< -o $@ where HOSTCC is defined as HOSTCC=gcc (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?) If using HOSTCC then I'm fine with it. Thanks, Qu Hi, CFLAGS needs also be replaced by something like HOSTCFLAGS, because if you use something like mips/architecture specific CFLAGS, they may be not applicably on the host system. Best regards - compile mktables - run 'make kernel-lib/tables.c' - commit the changes to git All of that happens very rarely, if ever, the raid56 tables and correction algorithms are set in stone. Any extensions need to be done on both sides kernel/userspace. What about using script to generate it? We do have the mktables utility to generate it and I'll regenerate it should there be a change to kernel-lib/mktables.c I mean to replace mktables.c with a script. So no cross compiler problems at all, and even easier Makefile. No dependence on "mktables" program. Somebody has to implement the script and verify that the output is the same, eventually sync changes. The cross-compilation should be fixed with the pregenerated tables.c . Is Makefile size really a concern? The number of related lines is like 7. I don't see any benefit in what you propose and hopefully explained my viewpoint enough so I don't have to continue. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 2017年08月15日 02:57, Goffredo Baroncelli wrote: On 08/14/2017 05:10 PM, David Sterba wrote: On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote: [...] mktables.c is synced from kernel sources, taking updates from there is easier than porting any changes to the proposed scripted implementation. The workflow is simple: - copy kernel mktables.c changes to btrfs-progs mktables.c How the kernel deals with this kind of problem ? Looking at the source of btrfs Makefile, it is more simple to replace mktables: kernel-lib/mktables.c @echo "[CC] $@" $(Q)$(CC) $(CFLAGS) $< -o $@ with mktables: kernel-lib/mktables.c @echo "[HOSTCC] $@" $(Q)$(HOSTCC) $(CFLAGS) $< -o $@ where HOSTCC is defined as HOSTCC=gcc (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?) If using HOSTCC then I'm fine with it. Thanks, Qu - compile mktables - run 'make kernel-lib/tables.c' - commit the changes to git All of that happens very rarely, if ever, the raid56 tables and correction algorithms are set in stone. Any extensions need to be done on both sides kernel/userspace. What about using script to generate it? We do have the mktables utility to generate it and I'll regenerate it should there be a change to kernel-lib/mktables.c I mean to replace mktables.c with a script. So no cross compiler problems at all, and even easier Makefile. No dependence on "mktables" program. Somebody has to implement the script and verify that the output is the same, eventually sync changes. The cross-compilation should be fixed with the pregenerated tables.c . Is Makefile size really a concern? The number of related lines is like 7. I don't see any benefit in what you propose and hopefully explained my viewpoint enough so I don't have to continue. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 08/14/2017 05:10 PM, David Sterba wrote: > On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote: [...] > mktables.c is synced from kernel sources, taking updates from there is > easier than porting any changes to the proposed scripted implementation. > > The workflow is simple: > - copy kernel mktables.c changes to btrfs-progs mktables.c How the kernel deals with this kind of problem ? Looking at the source of btrfs Makefile, it is more simple to replace mktables: kernel-lib/mktables.c @echo "[CC] $@" $(Q)$(CC) $(CFLAGS) $< -o $@ with mktables: kernel-lib/mktables.c @echo "[HOSTCC] $@" $(Q)$(HOSTCC) $(CFLAGS) $< -o $@ where HOSTCC is defined as HOSTCC=gcc (may be the same applied also to CFLAGS <-> HOSTCFLAGS ?) > - compile mktables > - run 'make kernel-lib/tables.c' > - commit the changes to git > > All of that happens very rarely, if ever, the raid56 tables and > correction algorithms are set in stone. Any extensions need to be done > on both sides kernel/userspace. > What about using script to generate it? >>> >>> We do have the mktables utility to generate it and I'll regenerate it >>> should there be a change to kernel-lib/mktables.c >> >> I mean to replace mktables.c with a script. >> So no cross compiler problems at all, and even easier Makefile. >> No dependence on "mktables" program. > > Somebody has to implement the script and verify that the output is the > same, eventually sync changes. The cross-compilation should be fixed > with the pregenerated tables.c . Is Makefile size really a concern? The > number of related lines is like 7. I don't see any benefit in what you > propose and hopefully explained my viewpoint enough so I don't have to > continue. > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- gpg @keyserver.linux.it: Goffredo Baroncelli Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On Mon, Aug 14, 2017 at 10:14:42PM +0800, Qu Wenruo wrote: > On 2017年08月14日 22:03, David Sterba wrote: > > On Mon, Aug 14, 2017 at 09:17:08PM +0800, Qu Wenruo wrote: > >> On 2017年08月14日 21:06, David Sterba wrote: > >>> On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote: > Since versions 4.12 btrfs-progs is complicated to cross compile for > other systems. > The problem is, that this version includes mktables, which needs to be > compiled for the host system and executed there for the creation of > tables.c. > > Are there any changes planed for the next version of btrfs-progs to make > the cross compiling as simple as in the past? A included tables.c for > example? > >>> > >>> Yes, keeping the generated tables.c around is fine. There's no reason it > >>> needs to be generated each time during build. I'll fix that in 4.12.1. > >> > >> But the number of lines and impossibility to review it makes it not > >> suitable to be managed by git. > > > > I don't understand your concern. The file is generated from a set of > > formulas, not intended to be updated directly. > > Yes, it should never be updated directly, so it's generated by a less > than 400 lines program, instead of a whole 10K+ lines file managed by git. mktables.c is synced from kernel sources, taking updates from there is easier than porting any changes to the proposed scripted implementation. The workflow is simple: - copy kernel mktables.c changes to btrfs-progs mktables.c - compile mktables - run 'make kernel-lib/tables.c' - commit the changes to git All of that happens very rarely, if ever, the raid56 tables and correction algorithms are set in stone. Any extensions need to be done on both sides kernel/userspace. > >> What about using script to generate it? > > > > We do have the mktables utility to generate it and I'll regenerate it > > should there be a change to kernel-lib/mktables.c > > I mean to replace mktables.c with a script. > So no cross compiler problems at all, and even easier Makefile. > No dependence on "mktables" program. Somebody has to implement the script and verify that the output is the same, eventually sync changes. The cross-compilation should be fixed with the pregenerated tables.c . Is Makefile size really a concern? The number of related lines is like 7. I don't see any benefit in what you propose and hopefully explained my viewpoint enough so I don't have to continue. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 2017年08月14日 22:03, David Sterba wrote: On Mon, Aug 14, 2017 at 09:17:08PM +0800, Qu Wenruo wrote: On 2017年08月14日 21:06, David Sterba wrote: On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote: Since versions 4.12 btrfs-progs is complicated to cross compile for other systems. The problem is, that this version includes mktables, which needs to be compiled for the host system and executed there for the creation of tables.c. Are there any changes planed for the next version of btrfs-progs to make the cross compiling as simple as in the past? A included tables.c for example? Yes, keeping the generated tables.c around is fine. There's no reason it needs to be generated each time during build. I'll fix that in 4.12.1. But the number of lines and impossibility to review it makes it not suitable to be managed by git. I don't understand your concern. The file is generated from a set of formulas, not intended to be updated directly. Yes, it should never be updated directly, so it's generated by a less than 400 lines program, instead of a whole 10K+ lines file managed by git. What about using script to generate it? We do have the mktables utility to generate it and I'll regenerate it should there be a change to kernel-lib/mktables.c I mean to replace mktables.c with a script. So no cross compiler problems at all, and even easier Makefile. No dependence on "mktables" program. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On Mon, Aug 14, 2017 at 09:17:08PM +0800, Qu Wenruo wrote: > > > On 2017年08月14日 21:06, David Sterba wrote: > > On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote: > >> Since versions 4.12 btrfs-progs is complicated to cross compile for > >> other systems. > >> The problem is, that this version includes mktables, which needs to be > >> compiled for the host system and executed there for the creation of > >> tables.c. > >> > >> Are there any changes planed for the next version of btrfs-progs to make > >> the cross compiling as simple as in the past? A included tables.c for > >> example? > > > > Yes, keeping the generated tables.c around is fine. There's no reason it > > needs to be generated each time during build. I'll fix that in 4.12.1. > > But the number of lines and impossibility to review it makes it not > suitable to be managed by git. I don't understand your concern. The file is generated from a set of formulas, not intended to be updated directly. > > What about using script to generate it? We do have the mktables utility to generate it and I'll regenerate it should there be a change to kernel-lib/mktables.c -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 2017年08月14日 21:06, David Sterba wrote: On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote: Since versions 4.12 btrfs-progs is complicated to cross compile for other systems. The problem is, that this version includes mktables, which needs to be compiled for the host system and executed there for the creation of tables.c. Are there any changes planed for the next version of btrfs-progs to make the cross compiling as simple as in the past? A included tables.c for example? Yes, keeping the generated tables.c around is fine. There's no reason it needs to be generated each time during build. I'll fix that in 4.12.1. But the number of lines and impossibility to review it makes it not suitable to be managed by git. What about using script to generate it? Thanks, Qu Thanks for the report, cross-compilation should stay supported. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On 2017年08月14日 20:17, Hallo32 wrote: Hello at all, I'm new at this list. If the mail is not in line with your standards please inform me. Since versions 4.12 btrfs-progs is complicated to cross compile for other systems. The problem is, that this version includes mktables, which needs to be compiled for the host system and executed there for the creation of tables.c. Are there any changes planed for the next version of btrfs-progs to make the cross compiling as simple as in the past? A included tables.c for example? Including tables.c seems quite crazy for me. Just check how many lines tables.c has. If we screw up any number of it, RAID6 is screwed up and we will never find the bug by reviewing the such meaningless numbers. We could enhance the makefile to use host-cc to compile and execute mktable. But it may make the makefile more complicated just for one file. (Well, I just don't want to screw traditional Makefile/configure any more) Another solution is to generate tables.c using scripts, other binary executable. I'll try to port mktable to bash or python, which should solve the problem withouth further complicating makefile. (And maybe I can update kernel too, but kernel has its host-cc and cross-cc already distinguished well, so it may not help much) Thanks, Qu Best regards -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-progs-v4.12: cross compiling
On Mon, Aug 14, 2017 at 02:17:26PM +0200, Hallo32 wrote: > Since versions 4.12 btrfs-progs is complicated to cross compile for > other systems. > The problem is, that this version includes mktables, which needs to be > compiled for the host system and executed there for the creation of > tables.c. > > Are there any changes planed for the next version of btrfs-progs to make > the cross compiling as simple as in the past? A included tables.c for > example? Yes, keeping the generated tables.c around is fine. There's no reason it needs to be generated each time during build. I'll fix that in 4.12.1. Thanks for the report, cross-compilation should stay supported. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
