Re: 10-CURRENT i386 memstick snapshots broken?
On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote:
Has anyone else tried the i386 memstick and having the same problem?
Hmm. Thanks for the report. I'll take a look at the logs for i386, but
they are generated the same way as the amd64, so in theory should not
have any noticable difference.
Jimmy, thank you for reporting this. I'm honestly not quite sure how
this went unnoticed for so long.
So, basically here is what happens:
The scripts that run the weekly snapshot builds on FreeBSD.org check out
clean svn trees of head/ and stable/9 to use to seed chroot
environments, where the builds actually happen.
For amd64 and i386, native binaries are built, and installed into
scratch directories; for powerpc and powerpc64, I just use the amd64
binaries, because I cannot directly use the chroot binaries for
non-native architecture.
The scripts chroot into the scratch directories, and run the real
release builds. As Jimmy noted, the
src/release/${ARCH}/make-memstick.sh script is what generates the
memstick images. Part of that procedure is to create md(4) device, and
partition layout with gpart(8). This is where things blow up.
Because the userland is 32-bit and the kernel is 64-bit, something
goes wrong, but interestingly not wrong enough that the script fails
entirely. So, the paritions appear to be created, but in reality, they
are not.
So, for the snapshots case, the solution is to write the memstick image
from outside of the chroot environment, which is easy to do because
I already do this for creating the VM disk images (interestingly for the
same reason as the memstick creation failure).
Thanks to Jimmy for his patience in helping me make sure my solution
does in fact fix the problem, and the next set of 10.0-CURRENT and
9.1-STABLE i386 memsticks will not have this issue (builds are in-flight
now).
Glen
pgpueHbr_ho0d.pgp
Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
On Jun 8, 2013, at 10:34 AM, Glen Barber wrote: On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote: Has anyone else tried the i386 memstick and having the same problem? Hmm. Thanks for the report. I'll take a look at the logs for i386, but they are generated the same way as the amd64, so in theory should not have any noticable difference. For amd64 and i386, native binaries are built, and installed into scratch directories; for powerpc and powerpc64, I just use the amd64 binaries, because I cannot directly use the chroot binaries for non-native architecture. The scripts chroot into the scratch directories, and run the real release builds. Have you tried using Crochet for this sort of thing? Since it was designed from the ground up for cross-building bootable images, it should avoid these issues. The only fundamental limit right now is that Crochet uses the host system to build the UFS filesystems, so it can't build big-endian MIPS images on i386, for example. Tim signature.asc Description: Message signed with OpenPGP using GPGMail
Re: 10-CURRENT i386 memstick snapshots broken?
On Sat, Jun 08, 2013 at 12:10:16PM -0700, Tim Kientzle wrote: On Jun 8, 2013, at 10:34 AM, Glen Barber wrote: On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote: Has anyone else tried the i386 memstick and having the same problem? Hmm. Thanks for the report. I'll take a look at the logs for i386, but they are generated the same way as the amd64, so in theory should not have any noticable difference. For amd64 and i386, native binaries are built, and installed into scratch directories; for powerpc and powerpc64, I just use the amd64 binaries, because I cannot directly use the chroot binaries for non-native architecture. The scripts chroot into the scratch directories, and run the real release builds. Have you tried using Crochet for this sort of thing? Since it was designed from the ground up for cross-building bootable images, it should avoid these issues. I have not, primarily because I was not aware of crochet when I originally started this. Although, by using the release stuff from the base system, we do get a weekly run-test of the 'make release' bits in head/ and stable/9/, so in theory, there would be no surprises when it is -RELEASE time. The only fundamental limit right now is that Crochet uses the host system to build the UFS filesystems, so it can't build big-endian MIPS images on i386, for example. Yes, I have this same issue with sparc64. Glen pgpO68TBLrIbU.pgp Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
Glen Barber g...@freebsd.org wrote in 20130608173411.gd13...@glenbarber.us: gj On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote: gj Because the userland is 32-bit and the kernel is 64-bit, something gj goes wrong, but interestingly not wrong enough that the script fails gj entirely. So, the paritions appear to be created, but in reality, they gj are not. gj gj So, for the snapshots case, the solution is to write the memstick image gj from outside of the chroot environment, which is easy to do because gj I already do this for creating the VM disk images (interestingly for the gj same reason as the memstick creation failure). I do not think there is a problem with cross building in chroot. allbsd.org is also generating i386 snapshots on an amd64 box in almost the same way as generate-release.sh, but the memstick images already generated were not broken as far as I can check. Although I do not use generate-release.sh on it because I added another build world stage in chroot before cross compiling, the difference is small. What was exactly gone wrong in 32-bit binary on 64-bit kernel? -- Hiroki pgpa7n05DnipG.pgp Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
Tim Kientzle t...@kientzle.com wrote in 926ef579-8ac9-4a98-8a81-4e978a627...@kientzle.com: ti ti On Jun 8, 2013, at 10:34 AM, Glen Barber wrote: ti ti On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote: ti Has anyone else tried the i386 memstick and having the same problem? ti ti ti Hmm. Thanks for the report. I'll take a look at the logs for i386, but ti they are generated the same way as the amd64, so in theory should not ti have any noticable difference. ti ti ti For amd64 and i386, native binaries are built, and installed into ti scratch directories; for powerpc and powerpc64, I just use the amd64 ti binaries, because I cannot directly use the chroot binaries for ti non-native architecture. ti ti The scripts chroot into the scratch directories, and run the real ti release builds. ti ti Have you tried using Crochet for this sort of thing? ti ti Since it was designed from the ground up for cross-building ti bootable images, it should avoid these issues. ti ti The only fundamental limit right now is that Crochet uses ti the host system to build the UFS filesystems, so it can't ti build big-endian MIPS images on i386, for example. makefs does not work? It can build BE FFS images on LE platforms. -- Hiroki pgpSfiguwNi65.pgp Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
On 06/08/13 14:17, Glen Barber wrote: On Sat, Jun 08, 2013 at 12:10:16PM -0700, Tim Kientzle wrote: On Jun 8, 2013, at 10:34 AM, Glen Barber wrote: On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote: Has anyone else tried the i386 memstick and having the same problem? Hmm. Thanks for the report. I'll take a look at the logs for i386, but they are generated the same way as the amd64, so in theory should not have any noticable difference. For amd64 and i386, native binaries are built, and installed into scratch directories; for powerpc and powerpc64, I just use the amd64 binaries, because I cannot directly use the chroot binaries for non-native architecture. The scripts chroot into the scratch directories, and run the real release builds. Have you tried using Crochet for this sort of thing? Since it was designed from the ground up for cross-building bootable images, it should avoid these issues. I have not, primarily because I was not aware of crochet when I originally started this. Although, by using the release stuff from the base system, we do get a weekly run-test of the 'make release' bits in head/ and stable/9/, so in theory, there would be no surprises when it is -RELEASE time. The only fundamental limit right now is that Crochet uses the host system to build the UFS filesystems, so it can't build big-endian MIPS images on i386, for example. Yes, I have this same issue with sparc64. Why not use makefs? It can build cross-endian UFS images. -Nathan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: 10-CURRENT i386 memstick snapshots broken?
On Sat, Jun 08, 2013 at 02:18:48PM -0500, Nathan Whitehorn wrote: On 06/08/13 14:17, Glen Barber wrote: On Sat, Jun 08, 2013 at 12:10:16PM -0700, Tim Kientzle wrote: On Jun 8, 2013, at 10:34 AM, Glen Barber wrote: On Fri, Jun 07, 2013 at 05:22:56PM -0400, Glen Barber wrote: Has anyone else tried the i386 memstick and having the same problem? Hmm. Thanks for the report. I'll take a look at the logs for i386, but they are generated the same way as the amd64, so in theory should not have any noticable difference. For amd64 and i386, native binaries are built, and installed into scratch directories; for powerpc and powerpc64, I just use the amd64 binaries, because I cannot directly use the chroot binaries for non-native architecture. The scripts chroot into the scratch directories, and run the real release builds. Have you tried using Crochet for this sort of thing? Since it was designed from the ground up for cross-building bootable images, it should avoid these issues. I have not, primarily because I was not aware of crochet when I originally started this. Although, by using the release stuff from the base system, we do get a weekly run-test of the 'make release' bits in head/ and stable/9/, so in theory, there would be no surprises when it is -RELEASE time. The only fundamental limit right now is that Crochet uses the host system to build the UFS filesystems, so it can't build big-endian MIPS images on i386, for example. Yes, I have this same issue with sparc64. Why not use makefs? It can build cross-endian UFS images. The scripts do (as far as I recall for sparc64) use makefs, but a port is needed to create the sparc-capable ISO image. Glen pgptKlnQa3HD9.pgp Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
On Sun, Jun 09, 2013 at 05:01:00AM +0900, Hiroki Sato wrote: gj Because the userland is 32-bit and the kernel is 64-bit, something gj goes wrong, but interestingly not wrong enough that the script fails gj entirely. So, the paritions appear to be created, but in reality, they gj are not. gj gj So, for the snapshots case, the solution is to write the memstick image gj from outside of the chroot environment, which is easy to do because gj I already do this for creating the VM disk images (interestingly for the gj same reason as the memstick creation failure). I do not think there is a problem with cross building in chroot. cross-building, no, there is no problem. allbsd.org is also generating i386 snapshots on an amd64 box in almost the same way as generate-release.sh, but the memstick images already generated were not broken as far as I can check. Although I do not use generate-release.sh on it because I added another build world stage in chroot before cross compiling, the difference is small. What was exactly gone wrong in 32-bit binary on 64-bit kernel? The problem is creating the gpart(8) partition scheme on the md(4) device. Below follows script(1) output of what the make-memstick.sh script does: Script started on Sun Jun 9 00:41:08 2013 root@snap:/snap/releng # chroot /snap/releng/10-i386-snap root@snap:/ # cd /usr/obj/usr/src/release root@snap:/usr/obj/usr/src/release # echo \ '/dev/ufs/FreeBSD_Install / ufs ro,noatime 1 1' release/etc/fstab root@snap:/usr/obj/usr/src/release # makefs -B \ little -o label=FreeBSD_Install test.img release Calculated size of `test.img': 649420800 bytes, 12922 inodes Extent size set to 8192 test.img: 619.3MB (1268400 sectors) block size 8192, fragment size 1024 using 12 cylinder groups of 54.40MB, 6963 blks, 1152 inodes. super-block backups (for fsck -b #) at: 32, 111440, 222848, 334256, 445664, 557072, 668480, 779888, 891296, 1002704, 1114112, 1225520, Populating `test.img' Image `test.img' complete root@snap:/usr/obj/usr/src/release # mdconfig -a -t vnode -f test.img md0 All fine up until this point. Now the gpart(8) partition is created: root@snap:/usr/obj/usr/src/release # gpart create -s BSD /dev/md0 gpart: Inappropriate ioctl for device root@snap:/usr/obj/usr/src/release # gpart list md0 Segmentation fault (core dumped) I also ran into this when initially creating the 8.4-RELEASE memory stick images, which uses fdisk(8) instead of gpart(8). I basically attributed this to probably shouldn't be expected to work, such as doing netstat(1) on within 32-bit chroot/jail on a 64-bit kernel. root@snap:/usr/obj/usr/src/release # gdb -c ./gpart.core gpart GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i386-marcel-freebsd...(no debugging symbols found)... Core was generated by `gpart'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libgeom.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libgeom.so.5 Reading symbols from /lib/libsbuf.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libsbuf.so.6 Reading symbols from /lib/libbsdxml.so.4...(no debugging symbols found)...done. Loaded symbols for /lib/libbsdxml.so.4 Reading symbols from /lib/libutil.so.9...(no debugging symbols found)...done. Loaded symbols for /lib/libutil.so.9 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /lib/geom/geom_part.so...(no debugging symbols found)...done. Loaded symbols for /lib/geom/geom_part.so Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x28070205 in geom_deletetree () from /lib/libgeom.so.5 (gdb) bt #0 0x28070205 in geom_deletetree () from /lib/libgeom.so.5 #1 0x08049b43 in ?? () #2 0xcbf0 in ?? () #3 0x28813050 in ?? () #4 0x0804cb1a in ?? () #5 0x28813030 in ?? () #6 0x0804e63d in __stderrp () #7 0x in ?? () I can rebuild gpart(8) with debugging symbols enabled tomorrow. The machine is churning through builds at the moment. Glen pgpBED76ulpGg.pgp Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
On Sat, 8 Jun 2013, Glen Barber wrote: The problem is creating the gpart(8) partition scheme on the md(4) device. Below follows script(1) output of what the make-memstick.sh script does: Script started on Sun Jun 9 00:41:08 2013 root@snap:/snap/releng # chroot /snap/releng/10-i386-snap root@snap:/ # cd /usr/obj/usr/src/release root@snap:/usr/obj/usr/src/release # echo \ '/dev/ufs/FreeBSD_Install / ufs ro,noatime 1 1' release/etc/fstab root@snap:/usr/obj/usr/src/release # makefs -B \ little -o label=FreeBSD_Install test.img release Calculated size of `test.img': 649420800 bytes, 12922 inodes Extent size set to 8192 test.img: 619.3MB (1268400 sectors) block size 8192, fragment size 1024 using 12 cylinder groups of 54.40MB, 6963 blks, 1152 inodes. super-block backups (for fsck -b #) at: 32, 111440, 222848, 334256, 445664, 557072, 668480, 779888, 891296, 1002704, 1114112, 1225520, Populating `test.img' Image `test.img' complete root@snap:/usr/obj/usr/src/release # mdconfig -a -t vnode -f test.img md0 All fine up until this point. Now the gpart(8) partition is created: root@snap:/usr/obj/usr/src/release # gpart create -s BSD /dev/md0 gpart: Inappropriate ioctl for device root@snap:/usr/obj/usr/src/release # gpart list md0 Segmentation fault (core dumped) This might be a good time to stop using a bare bsdlabel (aka dangerously dedicated). MBR plus bsdlabel is not great, but more widely understood, and other operating systems will at least recognize the MBR slice. I can help with this if needed. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
10-CURRENT i386 memstick snapshots broken?
Greetings - I had originally started playing with the 10-CURRENT amd64 release a couple of months ago (no complaints here - keep up the good work!) and had used the memstick snapshot to set things up back then with no problems. Now I'd like to wipe everything and switch to the i386 version, but the memstick images (all of the last 4 or 5) seem to be broken. My machine sees that the USB stick is there, but won't boot off of it at all. When I do 'fdisk' on /dev/da0 with the i386 stuff loaded, I get a invalid partition error, so on a hunch I decided to take a closer look at the memstick images using dd if=xxx-memstick bs=512 count=1 | hd The amd64 images look like what I would expect: some bootblock code with a partition table at the end and the a5 type marker. The i386 image, however, is ALL '00'. It would appear the whatever puts those memstick images together (src/release/i386/make-memstick.sh, perhaps?) is failing for some reason. Has anyone else tried the i386 memstick and having the same problem? Jimmy ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: 10-CURRENT i386 memstick snapshots broken?
On Fri, Jun 07, 2013 at 03:51:29PM -0500, Jimmy wrote: Greetings - I had originally started playing with the 10-CURRENT amd64 release a couple of months ago (no complaints here - keep up the good work!) and had used the memstick snapshot to set things up back then with no problems. Now I'd like to wipe everything and switch to the i386 version, but the memstick images (all of the last 4 or 5) seem to be broken. My machine sees that the USB stick is there, but won't boot off of it at all. When I do 'fdisk' on /dev/da0 with the i386 stuff loaded, I get a invalid partition error, so on a hunch I decided to take a closer look at the memstick images using dd if=xxx-memstick bs=512 count=1 | hd The amd64 images look like what I would expect: some bootblock code with a partition table at the end and the a5 type marker. The i386 image, however, is ALL '00'. It would appear the whatever puts those memstick images together (src/release/i386/make-memstick.sh, perhaps?) is failing for some reason. Has anyone else tried the i386 memstick and having the same problem? Hmm. Thanks for the report. I'll take a look at the logs for i386, but they are generated the same way as the amd64, so in theory should not have any noticable difference. Glen pgpXEttKEd1kP.pgp Description: PGP signature
Re: 10-CURRENT i386 memstick snapshots broken?
On 06/07/13 22:51, Jimmy wrote: Greetings - I had originally started playing with the 10-CURRENT amd64 release a couple of months ago (no complaints here - keep up the good work!) and had used the memstick snapshot to set things up back then with no problems. Now I'd like to wipe everything and switch to the i386 version, but the memstick images (all of the last 4 or 5) seem to be broken. My machine sees that the USB stick is there, but won't boot off of it at all. When I do 'fdisk' on /dev/da0 with the i386 stuff loaded, I get a invalid partition error, so on a hunch I decided to take a closer look at the memstick images using dd if=xxx-memstick bs=512 count=1 | hd The amd64 images look like what I would expect: some bootblock code with a partition table at the end and the a5 type marker. The i386 image, however, is ALL '00'. It would appear the whatever puts those memstick images together (src/release/i386/make-memstick.sh, perhaps?) is failing for some reason. Has anyone else tried the i386 memstick and having the same problem? Jimmy Hi, The memstick images does not contain an MBR. That's why some bioses won't boot them. You can fix this by using: fdisk -BI /dev/daX dd if=xxx.img of=/dev/daXs1 count=65536 --HPS ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
