Re: my experience with FreeBSD Xen guest
I've only just started seeing filesystem corruption under high load. -current Xen survived buildworlds in my initial testing so I was focusing more on fixing network related bugs. It is quite possible there are bugs in the VM system and/or Xen block device code which are causing instabilities. Please file a FreeBSD PR so the problem isn't forgotten about. Thanks, Adrian 2009/6/17 Alex Urbanowicz : > Hello > > I've spent last four days trying to get a stable 8.0-CURRENT on a CentOS > 5.3. I am not very versed in BSD, so some of the problems may come from > applying generic Unix approach to BSD environment. > > * disk operations are unstable. I attach dmesg from boot of a freshly > fscked filesystem (16G LVM on host), the two stack backtraces are > routinely happening during boot sequence. Also massive operations like > buildworld usually lead to more syscall errors and routine filesystem > corruption > > * sometimes the guest just crashes to kernel debugger, I have not > isolated any particular reason why it happens > > * during the four days I only managed to successfully compile kernel > once and world twice, installworld always fails, the last one with > following error: > > [...] > ===> secure (install) > ===> secure/lib (install) > ===> secure/lib/libcrypto (install) > ".depend", line 19879: Unassociated shell command "��ɫ���'�F/^�>� > 闆 �� � > ".depend", line 19880: Need an operator > ".depend", line 19881: Need an operator > ".depend", line 19883: Missing dependency operator > ".depend", line 19884: Need an operator > No closing parenthesis in archive specification > ".depend", line 19885: Error in archive specification: " % > Z��F��� ��_o o�x;w,� �Fގ�/ 7�͚��" > No closing parenthesis in archive specification > ".depend", line 19886: Error in archive specification: "�" > ".depend", line 19888: Need an operator > ".depend", line 19889: Need an operator > ".depend", line 19893: Need an operator > ".depend", line 19894: Missing dependency operator > No closing parenthesis in archive specification > ".depend", line 19899: Error in archive specification: > "6pH�EX�� ��]9� 䝺8�" > ".depend", line 19910: Need an operator > make: fatal errors encountered -- cannot continue > *** Error code 1 > > Stop in /usr/src/secure/lib. > *** Error code 1 > > Stop in /usr/src/secure. > *** Error code 1 > > Stop in /usr/src. > *** Error code 1 > > Stop in /usr/src. > *** Error code 1 > > Stop in /usr/src. > *** Error code 1 > > Stop in /usr/src. > > Also, during the mentioned buildworld the following syscall error > happened: > > lock order reversal: > 1st 0xc393b81c snaplk (snaplk) > @ /usr/home/alex/bsd/sys/kern/vfs_vnops.c:295 > 2nd 0xc441037c ufs (ufs) > @ /usr/home/alex/bsd/sys/ufs/ffs/ffs_snapshot.c:1587 > KDB: stack backtrace: > X_db_sym_numargs(c0356957,e4d2f640,c0116b95,c01072bb,c035979e,...) at > X_db_sym_numargs+0x146 > kdb_backtrace(c01072bb,c035979e,c36ccf10,c36ccbd0,e4d2f69c,...) at > kdb_backtrace+0x29 > witness_display_spinlock(c035979e,c441037c,c034c8dc,c36ccbd0,c0371652,...) at > witness_display_spinlock+0x75 > witness_checkorder(c441037c,9,c0371652,633,0,...) at witness_checkorder > +0x839 > __lockmgr_args(c441037c,8,0,0,0,...) at __lockmgr_args+0x797 > ffs_snapremove(c4410324,0,e4d2f7d8,e4d2f7d8,c035e67e,...) at > ffs_snapremove+0x11f > ffs_truncate(c4410324,0,0,c00,0,...) at ffs_truncate+0x5f7 > ufs_inactive(e4d2fa58,c4410398,c4410324,c4410398,e4d2fa70,...) at > ufs_inactive+0x218 > VOP_INACTIVE_APV(c03b0200,e4d2fa58,c0360aae,925,c03c2160,...) at > VOP_INACTIVE_APV+0xa5 > vref(c03b0200,e4d2fa8c,c0360aae,8ab,127,...) at vref+0x136e > vput(c4410324,e4d2fac8,c0361c78,127,0,...) at vput+0x1cd > vn_close(c4410324,1,c3b02600,c3b1f240,c0c74880,...) at vn_close+0x19a > vn_close(c39611f8,c3b1f240,3,0,c39611f8,...) at vn_close+0x294 > _fdrop(c39611f8,c3b1f240,e4d2fb94,c01169dc,0,c0507a20,c3b1f2e4,c0507a20,c039b680,c4bd432c,6f4,c034e229,e4d2fbbc,c00de690,c4bd432c,8,c034e229,6f4) > at _fdrop+0x43 > closef(c39611f8,c3b1f240,6f4,6f1,c3b1f2e4,...) at closef+0x290 > fdfree(c3b1f240,0,c034eafc,103,1,...) at fdfree+0x3ea > exit1(c3b1f240,0,e4d2fd3c,c0329e23,c3b1f240,...) at exit1+0x533 > sys_exit(c3b1f240,e4d2fd08,4,c035a0f3,c039227c,...) at sys_exit+0x1d > syscall(e4d2fd48) at syscall+0x2a3 > Xint0x80_syscall() at Xint0x80_syscall+0x22 > --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x28129f63, esp = > 0xbf7fe33c, ebp = 0xbf7fe348 --- > > I guess that the both of the errors are related to fs corruption that > happened during the course of buildworld (before I started buildworld I > launched the guest from different image and fscked the / clean). > > The filesystem problems are severe, and block us from using FreeBSD/Xen. > > Regards > > Alex > > ___ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org" > > _
Re: my experience with FreeBSD Xen guest
Hi Alex, do you run 32bit on a 32 or 64bit dom0? Maybe you should try a more recent version of Xen (RHEL has 3.1) This repo might help you : http://www.gitco.de/linux/x86_64/centos/5/rpms/ cheers Kai - "Adrian Chadd" schrieb: > I've only just started seeing filesystem corruption under high load. > -current Xen survived buildworlds in my initial testing so I was > focusing more on fixing network related bugs. > > It is quite possible there are bugs in the VM system and/or Xen block > device code which are causing instabilities. > > Please file a FreeBSD PR so the problem isn't forgotten about. > > Thanks, > > > > Adrian > > 2009/6/17 Alex Urbanowicz : > > Hello > > > > I've spent last four days trying to get a stable 8.0-CURRENT on a > CentOS > > 5.3. I am not very versed in BSD, so some of the problems may come > from > > applying generic Unix approach to BSD environment. > > > > * disk operations are unstable. I attach dmesg from boot of a > freshly > > fscked filesystem (16G LVM on host), the two stack backtraces are > > routinely happening during boot sequence. Also massive operations > like > > buildworld usually lead to more syscall errors and routine > filesystem > > corruption > > > > * sometimes the guest just crashes to kernel debugger, I have not > > isolated any particular reason why it happens > > > > * during the four days I only managed to successfully compile > kernel > > once and world twice, installworld always fails, the last one with > > following error: > > > > [...] > > ===> secure (install) > > ===> secure/lib (install) > > ===> secure/lib/libcrypto (install) > > ".depend", line 19879: Unassociated shell command > "��ɫ���'�F/^�>� > > 闆 �� � > > ".depend", line 19880: Need an operator > > ".depend", line 19881: Need an operator > > ".depend", line 19883: Missing dependency operator > > ".depend", line 19884: Need an operator > > No closing parenthesis in archive specification > > ".depend", line 19885: Error in archive specification: " % > > Z��F��� ��_o o�x;w,� �Fގ�/ 7�͚��" > > No closing parenthesis in archive specification > > ".depend", line 19886: Error in archive specification: "�" > > ".depend", line 19888: Need an operator > > ".depend", line 19889: Need an operator > > ".depend", line 19893: Need an operator > > ".depend", line 19894: Missing dependency operator > > No closing parenthesis in archive specification > > ".depend", line 19899: Error in archive specification: > > "6pH�EX�� ��]9� 䝺8�" > > ".depend", line 19910: Need an operator > > make: fatal errors encountered -- cannot continue > > *** Error code 1 > > > > Stop in /usr/src/secure/lib. > > *** Error code 1 > > > > Stop in /usr/src/secure. > > *** Error code 1 > > > > Stop in /usr/src. > > *** Error code 1 > > > > Stop in /usr/src. > > *** Error code 1 > > > > Stop in /usr/src. > > *** Error code 1 > > > > Stop in /usr/src. > > > > Also, during the mentioned buildworld the following syscall error > > happened: > > > > lock order reversal: > > 1st 0xc393b81c snaplk (snaplk) > > @ /usr/home/alex/bsd/sys/kern/vfs_vnops.c:295 > > 2nd 0xc441037c ufs (ufs) > > @ /usr/home/alex/bsd/sys/ufs/ffs/ffs_snapshot.c:1587 > > KDB: stack backtrace: > > X_db_sym_numargs(c0356957,e4d2f640,c0116b95,c01072bb,c035979e,...) > at > > X_db_sym_numargs+0x146 > > kdb_backtrace(c01072bb,c035979e,c36ccf10,c36ccbd0,e4d2f69c,...) at > > kdb_backtrace+0x29 > > > witness_display_spinlock(c035979e,c441037c,c034c8dc,c36ccbd0,c0371652,...) > at witness_display_spinlock+0x75 > > witness_checkorder(c441037c,9,c0371652,633,0,...) at > witness_checkorder > > +0x839 > > __lockmgr_args(c441037c,8,0,0,0,...) at __lockmgr_args+0x797 > > ffs_snapremove(c4410324,0,e4d2f7d8,e4d2f7d8,c035e67e,...) at > > ffs_snapremove+0x11f > > ffs_truncate(c4410324,0,0,c00,0,...) at ffs_truncate+0x5f7 > > ufs_inactive(e4d2fa58,c4410398,c4410324,c4410398,e4d2fa70,...) at > > ufs_inactive+0x218 > > VOP_INACTIVE_APV(c03b0200,e4d2fa58,c0360aae,925,c03c2160,...) at > > VOP_INACTIVE_APV+0xa5 > > vref(c03b0200,e4d2fa8c,c0360aae,8ab,127,...) at vref+0x136e > > vput(c4410324,e4d2fac8,c0361c78,127,0,...) at vput+0x1cd > > vn_close(c4410324,1,c3b02600,c3b1f240,c0c74880,...) at > vn_close+0x19a > > vn_close(c39611f8,c3b1f240,3,0,c39611f8,...) at vn_close+0x294 > > > _fdrop(c39611f8,c3b1f240,e4d2fb94,c01169dc,0,c0507a20,c3b1f2e4,c0507a20,c039b680,c4bd432c,6f4,c034e229,e4d2fbbc,c00de690,c4bd432c,8,c034e229,6f4) > at _fdrop+0x43 > > closef(c39611f8,c3b1f240,6f4,6f1,c3b1f2e4,...) at closef+0x290 > > fdfree(c3b1f240,0,c034eafc,103,1,...) at fdfree+0x3ea > > exit1(c3b1f240,0,e4d2fd3c,c0329e23,c3b1f240,...) at exit1+0x533 > > sys_exit(c3b1f240,e4d2fd08,4,c035a0f3,c039227c,...) at > sys_exit+0x1d > > syscall(e4d2fd48) at syscall+0x2a3 > > Xint0x80_syscall() at Xint0x80_syscall+0x22 > > --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x28129f63, esp = > > 0xbf7fe33c, ebp = 0xbf7fe348 --- > > > > I guess that the both of the errors are related to fs
Re: my experience with FreeBSD Xen guest
On śro, 2009-06-17 at 14:27 +0200, Kai Mosebach wrote: > Hi Alex, > > do you run 32bit on a 32 or 64bit dom0? Maybe you should try a more recent > version of Xen (RHEL has 3.1) > This repo might help you : The host is x86_64. > http://www.gitco.de/linux/x86_64/centos/5/rpms/ I'll give it a try when I migrate the setup to experimental server, thank you. I reported the syscall failures and FS corruption to the bugtracker. Alex -- Janusz A. Urbanowicz | Artegence sp. z o.o. niniejsza wiadomość nie stanowi pisma ani zamówienia handlowego w rozumieniu kodeksu spółek handlowych (Dz.U. 2000 nr 94 poz. 1037) ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
