Re: FreeBSD-9.1: machine reboots during snapshot creation, LORs found
On Fri, Jul 19, 2013 at 07:32:47AM +0200, Andre Albsmeier wrote: On Thu, 04-Jul-2013 at 19:25:28 +0200, Konstantin Belousov wrote: Patch is to improve debugging. I probably commit it after the issue is closed. Arguments against the commit is that the change imposes small performance penalty due to save and restore of the %ebp (I doubt that this is measureable by any means). Also, arguably, such change should be done for all functions in support.s, but bcopy() is the hot spot. Thanks to this patch, we (you ;-) were able to track down the problem. So how are we going to deal with this debugging patch itself? My suggestion would be to #ifdef it somehow so on one hand it will be availabe in future (and with bcopy being used a lot probability is high it might help in other places), on the other hand it won't steal cycles during normal use. This was already committed as r253328 in HEAD. I dislike the #ifdef, since the stack access is interwinned through all text, making the function forked. pgp1Tt4X15zWq.pgp Description: PGP signature
Re: make buildworld is now 50% slower
On Sun, Jul 07, 2013 at 11:50:29AM +0300, Daniel Braniss wrote: On Fri, Jul 05, 2013 at 02:39:00PM +0200, Dimitry Andric wrote: [redirecting to the correct mailing list, freebsd-stable@ ...] On Jul 5, 2013, at 10:53, Daniel Braniss da...@cs.huji.ac.il wrote: after today's update of 9.1-STABLE I noticed that make build[world|kernel] are taking conciderable more time, is it because the upgrade of clang? and if so, is the code produced any better? before: buildwordl:26m4.52s real 2h28m32.12s user 36m6.27s sys buildkernel: 7m29.42s real 23m22.22s user 4m26.26s sys today: buildwordl: 34m29.80s real 2h38m9.37s user 37m7.61s sys buildkernel:15m31.52s real 22m59.40s user 4m33.06s sys Ehm, your user and sys times are not that much different at all, they add up to about 5% slower for buildworld, and 1% faster for build kernel. Are you sure nothing else is running on that machine, eating up CPU time while you are building? :) But yes, clang 3.3 is of course somewhat larger than 3.2. You might especially notice that, if you are using gcc, which is very slow at compiling C++. In any case, if you do not care about clang, just set WITHOUT_CLANG= in your /etc/src.conf, and you can shave off some build time. I just built world/kernel (stable/9 r252769) 5 hours ago. Results: time make -j4 buildworld = roughly 21 minutes on my hardware time make -j4 buildkernel = roughly 8 minutes on my hardware It's been a long time since I saw such numbers, maybe it's time to see where time is being spent, I will run it without clang to compare with your numbers. These numbers are about the norm for me, meaning I do not see a substantial increase in build times. Key point: I do not use/build/grok clang, i.e. WITHOUT_CLANG=true is in my src.conf. But I am aware of the big clang change in r252723. If hardware details are wanted, ask, but I don't think it's relevant to what the root cause is. from what you are saying, I guess clang is not responsible. looking for my Sherlock Holmes hat. Some points to those numbers I stated above: - System is an Intel Q9550 with 8GB of RAM - Single SSD (UFS2+SU+TRIM) is used for root, /usr, /var, /tmp, and swap - /usr/src is on ZFS (raidz1 + 3 disks) -- however I got equally small numbers when it was on the SSD - /usr/src is using compression=lz4 (to folks from -fs: yeah, I'm trying it out to see how much of an impact it has on interactivity. I can still tell when it kicks in, but it's way, way better than lzjb. Rather not get into that here) - Contents of /etc/src.conf (to give you some idea of what I disable): WITHOUT_ATM=true WITHOUT_BLUETOOTH=true WITHOUT_CLANG=true WITHOUT_FLOPPY=true WITHOUT_FREEBSD_UPDATE=true WITHOUT_INET6=true WITHOUT_IPFILTER=true WITHOUT_IPX=true WITHOUT_KERBEROS=true WITHOUT_LIB32=true WITHOUT_LPR=true WITHOUT_NDIS=true WITHOUT_NETGRAPH=true WITHOUT_PAM_SUPPORT=true WITHOUT_PPP=true WITHOUT_SENDMAIL=true WITHOUT_WIRELESS=true WITH_OPENSSH_NONE_CIPHER=true It's WITHOUT_CLANG that cuts down the buildworld time by a *huge* amount (I remember when it got introduced, my buildworld jumped up to something like 40 minutes); the rest probably save a minute or two at most. - /etc/make.conf doesn't contain much that's relevant, other than: CPUTYPE?=core2 # For DTrace; also affects ports STRIP= CFLAGS+= -fno-omit-frame-pointer - I do some tweaks in /etc/sysctl.conf (mainly vfs.read_min and vfs.read_max), but I will admit I am not completely sure what those do quite yet (I just saw the commit from scottl@ a while back talking about how an increased vfs.read_min helps them at Netflix quite a lot). I also adjust kern.maxvnodes. - Some ZFS ARC settings are adjusted in /boot/loader.conf (I'm playing with some stuff I read in Andriy Gapon's ZFS PDF), but they definitely do not have a major impact on the numbers I listed off. - I do increase kern.maxdsiz, kern.dfldsiz, and kern.maxssiz in /boot/loader.conf to 2560M/2560M/256M respectively, but that was mainly from the days when I ran MySQL and needed a huge userland processes. All in all my numbers are low/small because of two things: the SSD, and WITHOUT_CLANG. Hope this gives you somewhere to start/stuff to ponder. indeed! on my pretty much standard dev machine, PowerEdge R710, X5550 @ 2.67GHz, 16384 MB, root on ufs, the rest is via ZFS, this is what I'm getting for buildworld: with clang 27m32.91s real 2h42m52.82s user 36m20.69s sys without clang 13m24.19s real 1h23m26.52s user 29m5.18s sys on a similar machine but with root via NFS, and /var on ZFS, the numbers are similar: with clang 23m30.92s real 2h9m8.85s user 29m27.84s sys without clang 12m7.53s real 1h7m54.24s user 23m9.78s sys (this host is newer, PowerEdge
Re: 9.2PRERELEASE ZFS panic in lzjb_compress
19.07.2013 07:04, olivier wrote:
Hi,
Running 9.2-PRERELEASE #19 r253313 I got the following panic
Fatal trap 12: page fault while in kernel mode
cpuid = 22; apic id = 46
fault virtual address = 0xff827ebca30c
fault code = supervisor read data, page not present
instruction pointer = 0x20:0x81983055
stack pointer = 0x28:0xffcf75bd60a0
frame pointer = 0x28:0xffcf75bd68f0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 0 (zio_write_issue_hig)
trap number = 12
panic: page fault
cpuid = 22
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame
0xffcf75bd5b30
kdb_backtrace() at kdb_backtrace+0x37/frame 0xffcf75bd5bf0
panic() at panic+0x1ce/frame 0xffcf75bd5cf0
trap_fatal() at trap_fatal+0x290/frame 0xffcf75bd5d50
trap_pfault() at trap_pfault+0x211/frame 0xffcf75bd5de0
trap() at trap+0x344/frame 0xffcf75bd5fe0
calltrap() at calltrap+0x8/frame 0xffcf75bd5fe0
--- trap 0xc, rip = 0x81983055, rsp = 0xffcf75bd60a0, rbp =
0xffcf75bd68f0 ---
lzjb_compress() at lzjb_compress+0x185/frame 0xffcf75bd68f0
zio_compress_data() at zio_compress_data+0x92/frame 0xffcf75bd6920
zio_write_bp_init() at zio_write_bp_init+0x24b/frame 0xffcf75bd6970
zio_execute() at zio_execute+0xc3/frame 0xffcf75bd69b0
taskqueue_run_locked() at taskqueue_run_locked+0x74/frame 0xffcf75bd6a00
taskqueue_thread_loop() at taskqueue_thread_loop+0x46/frame
0xffcf75bd6a20
fork_exit() at fork_exit+0x11f/frame 0xffcf75bd6a70
fork_trampoline() at fork_trampoline+0xe/frame 0xffcf75bd6a70
--- trap 0, rip = 0, rsp = 0xffcf75bd6b30, rbp = 0 ---
lzjb_compress+0x185 corresponds to line 85 in
80 cpy = src - offset;
81 if (cpy = (uchar_t *)s_start cpy != src
82src[0] == cpy[0] src[1] == cpy[1] src[2] == cpy[2]) {
83 *copymap |= copymask;
84 for (mlen = MATCH_MIN; mlen MATCH_MAX; mlen++)
85 if (src[mlen] != cpy[mlen])
86 break;
87 *dst++ = ((mlen - MATCH_MIN) (NBBY - MATCH_BITS)) |
88(offset NBBY);
89 *dst++ = (uchar_t)offset;
I think it's the first time I've seen this panic. It happened while doing a
send/receive. I have two pools with lzjb compression; I don't know which of
these pools caused the problem, but one of them was the source of the
send/receive.
I only have a textdump but I'm happy to try to provide more information
that could help anyone look into this.
Thanks
Olivier
Oh, I can add to this one. I have a full core dump of the same problem
caused by copying large set of files from lzjb compressed pool to lz4
compressed pool. vfs.zfs.recover was set.
#1 0x8039d954 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:449
#2 0x8039ddce in panic (fmt=value optimized out)
at /usr/src/sys/kern/kern_shutdown.c:637
#3 0x80620a6a in trap_fatal (frame=value optimized out,
eva=value optimized out) at /usr/src/sys/amd64/amd64/trap.c:879
#4 0x80620d25 in trap_pfault (frame=0x0, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:700
#5 0x806204f6 in trap (frame=0xff821ca43600)
at /usr/src/sys/amd64/amd64/trap.c:463
#6 0x8060a032 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:232
#7 0x805a9367 in vm_page_alloc (object=0x80a34030,
pindex=16633, req=97) at /usr/src/sys/vm/vm_page.c:1445
#8 0x8059c42e in kmem_back (map=0xfe0001e8,
addr=18446743524021862400, size=16384, flags=value optimized out)
at /usr/src/sys/vm/vm_kern.c:362
#9 0x8059c2ac in kmem_malloc (map=0xfe0001e8, size=16384,
flags=257) at /usr/src/sys/vm/vm_kern.c:313
#10 0x80595104 in uma_large_malloc (size=value optimized out,
wait=257) at /usr/src/sys/vm/uma_core.c:994
#11 0x80386b80 in malloc (size=16384, mtp=0x80ea7c40,
flags=0)
at /usr/src/sys/kern/kern_malloc.c:492
#12 0x80c9e13c in lz4_compress (s_start=0xff80d0b19000,
d_start=0xff8159445000, s_len=131072, d_len=114688, n=-2)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/lz4.c:843
#13 0x80cdde25 in zio_compress_data (c=value optimized out,
src=value optimized out, dst=0xff8159445000, s_len=131072)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c:109
#14 0x80cda012 in zio_write_bp_init (zio=0xfe0143a12000)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1107
#15 0x80cd8ec6 in zio_execute (zio=0xfe0143a12000)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1305
#16 0x803e25e6 in taskqueue_run_locked (queue=0xfe00060ca300)
at /usr/src/sys/kern/subr_taskqueue.c:312
Virtio drivers and FreeBSD 9.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello everybody, I have some trouble with the following configuration: - - a Samsung SSD 840 Pro drive of 128 GB - - a Debian Wheezy (7.1) as a KVM host - - multiple Debian 7.1 and FreeBSD 9.1 guests on top of it I have installed the virtio drivers (from http://people.freebsd.org/~kuriyama/virtio/) on all guests, and configured the KVM domain.xml files accordingly. Then I ran some I/O tests using iozone. And as you can see on the results below, there is a big difference between the Debian guest and the FreeBSD guest. Is there any option I could have forgotten to set up in FreeBSD that could explain this difference? On the host server: iozone -s 64M -r 4k -i 0 -i 1 -i 2 random random KB reclen write rewritereadrereadreadwrite 655364 258532 553602 922146 969225 1523242 530324 On the FreeBSD guest: random random KB reclen write rewritereadrereadreadwrite 655364 61687 128261 574681 580237 537510 20690 On the Debian guest: random random KB reclen write rewritereadrereadreadwrite 655364 119604 481670 878853 935172 1328093 473255 Thank you. - -- Vos conversations sont privées, sécurisez-les. Your conversations are private, secure them. https://gpgtools.org or http://gpg4win.org -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJR6V0qAAoJEIylYwQV0UQcGAgIAJB6PCo6Bc5ewpBPVFf1ZaoU aePblYD2YNgkIRA074WHMFgdudoIzc+U/0IpVSM+N+OvpRwyFXrAW8PuY46Hrdxp Qnh+d9uYFaN8tZqKbpbWvmSziwepHBfozrsIugoP+7fohNAro0/ycpC8EYvuqJI6 JuOqVAOK87fuTXKV0GbwSuhxVvrvaxOTY083QxGoiS0isieEE2DCwFa5y/blmQPe yJTMn45ykbkkskDUWrdNne3lsmM1PFktfyFkslxSuaaNN1Nm+tWKgRDQm4lusRup 3IRy/ZNVI2Gu0/fnqs3Z6nY+LSke1qsKL9WveQEZCZLt/BhkyHCQVlclwGKhZjA= =Kl+S -END PGP SIGNATURE- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Help with filing a [maybe] ZFS/mmap bug.
Richard Todd writes: On Thu, Jul 18, 2013 at 11:40:51AM -0700, George Hartzell wrote: [...] [...] In my case I'd want to find a particular set of file size, offset, and insertion size that triggers the problem and code up a c/c++ equiv. of the mmap calls that py-mutagen does. Right? Yeah. I'm stuck. Or I've discovered something relevant. Or both. I've identified a slightly simpler test case. I load my handful of test albums, look up single, particular album, and save a particular track. The tagged flac file appears to be valid. Then I reboot. Now the flac file is invalid. It's repeatable, which is useful. Following the lead of your test script I created a new zfs filesystem, mounted it, and had picard save the tagged files there. After exiting picard the files appears to be valid. After unmounting and remounting the filesystem the file *still* appears to be valid. After rebooting, the file *still* appears to be valid. So, it would seem that there's something about the filesystem in which my home directory resides that contributes to the problem. The only obvious thing I saw is that my homedir filesystem has a quota and is 80% full. I tried creating a new, small, zfs filesystem and running the test there. The tagged flac file validates successfully, I do not see the problem (the single file makes the filesystem 88% full). All of the filesystems have automagically created snapshots, so I tried creating a snapshot of the new zfs filesystem before running through the test case. I was still unable to replicate the problem. My spin on your gen4.cpp test case (modified to use the filesize and offset that picard uses) does not generate a difference when run in my home directory followed by a reboot (picard calls insert_bytes twice, using either set of values does not cause a problem). The only difference I see in zfs get all output (excluding obvious sizes, etc...) is that the new filesystem has xattr on via the default, whereas my home directory has it off via temporary. I'm not sure why it's off. So, I currently have a repeatable, not-too-efficient test case using my home directory. I am unable to repeat the test case using a newly created zfs filesystem (even a very small one) nor am I able to make any headway with Richard's test case. As I described in another thread with Andriy, add INVARIANTS and INVARIANT_SUPPORT into the kernel did not lead to any different behaviour, in fact the experiments described above were run on this new kernel. Any suggestions for a next step? g. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Help with filing a [maybe] ZFS/mmap bug.
George Hartzell writes: [...] So, it would seem that there's something about the filesystem in which my home directory resides that contributes to the problem. [...] Another data point. I just ran through my test case, saving the tagged and transcoded files into /tmp, a zfs filesystem that was created back when I built up the system (contemporaneously with /usr/home). I was unable to trigger the bug there. As I control, I then ran through the test case, saving a directory in my home directory and triggered the bug. I then created a new directory /usr/home/foo (within the same zfs filesystem as my home directory). I was unable to trigger the bug there either. I then ran through all 165 flac files in the full test case, saving the results to /usr/home/foo. After exiting picard and running flac -t on all of the files I had errors on many files, including the file in my single-file test case above. I did not even need to reboot. I then ran the single file test case, saving into /usr/foo (as above) and was now able to observe the error after a reboot. I then ran the single file test case (again to make sure I wasn't crazy), saving into /usr/foo (as above) and was now able to observe the error after a reboot. One more control. Create /usr/home/bar. Run single file test case. Reboot. This time I observed an invalid flac. Not sure what this means about the test case above. Sigh. g. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
