Re: Fatal trap 9 triggered by zfs?
Am 04.12.2009 um 17:52 schrieb Stefan Bethke: I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. Thanks to all who suggested bad hardware: it turned out to be a case of capacitor plague. With a new mainboard, everything appears to be working solidly again. Stefan -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Fatal trap 9 triggered by zfs?
I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC 2009 r...@xxx.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64 # zpool status pool: tank state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 ad4s1dONLINE 0 0 0 # cat /boot/loader.conf vfs.zfs.arc_max=512M vfs.zfs.prefetch_disable=1 vfs.zfs.zil_disable=1 Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0x80a39900 stack pointer = 0x28:0xff80622ddae0 frame pointer = 0x28:0xff80622ddb10 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 0 (spa_zio) trap number = 9 panic: general protection fault cpuid = 0 Uptime: 17h44m5s Physical memory: 3313 MB Dumping 1843 MB: 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652 1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428 1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204 1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 68 52 36 20 4 #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0x803374b9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8033790c in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x805cbb8d in trap_fatal (frame=0x9, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x805cc6f2 in trap (frame=0xff80622dda30) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x805b2223 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0x80a39900 in vdev_queue_agg_io_done (aio=0xff00374562d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:174 #7 0x80a4be6f in zio_done (zio=0xff00374562d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2243 #8 0x80a49e87 in zio_execute (zio=0xff00374562d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #9 0x809ed603 in taskq_run (arg=0xff008d8d0420, pending=Variable pending is not available. ) at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:108 #10 0x80373533 in taskqueue_run (queue=0xff00017e1400) at /usr/src/sys/kern/subr_taskqueue.c:239 #11 0x803737b6 in taskqueue_thread_loop (arg=Variable arg is not available. ) at /usr/src/sys/kern/subr_taskqueue.c:360 #12 0x8030e0b8 in fork_exit ( callout=0x80373770 taskqueue_thread_loop, arg=0xff00016434e0, frame=0xff80622ddc80) at /usr/src/sys/kern/kern_fork.c:843 #13 0x805b26fe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:561 #14 0x in ?? () #15 0x in ?? () #16 0x in ?? () #17 0x in ?? () #18 0x in ?? () #19 0x in ?? () #20 0x in ?? () #21 0x in ?? () #22 0x in ?? () #23 0x in ?? () #24 0x in ?? () #25 0x in ?? () #26 0x in ?? () #27 0x in ?? () #28 0x in ?? () #29 0x in ?? () #30 0x in ?? () #31 0x in ?? () #32 0x in ?? () #33 0x in ?? () #34 0x in ?? () #35 0x in ?? () #36 0x in ?? () #37 0x in ?? () #38 0x00c6c000 in ?? () #39 0x in ?? () #40 0x000b in ?? () #41 0x80832500 in affinity () #42 0xff000173c390 in ?? () #43 0xff80622dd240 in ?? () #44 0xff80622dd1f8 in ?? () #45 0xff00015ecab0 in ?? () #46 0x8035aa48 in sched_switch (td=0x80373770, newtd=0xff00016434e0, flags=Variable flags is not available. ) at /usr/src/sys/kern/sched_ule.c:1858 Previous frame inner to this frame (corrupt stack?) (kgdb) -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___
Re: Fatal trap 9 triggered by zfs?
Am 04.12.2009 um 17:52 schrieb Stefan Bethke: I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC 2009 r...@xxx.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64 # zpool status pool: tank state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 ad4s1dONLINE 0 0 0 # cat /boot/loader.conf vfs.zfs.arc_max=512M vfs.zfs.prefetch_disable=1 vfs.zfs.zil_disable=1 Got another, different one. Any tuning suggestions or similar? #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0x80337bd9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8033802c in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x805cc2ad in trap_fatal (frame=0x9, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x805cce12 in trap (frame=0xff80625db030) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x805b2943 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0x80586c7a in vm_map_entry_splay (addr=Variable addr is not available. ) at /usr/src/sys/vm/vm_map.c:771 #7 0x80587f37 in vm_map_lookup_entry (map=0xff0001e8, address=18446743523979624448, entry=0xff80625db170) at /usr/src/sys/vm/vm_map.c:1021 #8 0x80588aa3 in vm_map_delete (map=0xff0001e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2685 #9 0x80588e61 in vm_map_remove (map=0xff0001e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2774 #10 0x8057db85 in uma_large_free (slab=0xff005fcc7000) at /usr/src/sys/vm/uma_core.c:3021 #11 0x80325987 in free (addr=0xff80018b, mtp=0x80ac61e0) at /usr/src/sys/kern/kern_malloc.c:471 #12 0x80a36d03 in vdev_cache_evict (vc=0xff0001723ce0, ve=0xff003dd52200) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:151 #13 0x80a372ad in vdev_cache_read (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:182 #14 0x80a4a954 in zio_vdev_io_start (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1814 #15 0x80a4ae87 in zio_execute (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #16 0x80a3a080 in vdev_mirror_io_start (zio=0xff005f811b40) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303 #17 0x80a4ae87 in zio_execute (zio=0xff005f811b40) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #18 0x809ff45a in arc_read_nolock (pio=0xff005f66d5a0, spa=0xff000150a000, bp=0xff800a91c440, done=0x80a02630 dbuf_read_done, private=Variable private is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2763 #19 0x809ff8ec in arc_read (pio=0xff005f66d5a0, spa=0xff000150a000, bp=0xff800a91c440, pbuf=0xff0042a3ca20, done=0x80a02630 dbuf_read_done, private=0xff005fbfc620, priority=0, zio_flags=1, arc_flags=0xff80625db5ec, zb=0xff80625db5c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2508 #20 0x80a02aba in dbuf_read (db=0xff005fbfc620, zio=0xff005f66d5a0, flags=2) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:521 #21 0x80a0602c in dmu_buf_hold (os=Variable os is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:106 #22 0x80a40db5 in zap_lockdir (os=0xff005f937610, obj=247890, tx=0x0, lti=RW_READER, fatreader=1, adding=0, zapp=0xff80625db888) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:388 #23 0x80a41724 in zap_cursor_retrieve (zc=0xff80625db880, za=0xff80625db8c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1004 #24 0x80a61b66 in zfs_freebsd_readdir (ap=Variable ap is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2157 #25 0x803cfde9 in kern_getdirentries
Re: Fatal trap 9 triggered by zfs?
On Dec 4, 2009, at 8:56 PM, Stefan Bethke wrote: Am 04.12.2009 um 17:52 schrieb Stefan Bethke: I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC 2009 r...@xxx.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64 # zpool status pool: tank state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 ad4s1dONLINE 0 0 0 # cat /boot/loader.conf vfs.zfs.arc_max=512M vfs.zfs.prefetch_disable=1 vfs.zfs.zil_disable=1 Got another, different one. Any tuning suggestions or similar? #6 0x80586c7a in vm_map_entry_splay (addr=Variable addr is not available. ) at /usr/src/sys/vm/vm_map.c:771 #7 0x80587f37 in vm_map_lookup_entry (map=0xff0001e8, address=18446743523979624448, entry=0xff80625db170) at /usr/src/sys/vm/vm_map.c:1021 #8 0x80588aa3 in vm_map_delete (map=0xff0001e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2685 #9 0x80588e61 in vm_map_remove (map=0xff0001e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2774 #10 0x8057db85 in uma_large_free (slab=0xff005fcc7000) at /usr/src/sys/vm/uma_core.c:3021 #11 0x80325987 in free (addr=0xff80018b, mtp=0x80ac61e0) at /usr/src/sys/kern/kern_malloc.c:471 #12 0x80a36d03 in vdev_cache_evict (vc=0xff0001723ce0, ve=0xff003dd52200) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:151 #13 0x80a372ad in vdev_cache_read (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:182 Bad RAM/motherboard? My first thought when I read your first mail (re: identical hardware) was bad hardware, and this seems to point towards that too, no? Have you tried memtest86+? Regards, Thomas___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Fatal trap 9 triggered by zfs?
On Fri, Dec 04, 2009 at 08:56:05PM +0100, Stefan Bethke wrote: Am 04.12.2009 um 17:52 schrieb Stefan Bethke: I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC 2009 r...@xxx.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64 # zpool status pool: tank state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 ad4s1dONLINE 0 0 0 # cat /boot/loader.conf vfs.zfs.arc_max=512M vfs.zfs.prefetch_disable=1 vfs.zfs.zil_disable=1 Got another, different one. Any tuning suggestions or similar? #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0x80337bd9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8033802c in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x805cc2ad in trap_fatal (frame=0x9, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x805cce12 in trap (frame=0xff80625db030) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x805b2943 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0x80586c7a in vm_map_entry_splay (addr=Variable addr is not available. ) at /usr/src/sys/vm/vm_map.c:771 #7 0x80587f37 in vm_map_lookup_entry (map=0xff0001e8, address=18446743523979624448, entry=0xff80625db170) at /usr/src/sys/vm/vm_map.c:1021 #8 0x80588aa3 in vm_map_delete (map=0xff0001e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2685 #9 0x80588e61 in vm_map_remove (map=0xff0001e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2774 #10 0x8057db85 in uma_large_free (slab=0xff005fcc7000) at /usr/src/sys/vm/uma_core.c:3021 #11 0x80325987 in free (addr=0xff80018b, mtp=0x80ac61e0) at /usr/src/sys/kern/kern_malloc.c:471 #12 0x80a36d03 in vdev_cache_evict (vc=0xff0001723ce0, ve=0xff003dd52200) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:151 #13 0x80a372ad in vdev_cache_read (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:182 #14 0x80a4a954 in zio_vdev_io_start (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1814 #15 0x80a4ae87 in zio_execute (zio=0xff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #16 0x80a3a080 in vdev_mirror_io_start (zio=0xff005f811b40) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303 #17 0x80a4ae87 in zio_execute (zio=0xff005f811b40) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #18 0x809ff45a in arc_read_nolock (pio=0xff005f66d5a0, spa=0xff000150a000, bp=0xff800a91c440, done=0x80a02630 dbuf_read_done, private=Variable private is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2763 #19 0x809ff8ec in arc_read (pio=0xff005f66d5a0, spa=0xff000150a000, bp=0xff800a91c440, pbuf=0xff0042a3ca20, done=0x80a02630 dbuf_read_done, private=0xff005fbfc620, priority=0, zio_flags=1, arc_flags=0xff80625db5ec, zb=0xff80625db5c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2508 #20 0x80a02aba in dbuf_read (db=0xff005fbfc620, zio=0xff005f66d5a0, flags=2) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:521 #21 0x80a0602c in dmu_buf_hold (os=Variable os is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:106 #22 0x80a40db5 in zap_lockdir (os=0xff005f937610, obj=247890, tx=0x0, lti=RW_READER, fatreader=1, adding=0, zapp=0xff80625db888) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:388 #23 0x80a41724 in zap_cursor_retrieve (zc=0xff80625db880, za=0xff80625db8c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1004 #24 0x80a61b66 in zfs_freebsd_readdir (ap=Variable ap is not
Re: Fatal trap 9 triggered by zfs?
Am 04.12.2009 um 21:20 schrieb Thomas Backman: Bad RAM/motherboard? My first thought when I read your first mail (re: identical hardware) was bad hardware, and this seems to point towards that too, no? Have you tried memtest86+? No, I haven't yet, since I don't have physical access right now, and the box is in production service. I've shifted a couple of services to the other, identical box to see if that changes anything in the behavior. Right now it seems that heavy CPU load triggers panics, so bad RAM, CPU, chipset, mainboard, or marginal power supply are all possibilities. Stefan -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Fatal trap 9 triggered by zfs?
Am 04.12.2009 um 21:33 schrieb Jeremy Chadwick: You only have one disk in your pool. I'm not sure how long your system stays up before it panics, but could you try doing zpool scrub tank and let that run for a while? The first ~5 minutes may show the time to completion (from zpool status) getting worse and worse, but it should decrease/catch up. If the scrub is able to finish, look for any errors in the resulting R/W/CK fields. Doh, should have though of that myself. Will get started right away. Stefan -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Fatal trap 9 triggered by zfs?
Am 04.12.2009 um 20:56 schrieb Stefan Bethke: Am 04.12.2009 um 17:52 schrieb Stefan Bethke: I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC 2009 r...@xxx.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64 # zpool status pool: tank state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 ad4s1dONLINE 0 0 0 # cat /boot/loader.conf vfs.zfs.arc_max=512M vfs.zfs.prefetch_disable=1 vfs.zfs.zil_disable=1 Third one. Since there's no mention of ZFS in this one, I'll start looking into pontential hardware issues. (kgdb) #0 doadump () at pcpu.h:223 #1 0x80337bd9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0x8033802c in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0x805cc2ad in trap_fatal (frame=0x9, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0x805cce12 in trap (frame=0xff800011ab00) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0x805b2943 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0x803405fc in msleep_spin (ident=0xff00015ce780, mtx=0xff00015ce7b0, wmesg=0x80638ffd -, timo=0) at /usr/src/sys/kern/kern_synch.c:312 #7 0x80373ef7 in taskqueue_thread_loop (arg=Variable arg is not available. ) at /usr/src/sys/kern/subr_taskqueue.c:89 #8 0x8030e7d8 in fork_exit ( callout=0x80373e90 taskqueue_thread_loop, arg=0xff80002e4768, frame=0xff800011ac80) at /usr/src/sys/kern/kern_fork.c:843 #9 0x805b2e1e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:561 Stefan -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
