Re: panic: kmem_map too small at heavy packet traffic
Specifically, I am taking this panic when doing ip spoof attack while syn-proxy activated. The output of system arguments below: kern.malloc_count: 315 vm.md_malloc_wait: 0 vfs.bufmallocspace: 0 vfs.maxmallocbufspace: 86269952 vm.kmem_size: 16686845952 vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size_scale: 1 vm.kmem_map_size: 543973376 vm.kmem_map_free: 15974895616 kern.maxvnodes: 350097 kern.minvnodes: 87524 vfs.numvnodes: 112329 vfs.wantfreevnodes: 87524 vfs.freevnodes: 87502 [root@ ~]# pfctl -si No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 0 days 00:17:39 Debug: Urgent State Table Total Rate current entries 5142886 searches2698214125478.9/s inserts 2905505327436.3/s removals2421865422869.4/s Counters match 2490130523514.0/s bad-offset 00.0/s fragment 00.0/s short 00.0/s normalize 00.0/s memory 00.0/s bad-timestamp 00.0/s congestion 00.0/s ip-option 180.0/s proto-cksum00.0/s state-mismatch 00.0/s state-insert 00.0/s state-limit00.0/s src-limit 00.0/s synproxy2937843927741.7/s [root@~]# panic: kmem_malloc(-1 814 425 600): kmem_map too small: 543 956 992 to tal allocated cpuid = 8 Uptime: 1d18h2m14s (ada0:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada0:ahcich1:0:0:0): CAM status: CCB request is in progress (ada0:ahcich1:0:0:0): Error 5, Retries exhausted (ada0:ahcich1:0:0:0): Synchronize cache failed (ada1:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada1:ahcich2:0:0:0): CAM status: CCB request is in progress (ada1:ahcich2:0:0:0): Error 5, Retries exhausted (ada1:ahcich2:0:0:0): Synchronize cache failed Dumping 8243 out of 16357 MB:..1%..11% On Thu, Jul 25, 2013 at 5:57 PM, Tugrul Erdogan h.tugrul.erdo...@gmail.comwrote: howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: panic: kmem_map too small at heavy packet traffic
I've been under the impression that synproxy was broken for quite some time, but I know there has been a lot of work on pf in HEAD so I can't be sure where it might stand there. Can anyone confirm/deny this? And not to discourage you, but the pf documentation does say Routine use of this option is not recommended, however, as it breaks expected TCP protocol behavior when the server can't process the request... However, panics are never good and hopefully someone can help you figure it out. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UFS related panic (daily - find)
I had 2 panics: (Both occured at 3 AM, so had to be daily task)
First (Jul 2 03:06:50 2013):
--
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x19
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc06caf34
stack pointer = 0x28:0xe76248fc
frame pointer = 0x28:0xe7624930
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 76562 (find)
trap number = 12
panic: page fault
Uptime: 23h0m41s
Physical memory: 1014 MB
Dumping 186 MB: 171 155 139 123 107 91 75 59 43 27 11
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990, vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at
/usr/src/sys/kern/vfs_cache.c:547
Can you go up to this frame and do 'l'?
--
John Baldwin
Sure,
-
(kgdb) up 7
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990, vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:547
547 numchecks++;
-
(kgdb) l
542 }
543
544 hash = fnv_32_buf(cnp-cn_nameptr, cnp-cn_namelen,
FNV1_32_INIT);
545 hash = fnv_32_buf(dvp, sizeof(dvp), hash);
546 LIST_FOREACH(ncp, (NCHHASH(hash)), nc_hash) {
547 numchecks++;
548 if (ncp-nc_dvp == dvp ncp-nc_nlen ==
cnp-cn_namelen
549 !bcmp(nc_get_name(ncp), cnp-cn_nameptr,
ncp-nc_nlen))
550 break;
551 }
-
Domagoj Smolčić
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Android applications for fsyscall/nexec
I report the current status of fsyscall/nexec. fsyscall[1] is a system to transfer system call requests over network. The goal of it is using FreeBSD applications at another machine (including non-FreeBSD machines) without any modifications of applications. nexec[2] is a system to make connection between a master and a slave of fsyscall. I developed Java version of fsyscall slave. This means that any Java platforms can use FreeBSD applications over network. Actually, I implemented some Android applications using fsyscall/nexec. One is nexec client demo for Android[3]. This is a simple application to try fsyscall/nexec. Another one is animator[4]. This is an application to make stop motion movie. It uses ffmpeg on FreeBSD to generate a movie. Finally, these two applications use one service, nexec client for Android[5]. This is a core service for fsyscall/nexec. It also manages file access for security. My one machine is ready for nexec server now. The above applications use this server (neko-daisuki.ddo.jp) by default. Anyone can try fsyscall/nexec. [1] http://neko-daisuki.ddo.jp/~SumiTomohiko/fsyscall/index.html [2] http://neko-daisuki.ddo.jp/~SumiTomohiko/nexec/index.html [3] http://neko-daisuki.ddo.jp/~SumiTomohiko/android-nexec-client-demo/index.html [4] http://neko-daisuki.ddo.jp/~SumiTomohiko/animator/index.html [5] http://neko-daisuki.ddo.jp/~SumiTomohiko/android-nexec-client/index.html -- Tomohiko Sumi sumitomoh...@neko-daisuki.ddo.jp http://neko-daisuki.ddo.jp/~SumiTomohiko/index.html ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UFS related panic (daily - find)
On Friday, July 26, 2013 9:04:42 am rank1see...@gmail.com wrote:
I had 2 panics: (Both occured at 3 AM, so had to be daily task)
First (Jul 2 03:06:50 2013):
--
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x19
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc06caf34
stack pointer = 0x28:0xe76248fc
frame pointer = 0x28:0xe7624930
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 76562 (find)
trap number = 12
panic: page fault
Uptime: 23h0m41s
Physical memory: 1014 MB
Dumping 186 MB: 171 155 139 123 107 91 75 59 43 27 11
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990, vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at
/usr/src/sys/kern/vfs_cache.c:547
Can you go up to this frame and do 'l'?
--
John Baldwin
Sure,
-
(kgdb) up 7
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990, vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:547
547 numchecks++;
-
(kgdb) l
542 }
543
544 hash = fnv_32_buf(cnp-cn_nameptr, cnp-cn_namelen,
FNV1_32_INIT);
545 hash = fnv_32_buf(dvp, sizeof(dvp), hash);
546 LIST_FOREACH(ncp, (NCHHASH(hash)), nc_hash) {
547 numchecks++;
548 if (ncp-nc_dvp == dvp ncp-nc_nlen ==
cnp-cn_namelen
549 !bcmp(nc_get_name(ncp), cnp-cn_nameptr,
ncp-nc_nlen))
550 break;
551 }
-
Hmm, 'p ncp' and 'p *ncp' at that frame perhaps?
--
John Baldwin
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: UFS related panic (daily - find)
I had 2 panics: (Both occured at 3 AM, so had to be daily task)
First (Jul 2 03:06:50 2013):
--
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x19
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc06caf34
stack pointer = 0x28:0xe76248fc
frame pointer = 0x28:0xe7624930
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 76562 (find)
trap number = 12
panic: page fault
Uptime: 23h0m41s
Physical memory: 1014 MB
Dumping 186 MB: 171 155 139 123 107 91 75 59 43 27 11
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990,
vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at
/usr/src/sys/kern/vfs_cache.c:547
Can you go up to this frame and do 'l'?
--
John Baldwin
Sure,
-
(kgdb) up 7
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990, vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:547
547 numchecks++;
-
(kgdb) l
542 }
543
544 hash = fnv_32_buf(cnp-cn_nameptr, cnp-cn_namelen,
FNV1_32_INIT);
545 hash = fnv_32_buf(dvp, sizeof(dvp), hash);
546 LIST_FOREACH(ncp, (NCHHASH(hash)), nc_hash) {
547 numchecks++;
548 if (ncp-nc_dvp == dvp ncp-nc_nlen ==
cnp-cn_namelen
549 !bcmp(nc_get_name(ncp), cnp-cn_nameptr,
ncp-nc_nlen))
550 break;
551 }
-
Hmm, 'p ncp' and 'p *ncp' at that frame perhaps?
(kgdb) p ncp
$1 = (struct namecache *) 0x1
(kgdb) p *ncp
Cannot access memory at address 0x1
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
CFR: improvments to cfi(4) flash driver
I've made a number of improvements to the write performance of the CFI
flash driver. The patch below merges the remaining changes. I've only
implemented buffered write on Intel flash as that's all I currently have
access to. I'm looking for review and/or testing of these changes.
http://people.freebsd.org/~brooks/patches/cfi.diff
-- Brooks
MFP4 217312, 222008, 222052, 222053, 222673, 231484, 231491
Rework the timeout code to use actual time rather than a DELAY() loop and
to use both typical and maximum to allow logging of timeout failures.
Also correct the erase timeout, it is specified in milliseconds not
microseconds like the other timeouts. Do not invoke DELAY() between
status queries as this adds significant latency which in turn reduced
write performance substantially.
Implement support for buffered writes (only enabled on Intel/Sharp parts
for now). This yields an order of magnitude speedup on the 64MB Intel
StrataFlash parts we use.
When making a copy of the block to modify, also keep a clean copy around
until we are ready to commit the block and use it to avoid unnecessary
erases. In the non-buffer write case, also use it to avoid
unnecessary writes when the block has not been erased. This yields a
significant speedup when doing things like zeroing a block.
Sponsored by: DARPA, AFRL
diff -ru /home/bed22/p4/freebsd/src/sys/dev/cfi/cfi_bus_nexus.c
./cfi_bus_nexus.c
--- /home/bed22/p4/freebsd/src/sys/dev/cfi/cfi_bus_nexus.c 2013-04-29
20:40:02.0 +0100
+++ ./cfi_bus_nexus.c 2013-07-26 20:36:34.0 +0100
@@ -4,6 +4,11 @@
* Copyright (c) 2009 Sam Leffler, Errno Consulting
* All rights reserved.
*
+ * Portions of this software were developed by SRI International and the
+ * University of Cambridge Computer Laboratory under DARPA/AFRL contract
+ * (FA8750-10-C-0237) (CTSRD), as part of the DARPA CRASH research
+ * programme.
+ *
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
diff -ru /home/bed22/p4/freebsd/src/sys/dev/cfi/cfi_core.c ./cfi_core.c
--- /home/bed22/p4/freebsd/src/sys/dev/cfi/cfi_core.c 2013-06-03
15:41:18.0 +0100
+++ ./cfi_core.c2013-07-26 20:40:32.0 +0100
@@ -1,7 +1,13 @@
/*-
* Copyright (c) 2007, Juniper Networks, Inc.
+ * Copyright (c) 2012-2013, SRI International
* All rights reserved.
*
+ * Portions of this software were developed by SRI International and the
+ * University of Cambridge Computer Laboratory under DARPA/AFRL contract
+ * (FA8750-10-C-0237) (CTSRD), as part of the DARPA CRASH research
+ * programme.
+ *
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
@@ -279,11 +285,31 @@
sc-sc_tag = rman_get_bustag(sc-sc_res);
sc-sc_handle = rman_get_bushandle(sc-sc_res);
- /* Get time-out values for erase and write. */
- sc-sc_write_timeout = 1 cfi_read_qry(sc, CFI_QRY_TTO_WRITE);
- sc-sc_erase_timeout = 1 cfi_read_qry(sc, CFI_QRY_TTO_ERASE);
- sc-sc_write_timeout *= 1 cfi_read_qry(sc, CFI_QRY_MTO_WRITE);
- sc-sc_erase_timeout *= 1 cfi_read_qry(sc, CFI_QRY_MTO_ERASE);
+ /* Get time-out values for erase, write, and buffer write. */
+ sc-sc_typical_timeouts[CFI_TIMEOUT_ERASE] =
+ SBT_1MS * (1 cfi_read_qry(sc, CFI_QRY_TTO_ERASE));
+ sc-sc_max_timeouts[CFI_TIMEOUT_ERASE] =
+ sc-sc_typical_timeouts[CFI_TIMEOUT_ERASE] *
+ (1 cfi_read_qry(sc, CFI_QRY_MTO_ERASE));
+
+ sc-sc_typical_timeouts[CFI_TIMEOUT_WRITE] =
+ SBT_1US * (1 cfi_read_qry(sc, CFI_QRY_TTO_WRITE));
+ sc-sc_max_timeouts[CFI_TIMEOUT_WRITE] =
+ sc-sc_typical_timeouts[CFI_TIMEOUT_WRITE] *
+ (1 cfi_read_qry(sc, CFI_QRY_MTO_WRITE));
+
+ sc-sc_typical_timeouts[CFI_TIMEOUT_BUFWRITE] =
+ SBT_1US * (1 cfi_read_qry(sc, CFI_QRY_TTO_BUFWRITE));
+ sc-sc_max_timeouts[CFI_TIMEOUT_BUFWRITE] =
+ sc-sc_typical_timeouts[CFI_TIMEOUT_BUFWRITE] *
+ (1 cfi_read_qry(sc, CFI_QRY_MTO_BUFWRITE));
+
+ /* Get the maximum size of a multibyte program */
+ if (sc-sc_typical_timeouts[CFI_TIMEOUT_BUFWRITE] != 0)
+ sc-sc_maxbuf = 1 (cfi_read_qry(sc, CFI_QRY_MAXBUF) |
+ cfi_read_qry(sc, CFI_QRY_MAXBUF) 8);
+ else
+ sc-sc_maxbuf = 0;
/* Get erase regions. */
sc-sc_regions = cfi_read_qry(sc, CFI_QRY_NREGIONS);
@@ -351,18 +377,20 @@
}
static int
-cfi_wait_ready(struct cfi_softc *sc, u_int ofs, u_int timeout)
+cfi_wait_ready(struct cfi_softc *sc, u_int ofs, sbintime_t start,
+enum cfi_wait_cmd cmd)
{
- int done, error;
+ int done, error, tto_exceeded;
uint32_t st0 = 0, st = 0;
+ sbintime_t mend, now, tend;
+
+ tend = start + sc-sc_typical_timeouts[cmd];
+ mend =
Re: UFS related panic (daily - find)
On Friday, July 26, 2013 3:00:33 pm rank1see...@gmail.com wrote:
I had 2 panics: (Both occured at 3 AM, so had to be daily task)
First (Jul 2 03:06:50 2013):
--
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x19
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc06caf34
stack pointer = 0x28:0xe76248fc
frame pointer = 0x28:0xe7624930
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 76562 (find)
trap number = 12
panic: page fault
Uptime: 23h0m41s
Physical memory: 1014 MB
Dumping 186 MB: 171 155 139 123 107 91 75 59 43 27 11
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990,
vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at
/usr/src/sys/kern/vfs_cache.c:547
Can you go up to this frame and do 'l'?
--
John Baldwin
Sure,
-
(kgdb) up 7
#7 0xc06caf34 in cache_lookup_times (dvp=0xc784a990, vpp=0xe7624ae8,
cnp=0xe7624afc, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:547
547 numchecks++;
-
(kgdb) l
542 }
543
544 hash = fnv_32_buf(cnp-cn_nameptr, cnp-cn_namelen,
FNV1_32_INIT);
545 hash = fnv_32_buf(dvp, sizeof(dvp), hash);
546 LIST_FOREACH(ncp, (NCHHASH(hash)), nc_hash) {
547 numchecks++;
548 if (ncp-nc_dvp == dvp ncp-nc_nlen ==
cnp-cn_namelen
549 !bcmp(nc_get_name(ncp), cnp-cn_nameptr,
ncp-nc_nlen))
550 break;
551 }
-
Hmm, 'p ncp' and 'p *ncp' at that frame perhaps?
(kgdb) p ncp
$1 = (struct namecache *) 0x1
(kgdb) p *ncp
Cannot access memory at address 0x1
Interesting. Maybe look at NCHHASH(hash) (you'll have to expand the macro
manually)
and see if the head node is corrupted or walk the list to find the corrupted
node.
Given that it is a single bit error, there is a chance this is a RAM problem.
If it
is in the hash table head entry then that would always be at the same physical
address
for the same kernel I think.
--
John Baldwin
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: panic: kmem_map too small at heavy packet traffic
Hi Have you filed a PR? This should get fixed. Also, being -ve is a problem. Is the value really negative? Is it wrapping badly? -adrian On 25 July 2013 07:57, Tugrul Erdogan h.tugrul.erdo...@gmail.com wrote: howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
