panic: rtqkill route really not free on freebsd 8.0-release update

2010-06-05 Thread Chao Shin

Hi, all

We add kdb/ddb and extra panic info printing into kernel and catch this  
panic again.


We have instrumented the kernel and found that this panic happens when  
draining == 1,
but seems to be confused with the fact that all access to radix trees are  
protected

by locks.  Can anyone familiar with these code shed us some light on this?

below is url to screenshot in ddb:
http://www.delphij.net/zhao/1.png
http://www.delphij.net/zhao/2.png

--
The Power to Serve
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


Re: atapicam issue

2010-06-05 Thread Alexander Motin
hans.da...@eml.cc wrote:
> Here's the full and uncensored log: http://pastebin.org/308021
> 
> On Fri, 04 Jun 2010 19:28:10 +0300, "Alexander Motin" 
> said:
>> hans.da...@eml.cc wrote:
>>> FreeBSD-stable clandestinely dropped support for my p-ata DVD drive some
>>> two or three weeks ago. The relevant "devcontrol list"-entry is still in
>>> its place, yet atapicam fails to create "/dev/cd0". 
>>>
>>> # camcontrol devlist
>>>at scbus0 target 0 lun 0 (pass0)
>>>
>>> dmesg reports:
>>> 
>>> (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
>>> (probe0:ata0:0:0:0): CAM status: SCSI Status Error
>>> (probe0:ata0:0:0:0): SCSI status: Check Condition
>>> (probe0:ata0:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on,
>>> reset, or bus device reset occurred)
>>> (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
>>> (probe0:ata0:0:0:0): CAM status: SCSI Status Error
>>> (probe0:ata0:0:0:0): SCSI status: Check Condition
>>> (probe0:ata0:0:0:0): SCSI sense: NOT READY asc:3a,1 (Medium not present
>>> - tray closed)

I don't see there not only cdX, but also acdX device. So either it is
not atapicam problem, or you have custom kernel without both acd and cd
drivers.

-- 
Alexander Motin
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


HEADS UP: FreeBSD 7.2 EoL coming soon

2010-06-05 Thread FreeBSD Security Officer
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello Everyone,

On June 30th, FreeBSD 7.2 will reach its End of Life and will no longer be
supported by the FreeBSD Security Team.  Users of this release are strongly
encouraged to upgrade to FreeBSD 7.3 before that date; FreeBSD 7.3 will be
supported until the end of March 2012.  Please note that since FreeBSD 7.1
has been designated for 'Extended' support, it will continue to be supported
until the end of January 2011, i.e., FreeBSD 7.1 will be supported longer
than FreeBSD 7.2.

The End of Life date for FreeBSD 7.2 was originally announced as May 31, but
was delayed by one month in accordance with Security Team policy in order to
allow a 3 month window between the release of FreeBSD 7.3 and the End of Life
of FreeBSD 7.2 to allow time for systems to be upgraded.

The freebsd-update(8) utility can be used to upgrade i386 and amd64 systems
from 7.2-RELEASE (or 7.2-RELEASE-pX for some X) to 7.3-RELEASE using binary
updates (i.e., without compiling from source) as described in the 7.3-RELEASE
announcement; given an adequate internet connection, this process usually takes
15 minutes or less.

The current supported branches and expected EoL dates are:

   +-+
   |  Branch   |  Release   |  Type  |   Release date  |  Estimated EoL  |
   |---+++-+-|
   |RELENG_6   |n/a |n/a |n/a  |November 30, 2010|
   |-|
   |RELENG_6_4 |6.4-RELEASE |Extended|November 18, 2008|November 30, 2010|
   |-|
   |RELENG_7   |n/a |n/a |n/a  |last release + 2y|
   |---+++-+-|
   |RELENG_7_1 |7.1-RELEASE |Extended|January 4, 2009  |January 31, 2011 |
   |---+++-+-|
   |RELENG_7_2 |7.2-RELEASE |Normal  |May 4, 2009  |June 30, 2010|
   |---+++-+-|
   |RELENG_7_3 |7.3-RELEASE |Extended|March 23, 2010   |March 31, 2012   |
   |---+++-+-|
   |RELENG_8   |n/a |n/a |n/a  |last release + 2y|
   |---+++-+-|
   |RELENG_8_0 |8.0-RELEASE |Normal  |November 25, 2009|November 30, 2010|
   |---+++-+-|
   |RELENG_8_1 |8.1-RELEASE |Extended|not yet  |release + 2 years|
   +-+

- --
Colin Percival
Security Officer, FreeBSD | freebsd.org | The power to serve
Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (FreeBSD)

iEYEARECAAYFAkwKZ8QACgkQFdaIBMps37LL9wCfRspIGXYatsdPDbBR8OZEDocs
BagAnAmTXen6TQ+2ER3oF6702stmxVIJ
=ydCN
-END PGP SIGNATURE-
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


Re: atapicam issue --- solved!

2010-06-05 Thread hans . dampf
Thank you, Alexander.
For some obscure reason, my kernel configuration lacked "device cd". 


Hans



On Sat, 05 Jun 2010 15:06:22 +0300, "Alexander Motin" 
said:
> hans.da...@eml.cc wrote:
> > Here's the full and uncensored log: http://pastebin.org/308021
> > 
> > On Fri, 04 Jun 2010 19:28:10 +0300, "Alexander Motin" 
> > said:
> >> hans.da...@eml.cc wrote:
> >>> FreeBSD-stable clandestinely dropped support for my p-ata DVD drive some
> >>> two or three weeks ago. The relevant "devcontrol list"-entry is still in
> >>> its place, yet atapicam fails to create "/dev/cd0". 
> >>>
> >>> # camcontrol devlist
> >>>at scbus0 target 0 lun 0 (pass0)
> >>>
> >>> dmesg reports:
> >>> 
> >>> (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
> >>> (probe0:ata0:0:0:0): CAM status: SCSI Status Error
> >>> (probe0:ata0:0:0:0): SCSI status: Check Condition
> >>> (probe0:ata0:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on,
> >>> reset, or bus device reset occurred)
> >>> (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
> >>> (probe0:ata0:0:0:0): CAM status: SCSI Status Error
> >>> (probe0:ata0:0:0:0): SCSI status: Check Condition
> >>> (probe0:ata0:0:0:0): SCSI sense: NOT READY asc:3a,1 (Medium not present
> >>> - tray closed)
> 
> I don't see there not only cdX, but also acdX device. So either it is
> not atapicam problem, or you have custom kernel without both acd and cd
> drivers.
> 
> -- 
> Alexander Motin
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


Re: arp -na performance w/ many permanent entries

2010-06-05 Thread Nick Rogers
On Mon, May 31, 2010 at 10:54 PM, Nick Rogers  wrote:

>
> [root@ ~]# time arp -na > /dev/null
>
> real 0m12.761s
> user 0m2.959s
> sys 0m9.753s
> [root@ ~]#
>
>
> Notice that "arp -na" takes about 13s to execute even though there is no
> other load. This can get a lot worse by a few orders of magnitude on a
> loaded machine in a production environment, and seems to scale up linearly
> when more aliases are added to the interface (permanent ARP entries
> created).
>
> Is this a reasonable problem that can be fixed/improved, or am I stuck with
> the slow arp -na output? Any help or comments is greatly appreciated.
>

I tried the same scenario on 8.1-BETA1 and it still takes a very long time
for arp(8) to complete.

I was able to isolate the performance bottleneck to a small piece of the
arp(8) code. It seems that looking up the interface for an ARP entry is a
very heavy operation when that entry corresponds to an alias assigned to the
interface. Permanent ARP entries that do not correspond with an interface
alias do not seem to cause arp(8) to puke on the interface lookup.

The following commands and code diff illustrates how arp(8) can be modified
to run a lot faster in this scenario, but obviously the associated interface
is no longer printed for each entry.

[root@ /usr/src/usr.sbin/arp]# uname -a
FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30 UTC
2010 r...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
[root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l
4100

real 0m14.903s
user 0m3.133s
sys 0m11.519s
[root@ /usr/src/usr.sbin/arp]# pwd
/usr/src/usr.sbin/arp
[root@ /usr/src/usr.sbin/arp]# !diff
diff -ruN arp.c.orig arp.c
--- arp.c.orig 2010-06-05 18:25:24.0 +
+++ arp.c 2010-06-05 18:28:19.0 +
@@ -562,7 +562,7 @@
  const char *host;
  struct hostent *hp;
  struct iso88025_sockaddr_dl_data *trld;
- char ifname[IF_NAMESIZE];
+ //char ifname[IF_NAMESIZE];
  int seg;

  if (nflag == 0)
@@ -591,8 +591,8 @@
  }
  } else
  printf("(incomplete)");
- if (if_indextoname(sdl->sdl_index, ifname) != NULL)
- printf(" on %s", ifname);
+ //if (if_indextoname(sdl->sdl_index, ifname) != NULL)
+ //printf(" on %s", ifname);
  if (rtm->rtm_rmx.rmx_expire == 0)
  printf(" permanent");
  else {
[root@ /usr/src/usr.sbin/arp]# make clean && make
rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz
Warning: Object directory not changed from original /usr/src/usr.sbin/arp
cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
-Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
-Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -c
arp.c
cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
-Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
-Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign
 -o arp arp.o
gzip -cn arp.4 > arp.4.gz
gzip -cn arp.8 > arp.8.gz
[root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l
4099

real 0m0.036s
user 0m0.015s
sys 0m0.021s
[root@ /usr/src/usr.sbin/arp]#

Notice that 0.036s without the interface lookup is a heck of a lot faster
than 14.903s when doing the interface lookup.

Is there something that can be done to speedup the call to if_indextoname(),
or would it be worthwhile for me to submit a patch that adds the ability to
skip the interface lookup as an arp(8) option?
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


Re: arp -na performance w/ many permanent entries

2010-06-05 Thread Jeremy Chadwick
On Sat, Jun 05, 2010 at 09:48:01PM -0400, Nick Rogers wrote:
> On Mon, May 31, 2010 at 10:54 PM, Nick Rogers  wrote:
> 
> >
> > [root@ ~]# time arp -na > /dev/null
> >
> > real 0m12.761s
> > user 0m2.959s
> > sys 0m9.753s
> > [root@ ~]#
> >
> >
> > Notice that "arp -na" takes about 13s to execute even though there is no
> > other load. This can get a lot worse by a few orders of magnitude on a
> > loaded machine in a production environment, and seems to scale up linearly
> > when more aliases are added to the interface (permanent ARP entries
> > created).
> >
> > Is this a reasonable problem that can be fixed/improved, or am I stuck with
> > the slow arp -na output? Any help or comments is greatly appreciated.
> >
> 
> I tried the same scenario on 8.1-BETA1 and it still takes a very long time
> for arp(8) to complete.
> 
> I was able to isolate the performance bottleneck to a small piece of the
> arp(8) code. It seems that looking up the interface for an ARP entry is a
> very heavy operation when that entry corresponds to an alias assigned to the
> interface. Permanent ARP entries that do not correspond with an interface
> alias do not seem to cause arp(8) to puke on the interface lookup.
> 
> The following commands and code diff illustrates how arp(8) can be modified
> to run a lot faster in this scenario, but obviously the associated interface
> is no longer printed for each entry.
> 
> [root@ /usr/src/usr.sbin/arp]# uname -a
> FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30 UTC
> 2010 r...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
> [root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l
> 4100
> 
> real 0m14.903s
> user 0m3.133s
> sys 0m11.519s
> [root@ /usr/src/usr.sbin/arp]# pwd
> /usr/src/usr.sbin/arp
> [root@ /usr/src/usr.sbin/arp]# !diff
> diff -ruN arp.c.orig arp.c
> --- arp.c.orig 2010-06-05 18:25:24.0 +
> +++ arp.c 2010-06-05 18:28:19.0 +
> @@ -562,7 +562,7 @@
>   const char *host;
>   struct hostent *hp;
>   struct iso88025_sockaddr_dl_data *trld;
> - char ifname[IF_NAMESIZE];
> + //char ifname[IF_NAMESIZE];
>   int seg;
> 
>   if (nflag == 0)
> @@ -591,8 +591,8 @@
>   }
>   } else
>   printf("(incomplete)");
> - if (if_indextoname(sdl->sdl_index, ifname) != NULL)
> - printf(" on %s", ifname);
> + //if (if_indextoname(sdl->sdl_index, ifname) != NULL)
> + //printf(" on %s", ifname);
>   if (rtm->rtm_rmx.rmx_expire == 0)
>   printf(" permanent");
>   else {
> [root@ /usr/src/usr.sbin/arp]# make clean && make
> rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz
> Warning: Object directory not changed from original /usr/src/usr.sbin/arp
> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -c
> arp.c
> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign
>  -o arp arp.o
> gzip -cn arp.4 > arp.4.gz
> gzip -cn arp.8 > arp.8.gz
> [root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l
> 4099
> 
> real 0m0.036s
> user 0m0.015s
> sys 0m0.021s
> [root@ /usr/src/usr.sbin/arp]#
> 
> Notice that 0.036s without the interface lookup is a heck of a lot faster
> than 14.903s when doing the interface lookup.
> 
> Is there something that can be done to speedup the call to if_indextoname(),
> or would it be worthwhile for me to submit a patch that adds the ability to
> skip the interface lookup as an arp(8) option?

This might be a better question for either freebsd-net or
freebsd-hackers.  I should warn you in advance that you might receive a
bit of flack given that you have over 4000 IP aliases assigned to an
interface.  Explaining your setup may also help people understand why it
is you need what you do.

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


Re: arp -na performance w/ many permanent entries

2010-06-05 Thread Garrett Cooper
On Sat, Jun 5, 2010 at 8:16 PM, Jeremy Chadwick
 wrote:
> On Sat, Jun 05, 2010 at 09:48:01PM -0400, Nick Rogers wrote:
>> On Mon, May 31, 2010 at 10:54 PM, Nick Rogers  wrote:
>>
>> >
>> > [root@ ~]# time arp -na > /dev/null
>> >
>> > real 0m12.761s
>> > user 0m2.959s
>> > sys 0m9.753s
>> > [root@ ~]#
>> >
>> >
>> > Notice that "arp -na" takes about 13s to execute even though there is no
>> > other load. This can get a lot worse by a few orders of magnitude on a
>> > loaded machine in a production environment, and seems to scale up linearly
>> > when more aliases are added to the interface (permanent ARP entries
>> > created).
>> >
>> > Is this a reasonable problem that can be fixed/improved, or am I stuck with
>> > the slow arp -na output? Any help or comments is greatly appreciated.
>> >
>>
>> I tried the same scenario on 8.1-BETA1 and it still takes a very long time
>> for arp(8) to complete.
>>
>> I was able to isolate the performance bottleneck to a small piece of the
>> arp(8) code. It seems that looking up the interface for an ARP entry is a
>> very heavy operation when that entry corresponds to an alias assigned to the
>> interface. Permanent ARP entries that do not correspond with an interface
>> alias do not seem to cause arp(8) to puke on the interface lookup.
>>
>> The following commands and code diff illustrates how arp(8) can be modified
>> to run a lot faster in this scenario, but obviously the associated interface
>> is no longer printed for each entry.
>>
>> [root@ /usr/src/usr.sbin/arp]# uname -a
>> FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30 UTC
>> 2010     r...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>> [root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l
>>     4100
>>
>> real 0m14.903s
>> user 0m3.133s
>> sys 0m11.519s
>> [root@ /usr/src/usr.sbin/arp]# pwd
>> /usr/src/usr.sbin/arp
>> [root@ /usr/src/usr.sbin/arp]# !diff
>> diff -ruN arp.c.orig arp.c
>> --- arp.c.orig 2010-06-05 18:25:24.0 +
>> +++ arp.c 2010-06-05 18:28:19.0 +
>> @@ -562,7 +562,7 @@
>>   const char *host;
>>   struct hostent *hp;
>>   struct iso88025_sockaddr_dl_data *trld;
>> - char ifname[IF_NAMESIZE];
>> + //char ifname[IF_NAMESIZE];
>>   int seg;
>>
>>   if (nflag == 0)
>> @@ -591,8 +591,8 @@
>>   }
>>   } else
>>   printf("(incomplete)");
>> - if (if_indextoname(sdl->sdl_index, ifname) != NULL)
>> - printf(" on %s", ifname);
>> + //if (if_indextoname(sdl->sdl_index, ifname) != NULL)
>> + //printf(" on %s", ifname);
>>   if (rtm->rtm_rmx.rmx_expire == 0)
>>   printf(" permanent");
>>   else {
>> [root@ /usr/src/usr.sbin/arp]# make clean && make
>> rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz
>> Warning: Object directory not changed from original /usr/src/usr.sbin/arp
>> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
>> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -c
>> arp.c
>> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
>> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign
>>  -o arp arp.o
>> gzip -cn arp.4 > arp.4.gz
>> gzip -cn arp.8 > arp.8.gz
>> [root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l
>>     4099
>>
>> real 0m0.036s
>> user 0m0.015s
>> sys 0m0.021s
>> [root@ /usr/src/usr.sbin/arp]#
>>
>> Notice that 0.036s without the interface lookup is a heck of a lot faster
>> than 14.903s when doing the interface lookup.
>>
>> Is there something that can be done to speedup the call to if_indextoname(),
>> or would it be worthwhile for me to submit a patch that adds the ability to
>> skip the interface lookup as an arp(8) option?
>
> This might be a better question for either freebsd-net or
> freebsd-hackers.  I should warn you in advance that you might receive a
> bit of flack given that you have over 4000 IP aliases assigned to an
> interface.  Explaining your setup may also help people understand why it
> is you need what you do.

I agree with Jeremy. I think that the problem that you've
discovered is the fact that it's using stdio-based buffered output
instead of buffering more of the contents in a string and punting it
out in larger chunks.
HTH,
-Garrett
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


gmirror refused to connect second disk after a reboot

2010-06-05 Thread Edwin Groothuis
For two years I've had a happy gmirror RAID1 system. And a week or
three ago I was found a degraded system due to a broken disk.

I tried to replace the disk, first with one three sectors too small
which didn't want to be entered in the array (as excepted), then
with a same brand/type one which I added without a problem. Rebuilding,
everything okay.

[~] ed...@k7>sudo fdisk -s /dev/ad1
/dev/ad1: 1938021 cyl 16 hd 63 sec
PartStartSize Type Flags
   1:  63  1953520002 0xa5 0x00
[~] ed...@k7>sudo fdisk -s /dev/ad3
/dev/ad3: 1938021 cyl 16 hd 63 sec
PartStartSize Type Flags
   1:  63  1953520002 0xa5 0x80

[~] ed...@k7>gmirror status
  NameStatus  Components
mirror/gm0  COMPLETE  ad1
  ad3


Until after a reboot, then GEOM complains about:

GEOM: ad3s1: geometry does not match label (255h,63s != 16h,63s).
GEOM_MIRROR: Force device gm0 start due to timeout.
GEOM_MIRROR: Device mirror/gm0 launched (1/2).

[~] ed...@k7>gmirror status
  NameStatus  Components
  mirror/gm0  DEGRADED  ad1

Forgetting and re-inserting the ad3 does attach it again and rebuild
everything, until the next reboot.

Booting from ad3 instead of from ad1 results in the same result.
This device is running 8.0-STABLE r204385.


Who has any idea what is going wrong here and which smart commands
I can to overcome this silly issues or are any clues which commands
I should run next to get more information?

Thanks heaps,
Edwin

-- 
Edwin Groothuis Website: http://www.mavetju.org/
ed...@mavetju.org   Weblog:  http://www.mavetju.org/weblog/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"