RE: High syscall overhead?
> -Original Message- > From: John S. Dyson [SMTP:dy...@iquest.net] > Sent: Saturday, June 12, 1999 6:58 PM > To: w...@softweyr.com > Cc: cro...@cs.rpi.edu; freebsd-hackers@FreeBSD.ORG > Subject: Re: High syscall overhead? > > Think of it like this: since alot of desktops sit in idle loops much > of the time, perhaps the Linux philosophy has been to improve such > behavior :-). > [ML] Do you remember the old anecdote about a profiled ATT unix kernel where they have found out that the kernel spends a lot of time in one loop and rewritten it in assembly--it turned out it was the IDLE loop :) /Marino To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Not to pick on Brian, but can we end this pathetic and sorry thread already? Everybody: Just play nice with your little brother and stop poking him in the back seat - we'll get to where we're going soon and then everybody can have a milkshake if they've been good. Thank you. - Jordan To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
On Sun, 13 Jun 1999 sth...@nethelp.no wrote: > > Linux is a Unix clone, while FreeBSD is Unix. Don't confuse people with > > this. > > I'm afraid that attitude isn't going to help Unix agains Windows... > > I use FreeBSD for all my systems. I still go around and tell people that > Linux is one several Unix variants, and I intend to continue doing this. > For an end user, the differences between for instance FreeBSD and Linux > are in most cases small and rather uninteresting - and it's much more > useful to point out the differences between Unix and Windows. > And the differences between a BSD system and Linux aren't huge? Nay, sir, surely you jest. I, personally, dislike people using the wrong terminology. Linux is a relatively Unix-like and relatively POSIX-compliant system, but it's not Unix. What's wrong with calling a system by its true name? > Steinar Haug, Nethelp consulting, sth...@nethelp.no > > Brian Feldman_ __ ___ ___ ___ ___ gr...@unixhelp.org_ __ ___ | _ ) __| \ FreeBSD: The Power to Serve! _ __ | _ \._ \ |) | http://www.freebsd.org _ |___/___/___/ " THAT'S WRONG WRONG WRONG!" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> Linux is a Unix clone, while FreeBSD is Unix. Don't confuse people with > this. I'm afraid that attitude isn't going to help Unix agains Windows... I use FreeBSD for all my systems. I still go around and tell people that Linux is one several Unix variants, and I intend to continue doing this. For an end user, the differences between for instance FreeBSD and Linux are in most cases small and rather uninteresting - and it's much more useful to point out the differences between Unix and Windows. Steinar Haug, Nethelp consulting, sth...@nethelp.no To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
On Sun, 13 Jun 1999, Chuck Robey wrote: > On Sat, 12 Jun 1999, Bill Huey wrote: > > > Well, so far I've heard alot of BS about Linux that isn't exactly true > > and much of it seems like a bunch of artificial problems that hold > > against the Linux folks. Most of it is just intentionally misrepresented > > bullshit. > > That is totally wrong, Bill. We have a public mail archive, so if you > don't like to believe me, go check our archive. YOU have been the only > source of either Linux or FreeBSD bashing. We have a pretty pleasing > assortment of grownups here, who have gone past the point of needing to > claim "my ball is better than your ball". Exactly. Do you have references? > > One of my own personal complaints about the Linux groups is that they > seem to grow the kind of people who need to disparage in order to feel > good. This might be because my only exposure to Linux persons is via > Usenet, which is immersed in that teenage-type philosophy. I don't know > of any Linux mailing lists of the order of the FreeBSD lists. The Linux-kernel mailing list seems to be closer to -hackers. > > We don't need that approach here, it doesn't increase knowledge, > and only hurts feelings. If you follow up on this thread, please DROP > all Linux/FreeBSD comparisons of that order. We all know that both > Linux AND FreeBSD are fine unixes, and we don't need bashing here. Linux is a Unix clone, while FreeBSD is Unix. Don't confuse people with this. > > This list does occaisonally have some comparisons, Linux vs. FreeBSD, > but if it isn't kept strictly technical, and non-accusatory, we don't > want it or need it. > > I have seen people in the past that were honestly surprised at the huge > difference between Usenet newsgroup rules, and our FreeBSD list rules. > If you actually fit in that category, maybe you should "lurk" these > lists for a while, so you can see the difference, and not find yourself > outside of our lists normal behaviour. The noise to content ratio is MUCH higher on Usenet, and probably will always be. > > > > > I came on this list initially to just check was the FreeBSD community > > was like, but what I've gotten since is alot of ego and hostility > > toward things that aren't completely FreeBSD. That's something that > > I didn't expect and reading this list has given me a particular > > negative view of FreeBSD that wasn't present before. > > People don't like your need to compare, in order to make one group seem > better or worse than another. You need to find some other way to get > your point across. > > > > > In-fighting with the current NFS maintainer and general rudeness to other > > potential devs make FreeBSD's kernel people look like bunch of dorks whether > > you like it or not. Don't confuse fighting with "not sweeping issues under the carpet." > > > > I'm also not big enough asshole ot put someone on a "kill-list" and is just > > a reflection of a kind of conservative need to dehumanize other folks > > so that your selfish comfort is "preserved". > > Profanity is also one of the things not accepted here. Damn straight! Sorry, couldn't resist. Profanity isn't that unacceptable, if appropriate (sometimes it is..), but personal attacks ARE. > > > > > > Wes Peters Softweyr LLC > > > > bill > > > > > > > > To Unsubscribe: send mail to majord...@freebsd.org > > with "unsubscribe freebsd-hackers" in the body of the message > > > > +--- > Chuck Robey | Interests include any kind of voice or data > chu...@picnic.mat.net | communications topic, C programming, and Unix. > 213 Lakeside Drive Apt T-1 | > Greenbelt, MD 20770 | I run picnic and jaunt, both FreeBSD-current. > (301) 220-2114 | > +--- > > > > > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > Brian Feldman_ __ ___ ___ ___ ___ gr...@unixhelp.org_ __ ___ | _ ) __| \ FreeBSD: The Power to Serve! _ __ | _ \._ \ |) | http://www.freebsd.org _ |___/___/___/ " THAT'S WRONG WRONG WRONG!" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
[ cc'd to -hackers for the archives, reply-to points back to me so that this doesn't perpetuate on the list ] On Sat, Jun 12, 1999 at 11:24:11PM -0700, Bill Huey wrote: > I came on this list initially to just check was the FreeBSD community > was like, Then you chose the wrong list. As the web pages point out, this list is for technical discussion. It's not somewhere you go to find out what the community is like. There's already too much noise on this list, and all this is doing is contributing to it. If you want to find out what the community is like then subscribe to -chat, and read and post, or subscribe to -questions, and just read. You'll get a much better idea that way. > I'm also not big enough asshole ot put someone on a "kill-list" and is just > a reflection of a kind of conservative need to dehumanize other folks > so that your selfish comfort is "preserved". Post technical content to a technical mailing list and expect to be treated with respect. Post ill advised non technical content and expect to be treated like an idiot. As I say, feel free to join -chat to see how the FreeBSD community behaves. N -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <37514...@cs.colorado.edu> To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
On Sat, 12 Jun 1999, Bill Huey wrote: > > > The Linux philosophy has always been about simplistic cycle counting > > exercises without understand whether the data had any meaning or not. > > You've once again displayed your wholehearted participation in this > > lack of understanding of what the data points might mean to any real- > > world application. > > Well, the thing I'm talking about isn't a philosophy, because > it removed data serialization doing what I suspect is a kernel/user > space copy from doing a superficial glance of the code. > > This is one of the detected bottlenecks in the Mindcraft study that > suck copy performance by approximately 4 times. > > Another things that was included on the like bottleneck list is > the waking of all the processes in the process queue when accept() > is being called. This was termed the "Thundering Herds" problem, which > they solve using a wake-one process implementation of accept(). > > > denial" part. Or, better yet, just fuck off and get the hell off our > > list. This is NOT an appropriate forum for Linux advocacy, which seems > > to be all you can do. > > Well, so far I've heard alot of BS about Linux that isn't exactly true > and much of it seems like a bunch of artificial problems that hold > against the Linux folks. Most of it is just intentionally misrepresented > bullshit. That is totally wrong, Bill. We have a public mail archive, so if you don't like to believe me, go check our archive. YOU have been the only source of either Linux or FreeBSD bashing. We have a pretty pleasing assortment of grownups here, who have gone past the point of needing to claim "my ball is better than your ball". One of my own personal complaints about the Linux groups is that they seem to grow the kind of people who need to disparage in order to feel good. This might be because my only exposure to Linux persons is via Usenet, which is immersed in that teenage-type philosophy. I don't know of any Linux mailing lists of the order of the FreeBSD lists. We don't need that approach here, it doesn't increase knowledge, and only hurts feelings. If you follow up on this thread, please DROP all Linux/FreeBSD comparisons of that order. We all know that both Linux AND FreeBSD are fine unixes, and we don't need bashing here. This list does occaisonally have some comparisons, Linux vs. FreeBSD, but if it isn't kept strictly technical, and non-accusatory, we don't want it or need it. I have seen people in the past that were honestly surprised at the huge difference between Usenet newsgroup rules, and our FreeBSD list rules. If you actually fit in that category, maybe you should "lurk" these lists for a while, so you can see the difference, and not find yourself outside of our lists normal behaviour. > > I came on this list initially to just check was the FreeBSD community > was like, but what I've gotten since is alot of ego and hostility > toward things that aren't completely FreeBSD. That's something that > I didn't expect and reading this list has given me a particular > negative view of FreeBSD that wasn't present before. People don't like your need to compare, in order to make one group seem better or worse than another. You need to find some other way to get your point across. > > In-fighting with the current NFS maintainer and general rudeness to other > potential devs make FreeBSD's kernel people look like bunch of dorks whether > you like it or not. > > I'm also not big enough asshole ot put someone on a "kill-list" and is just > a reflection of a kind of conservative need to dehumanize other folks > so that your selfish comfort is "preserved". Profanity is also one of the things not accepted here. > > > Wes Peters Softweyr LLC > > bill > > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > +--- Chuck Robey | Interests include any kind of voice or data chu...@picnic.mat.net | communications topic, C programming, and Unix. 213 Lakeside Drive Apt T-1 | Greenbelt, MD 20770 | I run picnic and jaunt, both FreeBSD-current. (301) 220-2114 | +--- To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> The Linux philosophy has always been about simplistic cycle counting > exercises without understand whether the data had any meaning or not. > You've once again displayed your wholehearted participation in this > lack of understanding of what the data points might mean to any real- > world application. Well, the thing I'm talking about isn't a philosophy, because it removed data serialization doing what I suspect is a kernel/user space copy from doing a superficial glance of the code. This is one of the detected bottlenecks in the Mindcraft study that suck copy performance by approximately 4 times. Another things that was included on the like bottleneck list is the waking of all the processes in the process queue when accept() is being called. This was termed the "Thundering Herds" problem, which they solve using a wake-one process implementation of accept(). > denial" part. Or, better yet, just fuck off and get the hell off our > list. This is NOT an appropriate forum for Linux advocacy, which seems > to be all you can do. Well, so far I've heard alot of BS about Linux that isn't exactly true and much of it seems like a bunch of artificial problems that hold against the Linux folks. Most of it is just intentionally misrepresented bullshit. I came on this list initially to just check was the FreeBSD community was like, but what I've gotten since is alot of ego and hostility toward things that aren't completely FreeBSD. That's something that I didn't expect and reading this list has given me a particular negative view of FreeBSD that wasn't present before. In-fighting with the current NFS maintainer and general rudeness to other potential devs make FreeBSD's kernel people look like bunch of dorks whether you like it or not. I'm also not big enough asshole ot put someone on a "kill-list" and is just a reflection of a kind of conservative need to dehumanize other folks so that your selfish comfort is "preserved". > Wes Peters Softweyr LLC bill To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Bill Huey wrote: > > > Think of it like this: since alot of desktops sit in idle loops much > > of the time, perhaps the Linux philosophy has been to improve such > > behavior :-). > > The Linux philosophy already has better performance and will also get > you much stronger TCP/IP user land copy performance under SMP since > it releases locks around the data copy. The Linux philosophy has always been about simplistic cycle counting exercises without understand whether the data had any meaning or not. You've once again displayed your wholehearted participation in this lack of understanding of what the data points might mean to any real- world application. > This certain is much better that the over simplistic single MP in > FreeBSD, which has since been abandon in the Linux kernel. And your deep understanding of the technical issues surrounding such a change lends you the credence to declare it was "abandon", rather that just "moved on from," right? > But I guess technical denial works in the FreeBSD community. ;-) I don't know about the rest of you guys, but this shithead just pole- vaulted onto the top of my email "kill" list. All noise, no content. Bye, Bill. It's been nice not knowing you. In case you're wondering, I know a tiny bit about Linux SMP. I was helping debug support for SMP in LinuxPPC over the phone this week, with one of the developers working on it. I know very little about the SMP support in Linux, but have had a fair amount of experience doing MP work on RISC processors. The engineer working on it needed a couple of good, or even not-so-good questions to get him unstuck; I was happy to provide a peanut gallery. So, explain to me again about that "technical denial" part. Or, better yet, just fuck off and get the hell off our list. This is NOT an appropriate forum for Linux advocacy, which seems to be all you can do. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC http://www.softweyr.com/~softweyr w...@softweyr.com To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> > > Think of it like this: since alot of desktops sit in idle loops much > > of the time, perhaps the Linux philosophy has been to improve such > > behavior :-). > > The Linux philosophy already has better performance and will also get > you much stronger TCP/IP user land copy performance under SMP since > it releases locks around the data copy. > In a general sense, FreeBSD will still outperform Linux, however in SMP, FreeBSD is behind... It is alot of fun to look towards the "new" linux VM code, "can you say tweak, tweak???" This is a perfect example of coding vs. design. There is NO need for explicit policies (steal a page here or there in specific places.) When DG and I were first playing with the code, DG admonished me continually to avoid nonsense "policies" and that is probably a very significant contribution in the end. We ended up with a scheme that develops it's own policy, and it is difficult to track what the system is doing, let alone "control" it better than the system can do itself. Note that in a non-trivial WWW server, the system isn't waiting on TCP, but is dealing with CGI. I suspect that it would be very useful for the FreeBSD team for Linux to "improve" their SMP. I hope they create really fine grained locks all over the place... With all of those fine grained locks, they can expand the domain of the optimized spin loop :-). John To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> I agree. However, I suggest that finegrained locking will be a loosing > proposition. Something in-between is probably good. My brains got fried > on trying to figure out a GOOD solution for the FreeBSD kernel. There > are no GOOD solutions, but a reasonable compromise is some kind of medium > grained scheme. Which is why FreeBSD will go through many of same locking problems in Linux, since they are coming from monolithic kernel origins. Race conditions processor synchronization, per processor cache hit issues, etc... > John bill To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> > Wes Peters said: > > > > Try a more meaningful benchmark, one that actually does something in > > the kernel before returning, and see how they do. Try calling kill > > or socket/close a few hundred thousand times and see how they do. This will only test how fast the kernel memory allocator will perform with small object. Linux will do well since the memory allocation stuff is pretty standard issue for kernel memory allocators. > Think of it like this: since alot of desktops sit in idle loops much > of the time, perhaps the Linux philosophy has been to improve such > behavior :-). The Linux philosophy already has better performance and will also get you much stronger TCP/IP user land copy performance under SMP since it releases locks around the data copy. This certain is much better that the over simplistic single MP in FreeBSD, which has since been abandon in the Linux kernel. But I guess technical denial works in the FreeBSD community. ;-) More smilie faces ;-) ;-) ;-) > John | Never try to teach a pig to sing, > dy...@iquest.net | it makes one look stupid > jdy...@nc.com | and it irritates the pig. bill > > -- > John | Never try to teach a pig to sing, > dy...@iquest.net | it makes one look stupid > jdy...@nc.com | and it irritates the pig. > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> "John S. Dyson" writes: > > > Finegrained locking either requires developers with IQ's of 200 or higher, > > or a different kernel structure. I suggest that finegrained locking is > > cool, > > and can be intelligently used to mitigate (but not solve) the effects of > > lots of problems > > Fine grained locking is hard - but it isn't exactly rocket > science. It's been tackled in a number of OSes, papers have been > written about it. > I also have played with the SVR4 SMP kernel (the real working one), not some of the toys that called themselves SMP -- but later versions were much better than even what I worked on. I can say that the maintainability of the code sucked, and the problems were due to the ARCHITECTURE of the original kernel. Sure, it can be done, but it has to be done methodically, and simple non-SMP changes preciptate massive SMP changes. The typical monlithic kernels are just the right answer to the wrong problem, if you are talking optimal SMP solutions. > > Sure. But 2 and 4-way boxes are becoming more and more mainstream. And > any IO bound job is not going to perform well on FreeBSD because of > giant locking. > I agree. However, I suggest that finegrained locking will be a loosing proposition. Something in-between is probably good. My brains got fried on trying to figure out a GOOD solution for the FreeBSD kernel. There are no GOOD solutions, but a reasonable compromise is some kind of medium grained scheme. John To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Brian Feldman writes: > > One way of tackling the problem is - to implement per lock profiling > > and detect which locks are being contested heavily and try breaking > > them down. That would be a practical way of doing things. > > But you can't generalize FreeBSD's usage, can you? > While it's true that no one can see all possible uses of FreeBSD, one has to make assumptions about the typical usage - web server, file server etc and use it as the design center, while making sure that it doesn't perform too badly on other less common workloads. -Arun To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Soren Schmidt wrote: > > It seems Arun Sharma wrote: > > An alternative way, which requires a good understanding of both the > > theory and implementation of the kernel is - > > > > (a) Implement per subsystem locking > > This was exactly what I experimented with some 7-8 month ago, it > showed significant improvements in performance, yet it was relatively > easy to do. > > > (b) Figure out in a "typical" workload, how much time is being spent > > in which subsystem and try increasing parallelism (i.e. finer > > grained locking) in subsystems where more time is being spent. > > Well, my simple tests showed no need for this, again its a fine > balance between spending the time waiting, and spending the time > processing locks. Implementing some (optional) benchmarking code for the locks would make it pretty simple to do this as a follow-on project, once the initial "several-sorta-big-locks" project is stable. There's always room for improvement. > > The result of this approach should be more logical, cleaner and > > possibly better performing than the previous one. > > It sure is easier to understand and to maintain. Maybe I should > try it again, and this time keep off-site backups :) I have disk space available. ;^) Actually, I have a DLT drive available now, too. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC http://www.softweyr.com/~softweyr w...@softweyr.com To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
It seems Arun Sharma wrote: > An alternative way, which requires a good understanding of both the > theory and implementation of the kernel is - > > (a) Implement per subsystem locking This was exactly what I experimented with some 7-8 month ago, it showed significant improvements in performance, yet it was relatively easy to do. > (b) Figure out in a "typical" workload, how much time is being spent > in which subsystem and try increasing parallelism (i.e. finer > grained locking) in subsystems where more time is being spent. Well, my simple tests showed no need for this, again its a fine balance between spending the time waiting, and spending the time processing locks. > The result of this approach should be more logical, cleaner and > possibly better performing than the previous one. It sure is easier to understand and to maintain. Maybe I should try it again, and this time keep off-site backups :) -Søren To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
On 12 Jun 1999, Arun Sharma wrote: > "John S. Dyson" writes: > > > Finegrained locking either requires developers with IQ's of 200 or higher, > > or a different kernel structure. I suggest that finegrained locking is > > cool, > > and can be intelligently used to mitigate (but not solve) the effects of > > lots of problems > > Fine grained locking is hard - but it isn't exactly rocket > science. It's been tackled in a number of OSes, papers have been > written about it. That's untrue if you're using asynchronous I/O. > > > -- however, it would be unwise to embark on an effort to make > > the FreeBSD kernel into an efficent 16way SMP kernel by using finegrained > > locking all over the place. > > Sure. But 2 and 4-way boxes are becoming more and more mainstream. And > any IO bound job is not going to perform well on FreeBSD because of > giant locking. Not quite as well, but it will still be much faster. Besides, FreeBSD would perform better under load than nearly any other OS, so it's not as if it would be slower than something else. > > One way of tackling the problem is - to implement per lock profiling > and detect which locks are being contested heavily and try breaking > them down. That would be a practical way of doing things. But you can't generalize FreeBSD's usage, can you? > > An alternative way, which requires a good understanding of both the > theory and implementation of the kernel is - > > (a) Implement per subsystem locking That seems to be the most winning solution. > (b) Figure out in a "typical" workload, how much time is being spent > in which subsystem and try increasing parallelism (i.e. finer > grained locking) in subsystems where more time is being spent. Once again, there is no "typical" unless you try to overgeneralize and say "FreeBSD is used mostly for XXX." This would be just an extension of a. But this is much less of a gain (from what I conjecture). It would make sense to have separate locks for, say, at least the I/O subsystems (all in total, not for each one. Why do you want to parallel something that's doing DMA?), FS interfaces, VM/trapping, and networking. > > The result of this approach should be more logical, cleaner and > possibly better performing than the previous one. > > -Arun > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > Brian Feldman_ __ ___ ___ ___ ___ gr...@unixhelp.org_ __ ___ | _ ) __| \ FreeBSD: The Power to Serve! _ __ | _ \._ \ |) | http://www.freebsd.org _ |___)___/___/ To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
On Sat, 12 Jun 1999, John S. Dyson wrote: > Soren Schmidt said: > [Charset ISO-8859-1 unsupported, filtering to ASCII...] > > It seems Christopher R. Bowman wrote: > > [exelent explanation snipped] > > > The alternative to the Giant Kernel Lock(tm) is so called fine grained > > > locking > > > wherein locking is pushed down closer to the data structures. In fine > > > grained > > > locking two processors might be executing in the kernel at the same time, > > > but > > > only if they didn't need the same resources. On might be doing a disk > > > read > > > while the other queues up a character for the serial port. The fine > > > grained > > > lock has the potential for higher parallelism and thus better throughput > > > since > > > process may not have to wait as long, but the larger number of locks with > > > their > > > many required lock and unlock operations add overhead and further the > > > design is > > > more difficult and error prone since the interaction of the numerous > > > locks may > > > result in deadlock or livelock situations every bit as problematical as > > > the > > > problem they try to solve. > > > > There are also those of us that dont belive in finegrained locking, exactly > > because of all the small locks you have to check/lock/open, the overhead is > > not worth it. > > > Finegrained locking either requires developers with IQ's of 200 or higher, > or a different kernel structure. I suggest that finegrained locking is cool, > and can be intelligently used to mitigate (but not solve) the effects of > lots of problems -- however, it would be unwise to embark on an effort to make > the FreeBSD kernel into an efficent 16way SMP kernel by using finegrained > locking all over the place. But your microkernel-hybrid BSD will do 16way SMP with a fully-parallelized kernel? > > -- > John | Never try to teach a pig to sing, > dy...@iquest.net | it makes one look stupid > jdy...@nc.com | and it irritates the pig. > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > Brian Feldman_ __ ___ ___ ___ ___ gr...@unixhelp.org_ __ ___ | _ ) __| \ FreeBSD: The Power to Serve! _ __ | _ \._ \ |) | http://www.freebsd.org _ |___)___/___/ To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
"John S. Dyson" writes: > Finegrained locking either requires developers with IQ's of 200 or higher, > or a different kernel structure. I suggest that finegrained locking is cool, > and can be intelligently used to mitigate (but not solve) the effects of > lots of problems Fine grained locking is hard - but it isn't exactly rocket science. It's been tackled in a number of OSes, papers have been written about it. > -- however, it would be unwise to embark on an effort to make > the FreeBSD kernel into an efficent 16way SMP kernel by using finegrained > locking all over the place. Sure. But 2 and 4-way boxes are becoming more and more mainstream. And any IO bound job is not going to perform well on FreeBSD because of giant locking. One way of tackling the problem is - to implement per lock profiling and detect which locks are being contested heavily and try breaking them down. That would be a practical way of doing things. An alternative way, which requires a good understanding of both the theory and implementation of the kernel is - (a) Implement per subsystem locking (b) Figure out in a "typical" workload, how much time is being spent in which subsystem and try increasing parallelism (i.e. finer grained locking) in subsystems where more time is being spent. The result of this approach should be more logical, cleaner and possibly better performing than the previous one. -Arun To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
"Christopher R. Bowman" writes: > > I can't speak authoritatively since I don't know specifically what > SYSCALL_LOCK is, but if it is what is often referred to on this list > as the Giant Kernel Lock(tm) then the following should generally > apply. > You're right. The SYSCALL_LOCK is the same as the giant lock. The name kinda misled me to assume that it's a different lock. i386/i386/lock.h: /* * Some handy macros to allow logical organization and * convenient reassignment of various locks. */ #define FPU_LOCKcall_get_fpu_lock #define ALIGN_LOCK call_get_align_lock #define SYSCALL_LOCKcall_get_syscall_lock #define ALTSYSCALL_LOCK call_get_altsyscall_lock All of the above routines seem to be identical. But the code is duplicated for some reason. Also, it might be beneficial to define these locks in a header file and inline them, instead of generating a call for each simple_lock. -Arun To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Aaron Smith writes: > I'm still trying to figure out the deal with "lockmgr". I found the following doc useful: http://www.freebsd.org/~fsmp/SMP/Locking.html -Arun To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Wes Peters said: > > Try a more meaningful benchmark, one that actually does something in > the kernel before returning, and see how they do. Try calling kill > or socket/close a few hundred thousand times and see how they do. > Historically, my emphasis on FreeBSD kernel work was to make it work best while under load, doing lots of things. By benchmarking the system while the system isn't really doing anything really pessimizes the advantages of the FreeBSD choices. Linux is indeed much faster at doing nothing!!! I suspect that some intrepid individual could improve FreeBSD a little, but the risk/reward would be too severe. Think of it like this: since alot of desktops sit in idle loops much of the time, perhaps the Linux philosophy has been to improve such behavior :-). -- John | Never try to teach a pig to sing, dy...@iquest.net | it makes one look stupid jdy...@nc.com | and it irritates the pig. To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Soren Schmidt said: [Charset ISO-8859-1 unsupported, filtering to ASCII...] > It seems Christopher R. Bowman wrote: > [exelent explanation snipped] > > The alternative to the Giant Kernel Lock(tm) is so called fine grained > > locking > > wherein locking is pushed down closer to the data structures. In fine > > grained > > locking two processors might be executing in the kernel at the same time, > > but > > only if they didn't need the same resources. On might be doing a disk read > > while the other queues up a character for the serial port. The fine grained > > lock has the potential for higher parallelism and thus better throughput > > since > > process may not have to wait as long, but the larger number of locks with > > their > > many required lock and unlock operations add overhead and further the > > design is > > more difficult and error prone since the interaction of the numerous locks > > may > > result in deadlock or livelock situations every bit as problematical as the > > problem they try to solve. > > There are also those of us that dont belive in finegrained locking, exactly > because of all the small locks you have to check/lock/open, the overhead is > not worth it. > Finegrained locking either requires developers with IQ's of 200 or higher, or a different kernel structure. I suggest that finegrained locking is cool, and can be intelligently used to mitigate (but not solve) the effects of lots of problems -- however, it would be unwise to embark on an effort to make the FreeBSD kernel into an efficent 16way SMP kernel by using finegrained locking all over the place. -- John | Never try to teach a pig to sing, dy...@iquest.net | it makes one look stupid jdy...@nc.com | and it irritates the pig. To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
It seems Christopher R. Bowman wrote: [exelent explanation snipped] > The alternative to the Giant Kernel Lock(tm) is so called fine grained locking > wherein locking is pushed down closer to the data structures. In fine grained > locking two processors might be executing in the kernel at the same time, but > only if they didn't need the same resources. On might be doing a disk read > while the other queues up a character for the serial port. The fine grained > lock has the potential for higher parallelism and thus better throughput since > process may not have to wait as long, but the larger number of locks with > their > many required lock and unlock operations add overhead and further the design > is > more difficult and error prone since the interaction of the numerous locks may > result in deadlock or livelock situations every bit as problematical as the > problem they try to solve. There are also those of us that dont belive in finegrained locking, exactly because of all the small locks you have to check/lock/open, the overhead is not worth it. I think a model where locks placed around resonably sized subsystems is the way to go, and can be made much easier to understand and free from (too many) bugs. I did some experiments some time ago with locks around each devicedriver, the netsubsystem, the vmsubsystem, the filesubsystem and the rest, and it performed admirably. Unfortunately I lost all that work due to "unforeseen physical happenings", but is was not that big a deal to do... I think we should consider it very carefully before going the finegrained lock route, it really doesn't give you that much in real world situations. -Søren To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
At 11:15 PM 6/11/99 -0700, Arun Sharma wrote: >"David E. Cross" writes: > >> Looking through the exception.s it appears that on entry to the >> kernel an MP lock is obtained... I thought we had splX(); to >> protect concurancy in the kernel. > >Can someone explain to me why is SYSCALL_LOCK necessary ? It certainly >seems to hurt system call performance on a MP machine. > >Also, is there any data on lock contention in FreeBSD ? Is anyone >working on decomposing some of the giant locks ? I can't speak authoritatively since I don't know specifically what SYSCALL_LOCK is, but if it is what is often referred to on this list as the Giant Kernel Lock(tm) then the following should generally apply. When dealing with data structures there are often areas of code that must run with out any possibility of being interrupted or executing at the same time as another specific piece of code. By way of example, consider the simple act of incrementing a shared variable. In most modern RISC processors this is essentially 3 instructions: LOAD r2, #var load address of variable var into register r2 LOAD r1,r2load the value stored in var into cpu register r1 ADD r1,r1,#1 add one to r1 and store result in r1 STOREr2,r1store the incremented value of var back into memory Suppose that this simple 4 instruction code sequence was interrupted after the second load and before the add, and sometime later execution resumes at the add instruction. Lets first assume that we are on a uniprocessor machine, in which case there are only 2 ways this sequence of instruction can stop before that add and come back later. 1) if we are in user mode then a page boundary can occur between the load and the add, and this could lead to a page fault exception (this would be a page fault on the instruction read for the add). This cannot occur in kernel mode since if we were in kernel mode we would be inside the kernel, and the kernel wires down all of it's code pages (this might change in the future but probably only for pages that can be guaranteed will never be executed again like the startup splash screen code maybe). 2) an interrupt could occur such that the load completes, but the add does not start. This could be for a variety of reasons: perhaps a disk requests has finished, or maybe the sound card needs new data, or a clock interrupt signaling the end of a process quanta may occur. This interrupt may occur weather the code is running in user mode or in kernel mode. In either of the above 2 cases the processor typically saves the current context/state of the processor, switches to and interrupt or excerption context/state, and goes off to execute either a page fault exception handler or an interrupt handler as appropriate for the particular case. If in kernel mode at the time of the interrupt it restores the context and resumes where it was interrupted at the conclusion of the handler. If in user mode the processor may or may not resume executing the user process immediately after the interrupt is processed, but generally (baring a signal delivery for instance) when the process does resume execution it will restore the context/state and resume at the add. Now having examined how the example code could have it's execution interrupted, let us further suppose that the code that executes in between the load and the add also increments the same variable. If it happens to run a similar 4 instruction sequence then we can see that the value that get stored in memory doesn't reflect the desired effect of the 2 pieces of code and it appears as if the variable was incremented only once when it was intended to be incremented twice. If this happens to kernel code it can have catastrophic effects. To avoid this we must have some way of preventing the two pieces of code from intermingling their execution. In the FreeBSD (and all single threaded) kernels this is handled by locking out interrupts around these so called critical sections of code. So, for instance, if a variable was shared with the only network interrupt handler, the kernel would make a x=splnet() call before the second load and an splx(x) call after the store. Since the network interrupt is prevented from being serviced during between the spl calls the kernel is guaranteed that the load, add, store will proceed with out problems, and thus our problem is solved. We can see that this is so: the network interrupt which changes the variable has it's execution delayed until after the kernel updates the variable, and by assumption the other interrupts which may occur and won't be delayed, don't change the variable and thus don't present a problem. Now consider the multiprocessor case. We have the same problem we had with the single processor case, but now we also have a new variant. Suppose that both processors enter the kernel and both want to execute kernel code which will update a variable. If the second processor perf
Re: High syscall overhead?
On 11 Jun 1999 23:15:15 PDT, Arun Sharma writes: >Can someone explain to me why is SYSCALL_LOCK necessary ? It certainly >seems to hurt system call performance on a MP machine. > >Also, is there any data on lock contention in FreeBSD ? Is anyone >working on decomposing some of the giant locks ? I have a follow-on question: is there any planned work to give FreeBSD some of the basic synch primitives? I would love to help finer-grain the kernel, (having just today built my first SMP FreeBSD system), but first I think I'd need to implement mutexes and condition variables. It looks like there may be spin/sleeplocks and rwlocks, but they're not called that. Is there any work being done in this area? I think implementing the SVR4 synch primitives (mutex, condvars, maybe semas, rwlocks) would be great, since that's what's taught, and they're intuitive. I'm still trying to figure out the deal with "lockmgr". -- Aaron Smith VERITAS Software File System Engineer"I'll call him mini me". To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
"David E. Cross" writes: > Looking through the exception.s it appears that on entry to the > kernel an MP lock is obtained... I thought we had splX(); to > protect concurancy in the kernel. Can someone explain to me why is SYSCALL_LOCK necessary ? It certainly seems to hurt system call performance on a MP machine. Also, is there any data on lock contention in FreeBSD ? Is anyone working on decomposing some of the giant locks ? -Arun To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
I have installed 3.0-RELEASE FreeBSD on AMD K6, 233MHz, 128MB, ABit
Motherboard, not overclocked/
Ran this program, got the following results:
5.2u 8.5s 0:14.02 98.3% 5+171k 0+0io 0pf+0w. :-(
-Original Message-
From: Chris Costello
To: David E. Cross
Cc: freebsd-hackers@FreeBSD.ORG ;
freebsd-...@freebsd.org
Date: Friday, June 11, 1999 11:01 AM
Subject: Re: High syscall overhead?
>On Fri, Jun 11, 1999, David E. Cross wrote:
>> Just doing some performance testing and I noticed something rather
>> disturbing
>>
>> Here is the test program:
>> int main (void)
>> {
>> int count=0;
>> for(count=0;count <1000;++count)
>> getppid();
>>
>> return 0;
>> }
>>
>> The time on linux for this program is ~5 seconds (linux "time" reports
3.x, but
>> a wall clock clearly shows 5.x, go fig). FreeBSD reports 18.x
seconds?!. I
>> have a dual processor system and decided to parallel run them... it took
>> 52!?! seconds, linux on the same was again about 5. Looking through the
>> exception.s it appears that on entry to the kernel an MP lock is
obtained...
>> I thought we had splX(); to protect concurancy in the kernel.
>
>('sc' being the program above, compiled without optimization,
> just with cc -o sc sc.c)
>$ time ./sc
> 10.04s real 3.89s user 5.64s system
>
> I counted between around 9 and a half to 10 and a half seconds
>on my wall clock (trusty old GE, same model they have in public
>schools).
>
>Copyright (c) 1992-1999 The FreeBSD Project.
>Copyright (c) 1982, 1986, 1989, 1991, 1993
>The Regents of the University of California. All rights reserved.
>FreeBSD 4.0-CURRENT #4: Sun May 30 04:22:23 CDT 1999
>r...@holly.dyndns.org:/usr/src/sys/compile/Holly
>Timecounter "i8254" frequency 1193182 Hz
>CPU: AMD-K6(tm) 3D processor (350.80-MHz 586-class CPU)
> Origin = "AuthenticAMD" Id = 0x580 Stepping=0
> Features=0x8001bf
>real memory = 67108864 (65536K bytes)
>sio0: system console
>avail memory = 62267392 (60808K bytes)
>
> SMP specific bug, perhaps?
>
>>
>> I am just curious what's the story with this. On some of my other tests
it is
>> clear that FreeBSD is handling concurancy much better than linux (by an
equal
>> factor actually, and on "real" tasks like real I/O handling).
>>
>> --
>> David Cross | email: cro...@cs.rpi.edu
>> Systems Administrator/Research Programmer | Web:
http://www.cs.rpi.edu/~crossd
>> Rensselaer Polytechnic Institute, | Ph: 518.276.2860
>> Department of Computer Science| Fax: 518.276.4033
>> I speak only for myself. | WinNT:Linux::Linux:FreeBSD
>>
>>
>> To Unsubscribe: send mail to majord...@freebsd.org
>> with "unsubscribe freebsd-hackers" in the body of the message
>
>--
>Chris Costello
>This message transmitted on 100% recycled electrons.
>
>
>To Unsubscribe: send mail to majord...@freebsd.org
>with "unsubscribe freebsd-smp" in the body of the message
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
> Try a more meaningful benchmark, one that actually does something in > the kernel before returning, and see how they do. Try calling kill > or socket/close a few hundred thousand times and see how they do. Or that horribily impracticle wake-one semantics implemented under SMP for the accept() function with recent Linux kernels to prevent overscheduling. Or another really useless thing like releasing a MP lock in the TCP/IP stack the increases user space copies by a factor of 4 times in the Linux kernel. FreeBSD still using a single big kernel lock which slows down certain unimportant things like getting every so slow course grained kernel lock ? Are things breaking with all the new changes in the FreeBSD kernel ? Uh, "yes" to the two above questions ? > Wes Peters Softweyr LLC bill To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
"David E. Cross" wrote:
>
> Just doing some performance testing and I noticed something rather
> disturbing
>
> Here is the test program:
> int main (void)
> {
> int count=0;
> for(count=0;count <1000;++count)
> getppid();
>
> return 0;
> }
>
> The time on linux for this program is ~5 seconds (linux "time" reports 3.x,
> but
> a wall clock clearly shows 5.x, go fig). FreeBSD reports 18.x seconds?!. I
> have a dual processor system and decided to parallel run them... it took
> 52!?! seconds, linux on the same was again about 5. Looking through the
> exception.s it appears that on entry to the kernel an MP lock is obtained...
> I thought we had splX(); to protect concurancy in the kernel.
>
> I am just curious what's the story with this. On some of my other tests it is
> clear that FreeBSD is handling concurancy much better than linux (by an equal
> factor actually, and on "real" tasks like real I/O handling).
My PII/300 reports:
w...@homer$ time ./a.out
real0m12.490s
user0m5.898s
sys 0m6.455s
Let's see, that's:
w...@homer$ bc
bc 1.04
Copyright (C) 1991, 1992, 1993, 1994, 1997 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
1000/1249
8006
10/1249
800640
800,000 syscalls/sec and you're complaining? Heh. ;^)
I suspect this is one of those "optimizations" where the Linux syscall
mechanism doesn't actually enter the kernel, since the parent pid is
probably stored in the equivalent of the proc struct. True to the
Linux "every cycle is precious" ethic, these optimizations are great
for looking good on meaningless benchmarks like this one, and make no
difference at all in "real life."
Try a more meaningful benchmark, one that actually does something in
the kernel before returning, and see how they do. Try calling kill
or socket/close a few hundred thousand times and see how they do.
--
"Where am I, and what am I doing in this handbasket?"
Wes Peters Softweyr LLC
http://www.softweyr.com/~softweyr w...@softweyr.com
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
David E. Cross was heard to mumble: > Oops, here is some additional information from my system: > I can reproduce this under -CURRENT: Dual P200MMX running FreeBSD 4.0-CURRENT-990528: fire /home/caesar/src/tmp $ time ./sc real0m34.695s user0m20.457s sys 0m13.850s Single P200MMX running RH 6.0 Linux: hindenburg /home/wduncan/src/tmp $ time ./sc 2.24user 5.09system 0:07.53elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (65major+8minor)pagefaults 0swaps -- William S. Duncansoncae...@starkreality.com Smash forehead on keyboard to continue... To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
Oops, here is some additional information from my system:
bash-2.02$ cat sc.c
int main (void)
{
int count=0;
for(count=0;count <1000;++count)
getppid();
return 0;
}
bash-2.02$ cc -o sc sc.c
bash-2.02$ uptime
11:19AM up 3 hrs, 4 users, load averages: 0.01, 0.01, 0.00
bash-2.02$ time ./sc
real0m18.523s
user0m0.000s
sys 0m18.521s
bash-2.02$ (date;time ./sc;date) & (date;time ./sc;date) &
[3] 516
[4] 517
Fri Jun 11 11:21:51 EDT 1999
[1] Donetime ./sc
Fri Jun 11 11:21:51 EDT 1999
[2] Donetime ./sc
bash-2.02$
real0m52.058s
user0m0.000s
sys 0m52.056s
real0m52.056s
user0m0.000s
sys 0m52.053s
Fri Jun 11 11:22:43 EDT 1999
Fri Jun 11 11:22:43 EDT 1999
bash-2.02$ dmesg | head
Copyright (c) 1992-1999 FreeBSD Inc.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
FreeBSD 3.2-STABLE #0: Fri Jun 11 08:18:12 EDT 1999
r...@phoenix.home:/usr/src/sys/compile/PHOENIX_DUAL
Timecounter "i8254" frequency 1193182 Hz
CPU: Pentium II/Xeon/Celeron (686-class CPU)
Origin = "GenuineIntel" Id = 0x652 Stepping=2
Features=0x183fbff>
real memory = 268435456 (262144K bytes)
avail memory = 257925120 (251880K bytes)
Programming 24 pins in IOAPIC #0
FreeBSD/SMP: Multiprocessor motherboard
cpu0 (BSP): apic id: 1, version: 0x00040011, at 0xfee0
cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee0
io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec0
Preloaded elf kernel "kernel" at 0xc02df000.
Probing for devices on PCI bus 0:
chip0: rev 0x03 on pci0.0.0
chip1: rev 0x03 on pci0.1.0
It is -STABLE from June 7th, mid-day.
--
David Cross | email: cro...@cs.rpi.edu
Systems Administrator/Research Programmer | Web: http://www.cs.rpi.edu/~crossd
Rensselaer Polytechnic Institute, | Ph: 518.276.2860
Department of Computer Science| Fax: 518.276.4033
I speak only for myself. | WinNT:Linux::Linux:FreeBSD
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: High syscall overhead?
On Fri, Jun 11, 1999, David E. Cross wrote:
> Just doing some performance testing and I noticed something rather
> disturbing
>
> Here is the test program:
> int main (void)
> {
> int count=0;
> for(count=0;count <1000;++count)
> getppid();
>
> return 0;
> }
>
> The time on linux for this program is ~5 seconds (linux "time" reports 3.x,
> but
> a wall clock clearly shows 5.x, go fig). FreeBSD reports 18.x seconds?!. I
> have a dual processor system and decided to parallel run them... it took
> 52!?! seconds, linux on the same was again about 5. Looking through the
> exception.s it appears that on entry to the kernel an MP lock is obtained...
> I thought we had splX(); to protect concurancy in the kernel.
('sc' being the program above, compiled without optimization,
just with cc -o sc sc.c)
$ time ./sc
10.04s real 3.89s user 5.64s system
I counted between around 9 and a half to 10 and a half seconds
on my wall clock (trusty old GE, same model they have in public
schools).
Copyright (c) 1992-1999 The FreeBSD Project.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
FreeBSD 4.0-CURRENT #4: Sun May 30 04:22:23 CDT 1999
r...@holly.dyndns.org:/usr/src/sys/compile/Holly
Timecounter "i8254" frequency 1193182 Hz
CPU: AMD-K6(tm) 3D processor (350.80-MHz 586-class CPU)
Origin = "AuthenticAMD" Id = 0x580 Stepping=0
Features=0x8001bf
real memory = 67108864 (65536K bytes)
sio0: system console
avail memory = 62267392 (60808K bytes)
SMP specific bug, perhaps?
>
> I am just curious what's the story with this. On some of my other tests it is
> clear that FreeBSD is handling concurancy much better than linux (by an equal
> factor actually, and on "real" tasks like real I/O handling).
>
> --
> David Cross | email: cro...@cs.rpi.edu
> Systems Administrator/Research Programmer | Web:
> http://www.cs.rpi.edu/~crossd
> Rensselaer Polytechnic Institute, | Ph: 518.276.2860
> Department of Computer Science| Fax: 518.276.4033
> I speak only for myself. | WinNT:Linux::Linux:FreeBSD
>
>
> To Unsubscribe: send mail to majord...@freebsd.org
> with "unsubscribe freebsd-hackers" in the body of the message
--
Chris Costello
This message transmitted on 100% recycled electrons.
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
