Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Hi everyone, I've been following this discussion almost from the beginning, and I have the feeling that we're not _really_ getting very far. There's good arguments for and against overcommit, depending on your point of view and your requirements. What I do see is a not-so-openly voiced consent that the way resource(sp?) shortages are handled in an overcommitting system (SIGKILL) makes some of us rather unhappy. I therefore suggest those of us who would like to see a change in this area pool their efforts and energies to work on a mechanism that handles resource shortage in a more graceful way. cheerio Michael -- [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: more amd hangs: problem really in syslog?
Mike Smith wrote: > > > On Tue, 13 Jul 1999, Mike Smith wrote: > > > > > 'siobi' is someone trying to open the serial console, for whatever > > > reason. Without knowing who it was that was stuck there, it's hard to > > > guess what is going on. > > > > D'uh, sorry. Long day. It was amd that was hung in the siobi > > state. No way to clear it without rebooting the box. > > Dang. Now I need that stack dump from amd that you posted and I > deleted. OK, sent under different cover. > Specifically, it'd be handy to know why amd felt it was > necessary to open the console. Yeah, I'm kind of curious myself. BTW, I was going to work on this some more today, but the boss thought that putting the box into production was more important. The good news is that under real world load my freebsd box had 20-40% free cpu and a load average of 1.5. With load as equal as the switch could make it, the linux box had no free cpu and a load average of 8. :) I also (finally) got the approval to install freebsd on the fourth box (there are already two linux machines up) so A) I'm making progress in the office, and B) I should have a chance to pound on the syslog stuff tomorrow. Happy, Doug To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: seg fault in mutex_queue_enq
Daniel Eischen writes: > There are some bugs in libc_r in stable that have been fixed in > -current. I think the one that you've hit is an uninitialized > TAILQ_HEAD in a statically declared mutex (in localtime). It's > probably about time for a MFC. If someone wants to give me the > go-ahead, I can do it... Just Do It (tm). At least you could merge the change from [pthread_private.h] 1.20 Sun Jun 20 8:28:08 1999 UTC by jb Diffs to 1.19 In the words of the author: o The polling mechanism for I/O readiness was changed from select() to poll(). In additon, a wrapped version of poll() is now provided. [...] plus the later formatting fixes. As an aside: some files still lack $Id$'s, and someone should `s/REGENTS/AUTHOR/' in all the copyright statements. tg To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: more amd hangs: problem really in syslog?
> On Tue, 13 Jul 1999, Mike Smith wrote: > > > 'siobi' is someone trying to open the serial console, for whatever > > reason. Without knowing who it was that was stuck there, it's hard to > > guess what is going on. > > D'uh, sorry. Long day. It was amd that was hung in the siobi > state. No way to clear it without rebooting the box. Dang. Now I need that stack dump from amd that you posted and I deleted. Specifically, it'd be handy to know why amd felt it was necessary to open the console. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
On Thu, Jul 15, 1999 at 01:48:40PM +0900, Daniel C. Sobral wrote: > > > > If you have a lot of users, all of which have buggy programs which eat > > a lot of memory, per-user swap quotas don't necessarily save your butt. > > The chance of these buggy programs running at the same time is not > exactly high... Well, it is higher than your probably giving credit for. Suppose Professor A. hands-out X assignment. Unfortunately, some piece of code he supplied to his, let's say 200 students ignorant first year students, has this particular memory-eating bug. Being ignorant first-year students, they will notice something is wrong, assume the problem is their fault, and repeat the exact same procedure five or so times. Again, being ignorant first year students, they will probably all be using the same shell server. To make things worse, some wise-ass may have told a bunch of them how to use ulimit or limit in order to push their available resources as high as possible (perhaps very high, since the admin hopefully recognizes that sometimes students need high resource limits to perform research). Fortunately, overcommit rescues the machine and kills those buggy programs instead of letting them spin around for ever in some kind of "malloc() failed ... must be temporary failure, wait and retry". -- This is my .signature which gets appended to the end of my messages. To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: gdb instead of adb
> Is the reason why adb hasn't been ported to freebsd because the source is > proprietary? You make no sense. > If gdb should suffice for my debugging needs, how can a breakpoint be set > at a particular interrupt, or even at any interrupt? The break command > only seems to accept functions, offsets, linenumbers and addresses... > I'm all out of ideas and the gdb info file isn't helping. You make little more sense, unless you are talking about using gdb-remote on a running kernel, in which case you should know that interrupts are vectored through functions, and thus the entire issue is moot. Note also that debugging through interrupt handlers can be problematic on PC hardware. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> > And what do you do, then, with the processes that happen to have > legitimate use for more stack? > > Or maybe you just find out how much stack each process uses, and > then set limits appropriate for each one? Which is the equivalent of > setting limits to each user, of course... You get a little program, like eg. Xenix and Minix had, which lets you modify the executable header to indicate how much stack the system should reserve. If the program decides to use more stack for some reason, then it dies; this is in effect "stack overcommit". 8) -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: seg fault in mutex_queue_enq
Daniel Eischen <[EMAIL PROTECTED]> writes: > There are some bugs in libc_r in stable that have been fixed in > -current. I think the one that you've hit is an uninitialized > TAILQ_HEAD in a statically declared mutex (in localtime). It's > probably about time for a MFC. If someone wants to give me the > go-ahead, I can do it... Just Do It (tm). At least you could merge the change from [pthread_private.h] 1.20 Sun Jun 20 8:28:08 1999 UTC by jb Diffs to 1.19 In the words of the author: o The polling mechanism for I/O readiness was changed from select() to poll(). In additon, a wrapped version of poll() is now provided. [...] plus the later formatting fixes. As an aside: some files still lack $Id$'s, and someone should `s/REGENTS/AUTHOR/' in all the copyright statements. tg To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: more amd hangs: problem really in syslog?
> On Tue, 13 Jul 1999, Mike Smith wrote: > > > 'siobi' is someone trying to open the serial console, for whatever > > reason. Without knowing who it was that was stuck there, it's hard to > > guess what is going on. > > D'uh, sorry. Long day. It was amd that was hung in the siobi > state. No way to clear it without rebooting the box. Dang. Now I need that stack dump from amd that you posted and I deleted. Specifically, it'd be handy to know why amd felt it was necessary to open the console. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ [EMAIL PROTECTED] \\-- Joseph Merrick \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
On Thu, Jul 15, 1999 at 01:48:40PM +0900, Daniel C. Sobral wrote: > > > > If you have a lot of users, all of which have buggy programs which eat > > a lot of memory, per-user swap quotas don't necessarily save your butt. > > The chance of these buggy programs running at the same time is not > exactly high... Well, it is higher than your probably giving credit for. Suppose Professor A. hands-out X assignment. Unfortunately, some piece of code he supplied to his, let's say 200 students ignorant first year students, has this particular memory-eating bug. Being ignorant first-year students, they will notice something is wrong, assume the problem is their fault, and repeat the exact same procedure five or so times. Again, being ignorant first year students, they will probably all be using the same shell server. To make things worse, some wise-ass may have told a bunch of them how to use ulimit or limit in order to push their available resources as high as possible (perhaps very high, since the admin hopefully recognizes that sometimes students need high resource limits to perform research). Fortunately, overcommit rescues the machine and kills those buggy programs instead of letting them spin around for ever in some kind of "malloc() failed ... must be temporary failure, wait and retry". -- This is my .signature which gets appended to the end of my messages. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: gdb instead of adb
> Is the reason why adb hasn't been ported to freebsd because the source is > proprietary? You make no sense. > If gdb should suffice for my debugging needs, how can a breakpoint be set > at a particular interrupt, or even at any interrupt? The break command > only seems to accept functions, offsets, linenumbers and addresses... > I'm all out of ideas and the gdb info file isn't helping. You make little more sense, unless you are talking about using gdb-remote on a running kernel, in which case you should know that interrupts are vectored through functions, and thus the entire issue is moot. Note also that debugging through interrupt handlers can be problematic on PC hardware. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ [EMAIL PROTECTED] \\-- Joseph Merrick \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
lyn...@orthanc.ab.ca wrote: > > What it so evil about having a reasonably intelligent malloc() that > tells the truth, and returns unused memory to the system? Overcommit > is for lazy programmers, plain and simple. At least the SGI documentation > about overcommit admits that (or at least, did at one time). Yes. So is high-level languages, as a matter of fact. True memory-conscious programmers will never use anything besides assembler. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Jason Thorpe wrote: > > If you have a lot of users, all of which have buggy programs which eat > a lot of memory, per-user swap quotas don't necessarily save your butt. The chance of these buggy programs running at the same time is not exactly high... > And maybe the individual programs didn't encounter their resource limits. > > ...but the sheer number of these runaway things caused the overcommit to > be a problem. If malloc() or whatever had actually returned NULL at the > right time (i.e. as backing store was about to become overcommitted), then > these runaway processes would have stopped running away (they would have > gotten a SIGSEGV and died). > > Anyhow, my "lame undergrads" example comes from a time when PCs weren't > really powerful enough for the job (or something; anyhow, we didn't have > any in the department :-). My example is from a Sequent Balance (16 > ns32032 processors, 64M RAM [I think; been a while], 4.2BSD variant). So, tell me... when NetBSD gets it's non-overcommit switch, would you use it in the environment you describe? -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
John Nemeth wrote: > > The machine in question has run out of swap, due to unforseeable > excessive memory demands. This was accompanied by processes > complaining about not being able to allocate memory and then cleaning > up after themselves. I did not see random processes being killed > because of it. That is the way things should be. From this, I can > assume that the OS doesn't overcommit. In case, you're wondering, the > OS in question is: > > SunOS 5.5 Generic_103093-25 sun4u sparc Uh... like any modern unix, Solaris overcommits. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Michael Richardson wrote: > > Ben> Tell me, Mr. Nemeth, has this ever happened to you? Have you ever > Ben> come *close*? > > Uh, since we don't run overcommit, the answer is specifically *NO*. And what system do you run? > I have had it happen on other systems. (Solaris, AIX) It was very > mystifying to diagnose. Sure, the systems were misconfigured for what we > were trying to do, but if I wanted build a custom system for every > application well... I'd be running NT. I have to agree about the mystifying diagnose... Specially when they *don't* page like hell. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Michael Richardson wrote: > > No, I don't agree. > > This is a biggest argument against solving the overcommit situation with > SIGKILL. I have no problem with overcommit as a concept, I have a problem > with being unable to keep my possibly big processes (X, rpc.nisd, > etc. depending on cicumstances) from being victims. It is no more difficult to protect big processes than it is to create user limits. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
John Nemeth wrote: > > On one system I administrate, the largest process is typically > rpc.nisd (the NIS+ server daemon). Killing that process would be a > bad thing (TM). You're talking about killing random processes. This > is no way to run a system. It is not possible for any arbitrary > decision to always hit the correct process. That is a decision that > must be made by a competent admin. This is the biggest argument > against overcommit: there is no way to gracefully recover from an > out of memory situation, and that makes for an unreliable system. If you run out of memory, it is either a misconfigured system, or a runaway program. If a program is runaway, then: 1) It is larger than your typical rpc.nisd. 2) You cannot tell the system a priori to kill it, because you don't know about it (or else, you wouldn't be running it in first place). A system running in overcommit assumes that you have it correctly configured so it will *not* run out of memory under normal conditions. This happens to be the same assumption Unix does. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> > And what do you do, then, with the processes that happen to have > legitimate use for more stack? > > Or maybe you just find out how much stack each process uses, and > then set limits appropriate for each one? Which is the equivalent of > setting limits to each user, of course... You get a little program, like eg. Xenix and Minix had, which lets you modify the executable header to indicate how much stack the system should reserve. If the program decides to use more stack for some reason, then it dies; this is in effect "stack overcommit". 8) -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ [EMAIL PROTECTED] \\-- Joseph Merrick \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
: :On Jul 15, 12:20am, "Daniel C. Sobral" wrote: :} "Charles M. Hannum" wrote: :} > :} > That's also objectively false. Most such environments I've had :} > experience with are, in fact, multi-user systems. As you've pointed :} > out yourself, there is no combination of resource limits and whatnot :} > that are guaranteed to prevent `crashing' a multi-user system due to :} > overcommit. My simulation should not be axed because of a bug in :} > someone else's program. (This is also not hypothetical. There was a :} > bug in one version of bash that caused it to consume all the memory it :} > could and then fall over.) :} :} In which case the program that consumed all memory will be killed. :} The program killed is +NOT+ the one demanding memory, it's the one :} with most of it. : : On one system I administrate, the largest process is typically :rpc.nisd (the NIS+ server daemon). Killing that process would be a :bad thing (TM). You're talking about killing random processes. This :is no way to run a system. It is not possible for any arbitrary :decision to always hit the correct process. That is a decision that :must be made by a competent admin. This is the biggest argument :against overcommit: there is no way to gracefully recover from an :out of memory situation, and that makes for an unreliable system. : :}-- End of excerpt from "Daniel C. Sobral" ... and the chance of that system running out of swap space is? The machine has hit the wall, the admin can't login. What is the kernel to do? -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
:> I mean, jeeze, the reservation for the program stack alone would eat :> up all your available swap space! What is a reasonable stack size? The :> system defaults to 8MB. Do we rewrite every program to specify its own :> stack size? How do we account for architectural differences? : :The alternative is to rewrite every program that assumes the semantics :of malloc() are being followed. The problem I have as an applications :writer is that I tend to believe malloc. To pick a specific example, :our IMAP client takes steps to ensure it won't run out of memory in :critical sections. We maintain a "rainy day" pool block of memory. If :... :--lyndon We just put a cap on the number of imap clients we allow running at any given moment... say, a few hundred. Not only does it work just dandy, it also prevents the machine from overloading and gives us a nice "you may want to look into this" alarm. We do the same thing with sendmail, popper, the web server, named, and every other service which can be forked. This also prevents one subsystem from overly interfering with another. For example, if popper saturates it does not overly interfere with imapd operation. The limit is set to around 3x the monday peak load, and sufficient swap is configured to deal with it should the limit be hit. Problem solved. No fancy modifications required. If any of these subsystems actually ever got close to using all available swap, the other subsystems would be up the creek anyway so, really, it doesn't make much sense hacking the source to allow the subsystem to run into the wall anyway. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit
:Ted Faber :>For every strategy there's a counterstrategy. :exactly: the disappointing thing about this whole thread is there's been :little discussion of implementing a (tunable) policy how to handle the :situation when resource shortage materialises. : :Overcommitment can be useful, maybe even for most people... : :>Killing the biggest is simple to implement and usually right. :... but some people don't want that policy, at least on some of their :systems. Does FreeBSD offer alternatives? Is so, they've been conspicuously :absent from discussion, which might have taken things into a more :productive vein. What do other over-committing systems offer? : :Danny Thomas Here's an alternative: whlie (1) sleep 60 blah blah blah run pstat -s, get available swap. if available swap < 200MB then blow away some non-critical processes if no non-critical processes remain blow away everything not owned by root and yell for help. if no no-root processes remain do nothing. let the kernel blow away the biggest process when swap actually runs out. endif endif endif end How long do you suppose it would take to actually write that script? One hour? Two hours? Not long, I think. Problem solved. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:Our IMAP server routinely show a footprint of about 1MB private storage. :This is constant for most operations. However, when you get into doing :SEARCH and SORT, there are certain cases where we need memory, sometimes :a *lot* of memory. : :Your proposal is that my *well behaved* application should be arbitrarily :killed, leaving the client stuck with a) no results and b) no IMAP :connection, in this situation. (And think threaded. That one server :could be handling *hundreds* of clients.) This is preferable to :returning a NULL to the malloc() request, which I can handle :gracefully by simply returning a NO response to the IMAP client? : :What it so evil about having a reasonably intelligent malloc() that :tells the truth, and returns unused memory to the system? Overcommit :is for lazy programmers, plain and simple. At least the SGI documentation :about overcommit admits that (or at least, did at one time). : :--lyndon If you are running an IMAP server that regularly runs out of swap space, you have a configuration problem which needs to be addressed. It's as simple as that. What you are putting forth is an example of something that will never happen on a properly configured server. In regards to the general case where one is running third-party applications. Here you are assuming that you can go in and modify every single piece of software running on the machine to deal with malloc() returning NULL. Because if you don't, the machine isn't going to be very stable. Not only that, you are assuming that you will make the correct decision on what action to take when malloc() *does* return NULL. If you decide to return an error code but not exit, what happens when a potential blowup situation results in thousands of imap processes being run on the system, and NONE of them exit when their malloc() fails? The problem is a whole lot more complex then simply having the OS return NULL from a malloc(). Currently the OS kills processes as a last resort. The idea is that no nominally running system runs out of swap. Now you propose to take away the kernel's ability to recover some memory as a last resort and instead put it into the hands of the very user or root-run processes that are causing the problem in the first place! A much better solution would be to write a simple watchdog script that notices when swap space is low and does the right thing -- e.g. kills the non-essential processes and leaves the essential ones alone. Then the kernel never actually reaches a state of last-resort. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Sergey Babkin wrote: > > > It would be nice to have a way to indicate that, a la SIGDANGER. > > Another option may be to add something like "importance classes". > Suppose we assign an one-byte "importance level" to each process. > When we get out of swap we start killing processes with the lowest > importance level. This seems to be both easy to implement and > a rather robust solution. This is as easy to do as setting limits, which has the added benefit of not having any process killed. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Jason Thorpe wrote: > > If you have a lot of users, all of which have buggy programs which eat > a lot of memory, per-user swap quotas don't necessarily save your butt. The chance of these buggy programs running at the same time is not exactly high... > And maybe the individual programs didn't encounter their resource limits. > > ...but the sheer number of these runaway things caused the overcommit to > be a problem. If malloc() or whatever had actually returned NULL at the > right time (i.e. as backing store was about to become overcommitted), then > these runaway processes would have stopped running away (they would have > gotten a SIGSEGV and died). > > Anyhow, my "lame undergrads" example comes from a time when PCs weren't > really powerful enough for the job (or something; anyhow, we didn't have > any in the department :-). My example is from a Sequent Balance (16 > ns32032 processors, 64M RAM [I think; been a while], 4.2BSD variant). So, tell me... when NetBSD gets it's non-overcommit switch, would you use it in the environment you describe? -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
[EMAIL PROTECTED] wrote: > > What it so evil about having a reasonably intelligent malloc() that > tells the truth, and returns unused memory to the system? Overcommit > is for lazy programmers, plain and simple. At least the SGI documentation > about overcommit admits that (or at least, did at one time). Yes. So is high-level languages, as a matter of fact. True memory-conscious programmers will never use anything besides assembler. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
Jason Thorpe wrote: > > > > ...um, so, make the code that deals with faulting in the stack a bit > smarter. > > > > Uh? Like what? Like overcommitting, for instance? The beauty of > > overcommitting is that either you do it or you don't. :-) > > One option is to special-case overcommit the stack. Another is to > set the default stack limits to something more reasonable on a system > where overcommit is disabled. And what do you do, then, with the processes that happen to have legitimate use for more stack? Or maybe you just find out how much stack each process uses, and then set limits appropriate for each one? Which is the equivalent of setting limits to each user, of course... -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
Robert Elz wrote: > > Note that all this (large) VM I have described was filled with real data > (except for the odd times hen innd or named had just forked), none of it > could be overcommitted and just ignored. Whatever policy was in place, > the physical VM resources would have run out. In a standard Unix system, with standard Unix programs, it is very unlikely that "all this VM" was filled with real data. Take, for instance, the stacks. > Now, with overcommit mode, we get an extra 30 seconds of life, because > no doubt there are a few pages floating around that have been allocated > to some process, but nothing has bothered to write into yet. An extra 30 > seconds if we're lucky (except if we followed the advice given here > earlier which would indicate that only 1/8 the amount of swap space would > be needed, in which case these processes would never have gotten started > in the first place). After that short grace period, during which the Which is what I claim. Have you run it in overcommit mode? Did you actually get just 30 extra seconds? Sure as hell, the AIX systems I ran would have gotten a LOT more than 30 extra seconds going from non-overcommit to overcommit. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
John Nemeth wrote: > > } > But that isn't always the best process to have killed off... > } > } Sure it is. :-) Let's see... > > This statement is absurd. Only a comptetant admin can decide > which process can be killed. No arbitrary decision is going to be > correct. We are talking about what process the OS should kill automatically when it reaches this situation. What is the criteria that should be used? Is the "biggest process" the "best" process to be killed? Or is there another, better criteria? In this context, the statement makes perfect sense, even if you disagree with it. > } interesting, and a good implementation will very probably be > } committed. *BUT*, this is not as useful as it seems. Since the > } correct solution is buy more memory/increase swap (correct solution > } for our target markets, anyway), there is little incentive to > } implement it. > > In case you hadn't noticed, this debate is cross-posted to > NetBSD. NetBSD's target market isn't the same as FreeBSD's target > market. This answer is NOT the correct solution for NetBSD's target > market. Heck, except for one rather vocal person, FreeBSD's target > market may not consider it to be the correct solution either. I most > certainly do not consider it to be correct, and I admin a lot of > mission critical servers. I noticed, but I do not speak for NetBSD. Well, I do not speak for FreeBSD either, but I have well informed opinions on it. What I say, I say about FreeBSD. As for being "correct", it's really simple. Either you have enough memory, or you do not. If you don't have enough memory, a number of programs cannot function correctly. Sure, some programs might be able to deal with low-memory situations, but *other* programs *cannot* deal with it. It's impossible for them to accomplish their tasks if there is not enough memory. So, if you want that server to accomplish it's job, you need more memory. Which, btw, is cheaper than the man-hours needed to implement SIGDANGER. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
John Nemeth wrote: > > The machine in question has run out of swap, due to unforseeable > excessive memory demands. This was accompanied by processes > complaining about not being able to allocate memory and then cleaning > up after themselves. I did not see random processes being killed > because of it. That is the way things should be. From this, I can > assume that the OS doesn't overcommit. In case, you're wondering, the > OS in question is: > > SunOS 5.5 Generic_103093-25 sun4u sparc Uh... like any modern unix, Solaris overcommits. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
John Nemeth wrote: > > } > This is based upon your somewhat strange definition of "work". I assure > } > you that I have run many systems which don't use overcommit, and which I > } > quite frequently run into "out of VM" conditions, and which I can assure > } > you, work just fine. When they're getting to run out of VM, the system > } > is approaching paging death, which is as you'd expect (they're > overloaded). > } > That is, adding more VM (more swap space) would be counterproductive. > } > } Would you care to name such systems? And, btw, a system consuming > } all memory is *not* necessarily approaching paging death. More > } likely, it is just storing a lot of data in the swap which will > } never be used (which is the whole point of overcommit in first > } place), and, thus, never paged in. > > If the data is stored in swap, then it is committed, whether the data > is used or not. Overcommitting only helps with memory that is allocated, > but not used. It's bad enough to have people in this debate that have > something of a clue, refusing to see the other side; we really don't need > people that have no clue at all. Or trollers. Letting it pass that you did not care to name such systems, let's try to make myself clear. You were talking about systems approaching "out of VM" condition, to use your words. This happens when almost all memory, RAM and swap, have been committed to programs, by definition. This can happen with things as simple as C++ objects being initialized, even if they then proceed to stay mainly inactive (C++ array initialization comes to mind). What I'm saying is that active usage of this committed memory is more unlikely than most of it being inactive, which still consumes swap but does not cause paging. > } Fine. Stay with the allocation of swap for DATA/BSS of each instance > } of the same program you are running. That alone is enough. > > Yes. In another post, you brought up the issue of TEXT. TEXT is > swapped in from the executable file and has been for a very long time. It > is simply not an issue. I never said TEXT. I said *CODE*. Like in Megabytes of modifiable Lisp code used by Emacs, the program I mentioned. Some of us have enough experience not to automatically equate code with TEXT, which is a C-ish thingy. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Michael Richardson wrote: > > Ben> Tell me, Mr. Nemeth, has this ever happened to you? Have you ever > Ben> come *close*? > > Uh, since we don't run overcommit, the answer is specifically *NO*. And what system do you run? > I have had it happen on other systems. (Solaris, AIX) It was very > mystifying to diagnose. Sure, the systems were misconfigured for what we > were trying to do, but if I wanted build a custom system for every > application well... I'd be running NT. I have to agree about the mystifying diagnose... Specially when they *don't* page like hell. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Michael Richardson wrote: > > No, I don't agree. > > This is a biggest argument against solving the overcommit situation with > SIGKILL. I have no problem with overcommit as a concept, I have a problem > with being unable to keep my possibly big processes (X, rpc.nisd, > etc. depending on cicumstances) from being victims. It is no more difficult to protect big processes than it is to create user limits. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
John Nemeth wrote: > > On one system I administrate, the largest process is typically > rpc.nisd (the NIS+ server daemon). Killing that process would be a > bad thing (TM). You're talking about killing random processes. This > is no way to run a system. It is not possible for any arbitrary > decision to always hit the correct process. That is a decision that > must be made by a competent admin. This is the biggest argument > against overcommit: there is no way to gracefully recover from an > out of memory situation, and that makes for an unreliable system. If you run out of memory, it is either a misconfigured system, or a runaway program. If a program is runaway, then: 1) It is larger than your typical rpc.nisd. 2) You cannot tell the system a priori to kill it, because you don't know about it (or else, you wouldn't be running it in first place). A system running in overcommit assumes that you have it correctly configured so it will *not* run out of memory under normal conditions. This happens to be the same assumption Unix does. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
: :On Jul 15, 12:20am, "Daniel C. Sobral" wrote: :} "Charles M. Hannum" wrote: :} > :} > That's also objectively false. Most such environments I've had :} > experience with are, in fact, multi-user systems. As you've pointed :} > out yourself, there is no combination of resource limits and whatnot :} > that are guaranteed to prevent `crashing' a multi-user system due to :} > overcommit. My simulation should not be axed because of a bug in :} > someone else's program. (This is also not hypothetical. There was a :} > bug in one version of bash that caused it to consume all the memory it :} > could and then fall over.) :} :} In which case the program that consumed all memory will be killed. :} The program killed is +NOT+ the one demanding memory, it's the one :} with most of it. : : On one system I administrate, the largest process is typically :rpc.nisd (the NIS+ server daemon). Killing that process would be a :bad thing (TM). You're talking about killing random processes. This :is no way to run a system. It is not possible for any arbitrary :decision to always hit the correct process. That is a decision that :must be made by a competent admin. This is the biggest argument :against overcommit: there is no way to gracefully recover from an :out of memory situation, and that makes for an unreliable system. : :}-- End of excerpt from "Daniel C. Sobral" ... and the chance of that system running out of swap space is? The machine has hit the wall, the admin can't login. What is the kernel to do? -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
:> I mean, jeeze, the reservation for the program stack alone would eat :> up all your available swap space! What is a reasonable stack size? The :> system defaults to 8MB. Do we rewrite every program to specify its own :> stack size? How do we account for architectural differences? : :The alternative is to rewrite every program that assumes the semantics :of malloc() are being followed. The problem I have as an applications :writer is that I tend to believe malloc. To pick a specific example, :our IMAP client takes steps to ensure it won't run out of memory in :critical sections. We maintain a "rainy day" pool block of memory. If :... :--lyndon We just put a cap on the number of imap clients we allow running at any given moment... say, a few hundred. Not only does it work just dandy, it also prevents the machine from overloading and gives us a nice "you may want to look into this" alarm. We do the same thing with sendmail, popper, the web server, named, and every other service which can be forked. This also prevents one subsystem from overly interfering with another. For example, if popper saturates it does not overly interfere with imapd operation. The limit is set to around 3x the monday peak load, and sufficient swap is configured to deal with it should the limit be hit. Problem solved. No fancy modifications required. If any of these subsystems actually ever got close to using all available swap, the other subsystems would be up the creek anyway so, really, it doesn't make much sense hacking the source to allow the subsystem to run into the wall anyway. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit
:Ted Faber <[EMAIL PROTECTED]> :>For every strategy there's a counterstrategy. :exactly: the disappointing thing about this whole thread is there's been :little discussion of implementing a (tunable) policy how to handle the :situation when resource shortage materialises. : :Overcommitment can be useful, maybe even for most people... : :>Killing the biggest is simple to implement and usually right. :... but some people don't want that policy, at least on some of their :systems. Does FreeBSD offer alternatives? Is so, they've been conspicuously :absent from discussion, which might have taken things into a more :productive vein. What do other over-committing systems offer? : :Danny Thomas Here's an alternative: whlie (1) sleep 60 blah blah blah run pstat -s, get available swap. if available swap < 200MB then blow away some non-critical processes if no non-critical processes remain blow away everything not owned by root and yell for help. if no no-root processes remain do nothing. let the kernel blow away the biggest process when swap actually runs out. endif endif endif end How long do you suppose it would take to actually write that script? One hour? Two hours? Not long, I think. Problem solved. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:Our IMAP server routinely show a footprint of about 1MB private storage. :This is constant for most operations. However, when you get into doing :SEARCH and SORT, there are certain cases where we need memory, sometimes :a *lot* of memory. : :Your proposal is that my *well behaved* application should be arbitrarily :killed, leaving the client stuck with a) no results and b) no IMAP :connection, in this situation. (And think threaded. That one server :could be handling *hundreds* of clients.) This is preferable to :returning a NULL to the malloc() request, which I can handle :gracefully by simply returning a NO response to the IMAP client? : :What it so evil about having a reasonably intelligent malloc() that :tells the truth, and returns unused memory to the system? Overcommit :is for lazy programmers, plain and simple. At least the SGI documentation :about overcommit admits that (or at least, did at one time). : :--lyndon If you are running an IMAP server that regularly runs out of swap space, you have a configuration problem which needs to be addressed. It's as simple as that. What you are putting forth is an example of something that will never happen on a properly configured server. In regards to the general case where one is running third-party applications. Here you are assuming that you can go in and modify every single piece of software running on the machine to deal with malloc() returning NULL. Because if you don't, the machine isn't going to be very stable. Not only that, you are assuming that you will make the correct decision on what action to take when malloc() *does* return NULL. If you decide to return an error code but not exit, what happens when a potential blowup situation results in thousands of imap processes being run on the system, and NONE of them exit when their malloc() fails? The problem is a whole lot more complex then simply having the OS return NULL from a malloc(). Currently the OS kills processes as a last resort. The idea is that no nominally running system runs out of swap. Now you propose to take away the kernel's ability to recover some memory as a last resort and instead put it into the hands of the very user or root-run processes that are causing the problem in the first place! A much better solution would be to write a simple watchdog script that notices when swap space is low and does the right thing -- e.g. kills the non-essential processes and leaves the essential ones alone. Then the kernel never actually reaches a state of last-resort. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Sergey Babkin wrote: > > > It would be nice to have a way to indicate that, a la SIGDANGER. > > Another option may be to add something like "importance classes". > Suppose we assign an one-byte "importance level" to each process. > When we get out of swap we start killing processes with the lowest > importance level. This seems to be both easy to implement and > a rather robust solution. This is as easy to do as setting limits, which has the added benefit of not having any process killed. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
Jason Thorpe wrote: > > > > ...um, so, make the code that deals with faulting in the stack a bit smarter. > > > > Uh? Like what? Like overcommitting, for instance? The beauty of > > overcommitting is that either you do it or you don't. :-) > > One option is to special-case overcommit the stack. Another is to > set the default stack limits to something more reasonable on a system > where overcommit is disabled. And what do you do, then, with the processes that happen to have legitimate use for more stack? Or maybe you just find out how much stack each process uses, and then set limits appropriate for each one? Which is the equivalent of setting limits to each user, of course... -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
Robert Elz wrote: > > Note that all this (large) VM I have described was filled with real data > (except for the odd times hen innd or named had just forked), none of it > could be overcommitted and just ignored. Whatever policy was in place, > the physical VM resources would have run out. In a standard Unix system, with standard Unix programs, it is very unlikely that "all this VM" was filled with real data. Take, for instance, the stacks. > Now, with overcommit mode, we get an extra 30 seconds of life, because > no doubt there are a few pages floating around that have been allocated > to some process, but nothing has bothered to write into yet. An extra 30 > seconds if we're lucky (except if we followed the advice given here > earlier which would indicate that only 1/8 the amount of swap space would > be needed, in which case these processes would never have gotten started > in the first place). After that short grace period, during which the Which is what I claim. Have you run it in overcommit mode? Did you actually get just 30 extra seconds? Sure as hell, the AIX systems I ran would have gotten a LOT more than 30 extra seconds going from non-overcommit to overcommit. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
John Nemeth wrote: > > } > But that isn't always the best process to have killed off... > } > } Sure it is. :-) Let's see... > > This statement is absurd. Only a comptetant admin can decide > which process can be killed. No arbitrary decision is going to be > correct. We are talking about what process the OS should kill automatically when it reaches this situation. What is the criteria that should be used? Is the "biggest process" the "best" process to be killed? Or is there another, better criteria? In this context, the statement makes perfect sense, even if you disagree with it. > } interesting, and a good implementation will very probably be > } committed. *BUT*, this is not as useful as it seems. Since the > } correct solution is buy more memory/increase swap (correct solution > } for our target markets, anyway), there is little incentive to > } implement it. > > In case you hadn't noticed, this debate is cross-posted to > NetBSD. NetBSD's target market isn't the same as FreeBSD's target > market. This answer is NOT the correct solution for NetBSD's target > market. Heck, except for one rather vocal person, FreeBSD's target > market may not consider it to be the correct solution either. I most > certainly do not consider it to be correct, and I admin a lot of > mission critical servers. I noticed, but I do not speak for NetBSD. Well, I do not speak for FreeBSD either, but I have well informed opinions on it. What I say, I say about FreeBSD. As for being "correct", it's really simple. Either you have enough memory, or you do not. If you don't have enough memory, a number of programs cannot function correctly. Sure, some programs might be able to deal with low-memory situations, but *other* programs *cannot* deal with it. It's impossible for them to accomplish their tasks if there is not enough memory. So, if you want that server to accomplish it's job, you need more memory. Which, btw, is cheaper than the man-hours needed to implement SIGDANGER. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Garance A Drosihn wrote: > > >> One of my main freebsd machines is mainly here to run one > >> process, which is a pretty good-sized process (>40meg). If > >> I did get into a memory-shortage problem, I do *not* want > >> that process killed, I'd want some other processes killed. > > > > Just size your swap so that it becomes unlikely that you run out of > > memory before a process exceeds 40Mb. > > > > And I mean it. > > For the moment I'll pretend that you honestly think that is an > answer, and I'll note that the very same machine may have well > over 100 processes each of which takes 1-2 meg of memory. If > the machine hits a really-out-of-memory error, I would be much > much happier to see all 100+ of those processes killed, at once, > than the one 40-meg process. > > Now tell me how I fix my swap under those circumstances. If > the answer is "buy infinite memory (ram or disk)", then we don't > need any overcommit policy in the first place. Note that the > problem might be that these 100 processes start taking up 5 or > 10 meg than the 2 meg I'm used to. As a matter of fact, I honestly think that's an answer. If 100 processes start taking 5 or 10 meg, there is something mightly wrong. It *won't* happen. A hostile attack may happen, which can be dealt with limits. Running out of memory is something that happens when you get runaway processes, when you are under attack, or when you have a badly configured system. The first two cases are dealt with limits. _IF_ you have trusted accounts, they better be *trusted* accounts. These may have runaway processes. So, you have to set up the swap to take this into account. This is a matter of *one* process suddenly spiking memory usage. 100 processes suddenly spiking memory usage just *doesn't* happen. If it can happen in normal usage, then, yes, you have to configure swap for that. It's normal usage, after all. Infinite memory? Of course not. Just be realistic. > I once participated in a discussion on this very list where people > would discuss SIGDANGER with some degree of intelligence, instead > of offhand smart-aleck remarks. That discussion (as I remember) > ended with the conclusion that "SIGDANGER can be useful, but there's > a fair amount of work to do first". My comment was actually meant as > a follow-up to that earlier discussion, just to see if anything had > happened to get us closer to this. I made no offhand smart-aleck remarks. Everything I said to you I meant seriously. I was explaining why, imo, no one was answering to repeated mentions of SIGDANGER. -- Daniel C. Sobral(8-DCS) d...@newsguy.com d...@freebsd.org "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
John Nemeth wrote: > > } > This is based upon your somewhat strange definition of "work". I assure > } > you that I have run many systems which don't use overcommit, and which I > } > quite frequently run into "out of VM" conditions, and which I can assure > } > you, work just fine. When they're getting to run out of VM, the system > } > is approaching paging death, which is as you'd expect (they're overloaded). > } > That is, adding more VM (more swap space) would be counterproductive. > } > } Would you care to name such systems? And, btw, a system consuming > } all memory is *not* necessarily approaching paging death. More > } likely, it is just storing a lot of data in the swap which will > } never be used (which is the whole point of overcommit in first > } place), and, thus, never paged in. > > If the data is stored in swap, then it is committed, whether the data > is used or not. Overcommitting only helps with memory that is allocated, > but not used. It's bad enough to have people in this debate that have > something of a clue, refusing to see the other side; we really don't need > people that have no clue at all. Or trollers. Letting it pass that you did not care to name such systems, let's try to make myself clear. You were talking about systems approaching "out of VM" condition, to use your words. This happens when almost all memory, RAM and swap, have been committed to programs, by definition. This can happen with things as simple as C++ objects being initialized, even if they then proceed to stay mainly inactive (C++ array initialization comes to mind). What I'm saying is that active usage of this committed memory is more unlikely than most of it being inactive, which still consumes swap but does not cause paging. > } Fine. Stay with the allocation of swap for DATA/BSS of each instance > } of the same program you are running. That alone is enough. > > Yes. In another post, you brought up the issue of TEXT. TEXT is > swapped in from the executable file and has been for a very long time. It > is simply not an issue. I never said TEXT. I said *CODE*. Like in Megabytes of modifiable Lisp code used by Emacs, the program I mentioned. Some of us have enough experience not to automatically equate code with TEXT, which is a C-ish thingy. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
gdb instead of adb
Is the reason why adb hasn't been ported to freebsd because the source is proprietary? If gdb should suffice for my debugging needs, how can a breakpoint be set at a particular interrupt, or even at any interrupt? The break command only seems to accept functions, offsets, linenumbers and addresses... I'm all out of ideas and the gdb info file isn't helping. Thanks in advance, Marc To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit
lyn...@orthanc.ab.ca wrote: > The semantics of malloc() have been defined since almost the dawn of > time. From the current manpage: > RETURN VALUES > The malloc() and calloc() functions return a pointer to the allocated > memory if successful; otherwise a NULL pointer is returned. > Nowhere does it say that allocated memory might not exist. Nowhere > does it say that I have to touch all the allocated pages to make > sure they are really there. Nowhere does it say process death at > some non-deterministic time in the future might be a side effect > of calling malloc(). It's just using a different definition of "successful return of malloc()" to the one you're trying to use :-) - mark Mark Newton Email: new...@internode.com.au (W) Network Engineer Email: new...@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82232999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223 To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Garance A Drosihn wrote: > > >> One of my main freebsd machines is mainly here to run one > >> process, which is a pretty good-sized process (>40meg). If > >> I did get into a memory-shortage problem, I do *not* want > >> that process killed, I'd want some other processes killed. > > > > Just size your swap so that it becomes unlikely that you run out of > > memory before a process exceeds 40Mb. > > > > And I mean it. > > For the moment I'll pretend that you honestly think that is an > answer, and I'll note that the very same machine may have well > over 100 processes each of which takes 1-2 meg of memory. If > the machine hits a really-out-of-memory error, I would be much > much happier to see all 100+ of those processes killed, at once, > than the one 40-meg process. > > Now tell me how I fix my swap under those circumstances. If > the answer is "buy infinite memory (ram or disk)", then we don't > need any overcommit policy in the first place. Note that the > problem might be that these 100 processes start taking up 5 or > 10 meg than the 2 meg I'm used to. As a matter of fact, I honestly think that's an answer. If 100 processes start taking 5 or 10 meg, there is something mightly wrong. It *won't* happen. A hostile attack may happen, which can be dealt with limits. Running out of memory is something that happens when you get runaway processes, when you are under attack, or when you have a badly configured system. The first two cases are dealt with limits. _IF_ you have trusted accounts, they better be *trusted* accounts. These may have runaway processes. So, you have to set up the swap to take this into account. This is a matter of *one* process suddenly spiking memory usage. 100 processes suddenly spiking memory usage just *doesn't* happen. If it can happen in normal usage, then, yes, you have to configure swap for that. It's normal usage, after all. Infinite memory? Of course not. Just be realistic. > I once participated in a discussion on this very list where people > would discuss SIGDANGER with some degree of intelligence, instead > of offhand smart-aleck remarks. That discussion (as I remember) > ended with the conclusion that "SIGDANGER can be useful, but there's > a fair amount of work to do first". My comment was actually meant as > a follow-up to that earlier discussion, just to see if anything had > happened to get us closer to this. I made no offhand smart-aleck remarks. Everything I said to you I meant seriously. I was explaining why, imo, no one was answering to repeated mentions of SIGDANGER. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] "Would you like to go out with me?" "I'd love to." "Oh, well, n... err... would you?... ahh... huh... what do I do next?" To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: changing argv[0] after fork()
In message Wayne Cuddy writes: : Even though I am developing on FBSD is there a "more portable" way to do this? No. Well, not short of execing. Warner To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
gdb instead of adb
Is the reason why adb hasn't been ported to freebsd because the source is proprietary? If gdb should suffice for my debugging needs, how can a breakpoint be set at a particular interrupt, or even at any interrupt? The break command only seems to accept functions, offsets, linenumbers and addresses... I'm all out of ideas and the gdb info file isn't helping. Thanks in advance, Marc To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:For the moment I'll pretend that you honestly think that is an :answer, and I'll note that the very same machine may have well :over 100 processes each of which takes 1-2 meg of memory. If :the machine hits a really-out-of-memory error, I would be much :much happier to see all 100+ of those processes killed, at once, :than the one 40-meg process. : :Now tell me how I fix my swap under those circumstances. If :the answer is "buy infinite memory (ram or disk)", then we don't :need any overcommit policy in the first place. Note that the :problem might be that these 100 processes start taking up 5 or :10 meg than the 2 meg I'm used to. Everything scales. If the load on your machine is such that you have hundreds of processes taking 1-2MB of memory, then lets assume that such a machine has a reasonable memory configuration of, say, 256MB of ram, and a reasonable swap configuration of, say, 1GB. Under normal operating conditions perhaps 100MB might be swapped out, giving you 900MB of margin. The actual VM footprint on such a machine might run on the order of 10 GB (rough guess) of which 350MB or so has actually been allocated). With 900MB of margin - which I might add is only about $30 worth of disk space, and reasonable process limits, it seems highly unlikely that the machine will ever run out of swap, even if a user makes an honest mistake. I also rather seriously doubt that a hostile user would have any more or less success blowing away your process with the non-overcommit model verses otherwise. If 1G isn't enough, spend another $30 and throw 2G of swap online. Or perhaps dedicate an entire $150 disk and throw 6+ GB of swap online. The equivalent setup using a non-overcommit model would require considerably more swap to have the same reliability. Plus you have to realize that with either model if you are talking about saving your work, the same code that does the save-and-exit in the non-overcommit model can just as easily do a checkpoint once an hour in the standard overcommit model. Code that can't save/checkpoint would not survive either model. Disk is cheap. Memory isn't (though it's getting better). Everything scales. :I didn't mean to be casting asperisions on the general idea of :overcommitting, or whatever it is that has your shorts all tied :up in a knot. : :--- :Garance Alistair Drosehn = g...@eclipse.acs.rpi.edu :Senior Systems Programmer or dro...@rpi.edu -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit
[EMAIL PROTECTED] wrote: > The semantics of malloc() have been defined since almost the dawn of > time. From the current manpage: > RETURN VALUES > The malloc() and calloc() functions return a pointer to the allocated > memory if successful; otherwise a NULL pointer is returned. > Nowhere does it say that allocated memory might not exist. Nowhere > does it say that I have to touch all the allocated pages to make > sure they are really there. Nowhere does it say process death at > some non-deterministic time in the future might be a side effect > of calling malloc(). It's just using a different definition of "successful return of malloc()" to the one you're trying to use :-) - mark Mark Newton Email: [EMAIL PROTECTED] (W) Network Engineer Email: [EMAIL PROTECTED] (H) Internode Systems Pty Ltd Desk: +61-8-82232999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: a BSD identd
On Thu, 15 Jul 1999, Harold Gutch wrote: > On Tue, Jul 13, 1999 at 11:47:33PM -0400, Brian F. Feldman wrote: > > We don't _need_ pidentd anymore. It will load down a system more than > > the inetd's implementation of ident will. Therefore, pidentd should be > > phased out. Other than that, pidentd should be using > > http://www.FreeBSD.org/~green/freebsd4.c and not linking with libkvm. > > > pidentd supports DES-encrypted tokens as replies instead of the > plaintext usernames. As long as your identd (or any other > replacement) does not support this, there is still valid use for > pidentd. And pidentd will still be supported. Eventually, I'd like to have those huge majority who do not use DES tokens with pidentd move to the inetd identd (when committed)... > > bye, > Harold > > -- > Sleep is an abstinence syndrome wich occurs due to lack of caffein. > Wed Mar 4 04:53:33 CET 1998 #unix, ircnet > Brian Fundakowski Feldman _ __ ___ ___ ___ ___ gr...@freebsd.org _ __ ___ | _ ) __| \ FreeBSD: The Power to Serve!_ __ | _ \._ \ |) | http://www.FreeBSD.org/ _ |___/___/___/ To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: changing argv[0] after fork()
In message <[EMAIL PROTECTED]> Wayne Cuddy writes: : Even though I am developing on FBSD is there a "more portable" way to do this? No. Well, not short of execing. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
At 3:18 PM -0700 7/14/99, Matthew Dillon wrote: >This conversation is getting silly. Do you actually believe >that an operating system can magically protect itself 100% >from armloads of hostile users? > >Give me a break. You people are crazy. If you have something >worthwhile to say i'll listen, but these "the sky is falling!" >arguments are idiotic. Hmm. I didn't notice any sky-is-falling arguments in this thread, so I finally started looking around to see why such nasty replies keep showing up to what I considered reasonable questions... So, I finally looked back into the "replacement for grep" thread (which I have been ignoring ever since it stopped talking about the grep replacement), and I see this topic is being thrashed to death over there. I still think there could be some useful discussion on what I was *trying* to talk about here, but I guess it will have to wait until some other time given how exasperated people are getting. --- Garance Alistair Drosehn = g...@eclipse.acs.rpi.edu Senior Systems Programmer or dro...@rpi.edu Rensselaer Polytechnic Institute To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:For the moment I'll pretend that you honestly think that is an :answer, and I'll note that the very same machine may have well :over 100 processes each of which takes 1-2 meg of memory. If :the machine hits a really-out-of-memory error, I would be much :much happier to see all 100+ of those processes killed, at once, :than the one 40-meg process. : :Now tell me how I fix my swap under those circumstances. If :the answer is "buy infinite memory (ram or disk)", then we don't :need any overcommit policy in the first place. Note that the :problem might be that these 100 processes start taking up 5 or :10 meg than the 2 meg I'm used to. Everything scales. If the load on your machine is such that you have hundreds of processes taking 1-2MB of memory, then lets assume that such a machine has a reasonable memory configuration of, say, 256MB of ram, and a reasonable swap configuration of, say, 1GB. Under normal operating conditions perhaps 100MB might be swapped out, giving you 900MB of margin. The actual VM footprint on such a machine might run on the order of 10 GB (rough guess) of which 350MB or so has actually been allocated). With 900MB of margin - which I might add is only about $30 worth of disk space, and reasonable process limits, it seems highly unlikely that the machine will ever run out of swap, even if a user makes an honest mistake. I also rather seriously doubt that a hostile user would have any more or less success blowing away your process with the non-overcommit model verses otherwise. If 1G isn't enough, spend another $30 and throw 2G of swap online. Or perhaps dedicate an entire $150 disk and throw 6+ GB of swap online. The equivalent setup using a non-overcommit model would require considerably more swap to have the same reliability. Plus you have to realize that with either model if you are talking about saving your work, the same code that does the save-and-exit in the non-overcommit model can just as easily do a checkpoint once an hour in the standard overcommit model. Code that can't save/checkpoint would not survive either model. Disk is cheap. Memory isn't (though it's getting better). Everything scales. :I didn't mean to be casting asperisions on the general idea of :overcommitting, or whatever it is that has your shorts all tied :up in a knot. : :--- :Garance Alistair Drosehn = [EMAIL PROTECTED] :Senior Systems Programmer or [EMAIL PROTECTED] -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: a BSD identd
On Thu, 15 Jul 1999, Harold Gutch wrote: > On Tue, Jul 13, 1999 at 11:47:33PM -0400, Brian F. Feldman wrote: > > We don't _need_ pidentd anymore. It will load down a system more than > > the inetd's implementation of ident will. Therefore, pidentd should be > > phased out. Other than that, pidentd should be using > > http://www.FreeBSD.org/~green/freebsd4.c and not linking with libkvm. > > > pidentd supports DES-encrypted tokens as replies instead of the > plaintext usernames. As long as your identd (or any other > replacement) does not support this, there is still valid use for > pidentd. And pidentd will still be supported. Eventually, I'd like to have those huge majority who do not use DES tokens with pidentd move to the inetd identd (when committed)... > > bye, > Harold > > -- > Sleep is an abstinence syndrome wich occurs due to lack of caffein. > Wed Mar 4 04:53:33 CET 1998 #unix, ircnet > Brian Fundakowski Feldman _ __ ___ ___ ___ ___ [EMAIL PROTECTED] _ __ ___ | _ ) __| \ FreeBSD: The Power to Serve!_ __ | _ \._ \ |) | http://www.FreeBSD.org/ _ |___/___/___/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Garance A Drosihn wrote: > > At 12:20 AM +0900 7/15/99, Daniel C. Sobral wrote: > > In which case the program that consumed all memory will be killed. > > The program killed is +NOT+ the one demanding memory, it's the one > > with most of it. > > But that isn't always the best process to have killed off... > > One of my main freebsd machines is mainly here to run one > process, which is a pretty good-sized process (>40meg). If > I did get into a memory-shortage problem, I do *not* want > that process killed, I'd want some other processes killed. > > It would be nice to have a way to indicate that, a la SIGDANGER. Another option may be to add something like "importance classes". Suppose we assign an one-byte "importance level" to each process. When we get out of swap we start killing processes with the lowest importance level. This seems to be both easy to implement and a rather robust solution. It can be extended to more flexible policies: say, we divide this 8-bit number into two 4-bit fields. The high field will be "major importance level" the low field will be "importance sublevel". We permit the user processes to change their sublevel to any value as long their major level stays the same or becomes lower. This will allow the users to make differences between their programs but only in certain limits. The initial importance level may be set in /etc/login.conf. One more extension would be to use one bit as the ihneritance flag: if it is set, the child inherits the importance value from the parent. But if it's reset the child inherits its major level from the parent but the sublevel gets reset to 0. It may be useful for transparent calls of system(). Yet another extension would be to use two separate inheritance bits: one as described above, the secone one if reset means that the importance value of the child must be reset to the lowest one. -SB To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
At 3:18 PM -0700 7/14/99, Matthew Dillon wrote: >This conversation is getting silly. Do you actually believe >that an operating system can magically protect itself 100% >from armloads of hostile users? > >Give me a break. You people are crazy. If you have something >worthwhile to say i'll listen, but these "the sky is falling!" >arguments are idiotic. Hmm. I didn't notice any sky-is-falling arguments in this thread, so I finally started looking around to see why such nasty replies keep showing up to what I considered reasonable questions... So, I finally looked back into the "replacement for grep" thread (which I have been ignoring ever since it stopped talking about the grep replacement), and I see this topic is being thrashed to death over there. I still think there could be some useful discussion on what I was *trying* to talk about here, but I guess it will have to wait until some other time given how exasperated people are getting. --- Garance Alistair Drosehn = [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Institute To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On 1999/07/14 at 11:17pm +0900, "Daniel C. Sobral" wrote: >Ladavac Marino wrote: >> >> This topic has been trashed to death a few months ago. There is no >> win-win situation in presence of processes which allocate a lot of memory >> without actually using it (read: your typical FORTRAN library). > >This is not about just Fortran libraries. OK, I'll bite. I'm curious as to why Fortran's library has been used as an example here. Some years ago I worked on a Fortran compiler; I wrote its library as well as its front end. The targets were (then) high-performance graphics cards, with little memory, no MMU, no swap, and certainly no sparse objects in the intrinsics. (Are you referring to user libraries that try to provide 'large enough' fixed-size arrays? One can't stop people from doing silly things. One *can* try to stop these silly things from interfering with non-silly things.) This thread originated with C's library, however, not with Fortran's -- in particular with malloc(), which is defined by the ANSI standard to either return NULL, or usable memory. Killing the program later for trying to use successfully malloc()'d memory (what the standard would call 'undefined behavior') isn't an option. If a program that calls only standard C functions can die when it uses malloc()'d memory then, as I read the document, the system does not support standard C. In standard C, malloc() returns something functionally equivalent to committed memory, or it returns NULL. On a system that has a writable file system, a standard-conforming version of malloc() could presumably be written by having malloc() create a file, make it non-sparse, and mmap() it. This seems unattractive. >Imagine a reasonably big >program, like Netscape or Emacs, of which you usually just use a >subset of features. There can easily be many megabytes of code and >data in them you never actually use, or you don't _usually_ use >(like the people who use emacs like it was vi :). Without >overcommit, you need to allocate all that memory for the code, no >matter whether you end up using it or not. With overcommit, there is >no such problem. Code, static data, and not-yet-written writable data should be backed by the executable file, not by swap space, so unused code and tables should not be a problem. >And that's not all, either. > >Hell, people, if overcommit was not useful, everybody wouldn't be >doing it. If a program wants uncommitted space, it should request it by means beyond the standard C library functions (e.g. mmap() with MAP_NORESERVE). A program that does so can then deal with the consequences (e.g. a signal when unavailable space is touched), or die, as the case may be. Innocent bystanders should not be hit. (Of course, programs that use standard C functions like malloc() should be prepared to deal with the consequences of *that*, but the consequences should be the ones defined by the C standard, i.e. a NULL return from malloc().) Stack is more interesting. There might be a place for a global overcommit switch. I think I'd be happier with a scheme in which stack the first page or first few pages are committed (so that reasonable programs will never run into trouble) and remaining stack is over-/un-committed by default, along with means for unusual programs to commit (and/or test commitability of) subsequent pages. -- Kevin Schoedel "Sorry, Opus. I only have NetBSD on my computer." "Darn it. There *has* to be a place where I can use some bad software." -- 'User Friendly' 99/06/14 To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
misc/12633: CMI8330 chip based integrated sound card produce no output
Hi, I have some minor probem with my CMI8330 chip based integrated sound card on m726 motherboard. Here's the URL where you can find how to fix it: http://www.cslab.vt.edu/~jobaldwi/cmi8330init.tar.gz = cut here = Thank you very much for your problem report. It has the internal identification `misc/12633'. The individual assigned to look at your report is: freebsd-bugs. >Category: misc >Responsible:freebsd-bugs >Synopsis: CMI8330 chip based integrated sound card produce no output >Arrival-Date: Tue Jul 13 15:30:01 PDT 1999 end === To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> :Returning NULL isn't an error, it's an indication that there is no more > :memory. Don't think if it as an error, think of it as a hint. > > It's only a hint if it is returned due to the process resource limit > being hit. If it is returned due to the system running out of swap, > it would be an error. Not necessarily. > This creates a dependancy between two completely independant processes > possibly owned two totally unrelated users. This is not acceptable, > and is precisely why this sort of behavior does not occur. In embedded systems, there are no such things as 'independant' processes that are given unlimited access to the system. Nate To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :Returning NULL isn't an error, it's an indication that there is no more :memory. Don't think if it as an error, think of it as a hint. It's only a hint if it is returned due to the process resource limit being hit. If it is returned due to the system running out of swap, it would be an error. Look at it this way: Processes run in their own VM address space and should theoretically be unaffected by other processes. If your resource limits allow a process, say, 32MB of ram, the process should be confident in its ability to utilize that much if it needs to without running out of memory. This is what FreeBSD and most UNIX's give to processes. But if you change the rules so that process A can run the system out of swap (or, in the case of a non-overcommit model, out of reservable space), then process B can get allocation errors without even coming close to their allocation limit. This creates a dependancy between two completely independant processes possibly owned two totally unrelated users. This is not acceptable, and is precisely why this sort of behavior does not occur. -Matt To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
Garance A Drosihn wrote: > > At 12:20 AM +0900 7/15/99, Daniel C. Sobral wrote: > > In which case the program that consumed all memory will be killed. > > The program killed is +NOT+ the one demanding memory, it's the one > > with most of it. > > But that isn't always the best process to have killed off... > > One of my main freebsd machines is mainly here to run one > process, which is a pretty good-sized process (>40meg). If > I did get into a memory-shortage problem, I do *not* want > that process killed, I'd want some other processes killed. > > It would be nice to have a way to indicate that, a la SIGDANGER. Another option may be to add something like "importance classes". Suppose we assign an one-byte "importance level" to each process. When we get out of swap we start killing processes with the lowest importance level. This seems to be both easy to implement and a rather robust solution. It can be extended to more flexible policies: say, we divide this 8-bit number into two 4-bit fields. The high field will be "major importance level" the low field will be "importance sublevel". We permit the user processes to change their sublevel to any value as long their major level stays the same or becomes lower. This will allow the users to make differences between their programs but only in certain limits. The initial importance level may be set in /etc/login.conf. One more extension would be to use one bit as the ihneritance flag: if it is set, the child inherits the importance value from the parent. But if it's reset the child inherits its major level from the parent but the sublevel gets reset to 0. It may be useful for transparent calls of system(). Yet another extension would be to use two separate inheritance bits: one as described above, the secone one if reset means that the importance value of the child must be reset to the lowest one. -SB To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> :Most of the work we've done wouldn't allow this, especially if we were > :using an OS like FreeBSD with a fairly long bootup time. Especially if > :it can be avoided. > : > :Yes, we could (and did) do our own memory management, but it seems to me > :that the kernel has alot more information available to it and would do > :it better than I could. Then again, maybe I'm totally confused about > :how the VM system 'does its thing', and in reality it's no better at it > :than our code, just alot more complex for no reason. :) :) :) > : > > The kernel just isn't the best place to do this. The level of > sophistication required to satisfy the larger set of programming > is much greater then anything discussed here. The last thing that the > kernel should be doing is returning NULL from malloc() to a program > which is operating within its specifications. You and I disagree on this statement. Returning NULL is completely acceptable, and is the 'specification' for many OS's, including FreeBSD if you set user/process limits. > That doesn't help anyone, least of all the programmer who now has > to deal not only with the complexity of the project he is working > on, but must also deal with the potential that the OS will step in > at any time and give him an error that he must deal with in a > sophisticated fashion even when his software is operating > properly. Returning NULL isn't an error, it's an indication that there is no more memory. Don't think if it as an error, think of it as a hint. General purpose computing tends to make one think that out-of-memory condition is an un-acceptable situation, and the program must exit. FreeBSD exacerbates this by rarely returning NULL and randomly killing off processes which may/may not be involved in the memory fracus. > > The same goes for non-embedded work. Why is it that programs generally > exit when they encounter a memory allocation failure? Because programmer's are lazy. Really, and I do it all the time as well. But, that's because on 'general purpose' computing hardware, re-starting the process or having to wait for a reboot is 'acceptable'. But, to many of my collegues who are running simulations that are running for DAYS and WEEKS, they write code that 'saves the state' of the system when they can't get critical memory, *THEN* abort the application. Yes, it's possible that the process of saving state *may* not work if the system is *really* low on memory, but then again it may work. But, more times than not it *does* (appear) to work, and the work is not lost. > with memory allocation errors" provides no solution. 98% of the source > code in the BSD code base will exit if a memory allocation fails, and > I don't know anyone who wants to go through and somehow "fix" it all > (and one could argue what the "fix" should be when something like grep or > a shell has a memory allocation failure). To require that this code be > made twice as sophisticated as it is already in order to deal with a > non-overcommit model reliably is a fantasy. Who said anything about using the bsd code base in an embedded system? Also, you *can* use limits for that stuff. Nate To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :Most of the work we've done wouldn't allow this, especially if we were :using an OS like FreeBSD with a fairly long bootup time. Especially if :it can be avoided. : :Yes, we could (and did) do our own memory management, but it seems to me :that the kernel has alot more information available to it and would do :it better than I could. Then again, maybe I'm totally confused about :how the VM system 'does its thing', and in reality it's no better at it :than our code, just alot more complex for no reason. :) :) :) : :Nate The kernel just isn't the best place to do this. The level of sophistication required to satisfy the larger set of programming is much greater then anything discussed here. The last thing that the kernel should be doing is returning NULL from malloc() to a program which is operating within its specifications. That doesn't help anyone, least of all the programmer who now has to deal not only with the complexity of the project he is working on, but must also deal with the potential that the OS will step in at any time and give him an error that he must deal with in a sophisticated fashion even when his software is operating properly. The same goes for non-embedded work. Why is it that programs generally exit when they encounter a memory allocation failure? Because it is too damn hard to ensure proper operation at every instance where a program's allocation may fail, especially if the program getting the failure is not responsible for the problem. The glib statement that "programs should deal with memory allocation errors" provides no solution. 98% of the source code in the BSD code base will exit if a memory allocation fails, and I don't know anyone who wants to go through and somehow "fix" it all (and one could argue what the "fix" should be when something like grep or a shell has a memory allocation failure). To require that this code be made twice as sophisticated as it is already in order to deal with a non-overcommit model reliably is a fantasy. Due to the side effects it causes, to put the capability in the kernel in this situation will make things even more unstable then the kernel's current operation. A situation made much worse by the magnitude of the resources necessary to bring such a system up to the same performance levels attainable by a standard overcommit system. The only case where this does not occur is also the case where the programmer can fairly easily manage the memory to avoid the overcommit from occuring in the first place. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: tcp windowsize query?
> In article > you write: > >>delayed ack sounds interesting > > > >Turning that off disables TCP slow-start. It's a huge performance > >booster for things like SMB service, where you have lots of short-lived > >TCP connections on a local net. > > Mike probably already knows this, but just a general clarification: > > No, it does _not_ turn off slow-start. Normally a TCP implementation > will delay sending an ACK to the peer until the TCP fasttimer expires, > or certain other conditions are met. The goal here is to amortize an > ACK over several incoming data packets. Yeah, brain fart. Slow start != delayed ACK. Blah. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ msm...@freebsd.org \\-- Joseph Merrick \\ msm...@cdrom.com To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> :> > Quite true. In the embedded world we preallocate memory and shape > :> > the programs to what is available in the system. But if we run out > :> > of memory we usually panic and reboot - because the code is designed > :> > to NOT run out of memory and thus running out of memory is a > catastrophic > :> > situation. > : > :*ACK* This is unacceptable in many 'embedded' systems. > > Don't confuse a watchdog panic from other conditions. If the embedded > system software is supposed to deal with a low-memory condition and can't, > the failsafe is all that's left between it and infinity. > > The statement that the kernel's overcommit methodology somehow prevents > one from being able to build embedded systems on top of it is just plain > incorrect. The embedded system is perfectly capable of implementing its > own memory management to avoid the filesafe provided by the kernel. It is. But, it's not the most effecient/effective way of doing it. Sometimes it's best to let the 'kernel' do the things it does best, and let the users worry about the things they do best. The kernel manages memory alot more effeciently than any userland process can do, so why not let it? > Most of the embedded work I've done -- mainly remote telemetry > units running with flash and a megabyte or so of ram -- panic and > reboot if they run out of memory. Most of the work we've done wouldn't allow this, especially if we were using an OS like FreeBSD with a fairly long bootup time. Especially if it can be avoided. Yes, we could (and did) do our own memory management, but it seems to me that the kernel has alot more information available to it and would do it better than I could. Then again, maybe I'm totally confused about how the VM system 'does its thing', and in reality it's no better at it than our code, just alot more complex for no reason. :) :) :) Nate To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On 1999/07/14 at 11:17pm +0900, "Daniel C. Sobral" <[EMAIL PROTECTED]> wrote: >Ladavac Marino wrote: >> >> This topic has been trashed to death a few months ago. There is no >> win-win situation in presence of processes which allocate a lot of memory >> without actually using it (read: your typical FORTRAN library). > >This is not about just Fortran libraries. OK, I'll bite. I'm curious as to why Fortran's library has been used as an example here. Some years ago I worked on a Fortran compiler; I wrote its library as well as its front end. The targets were (then) high-performance graphics cards, with little memory, no MMU, no swap, and certainly no sparse objects in the intrinsics. (Are you referring to user libraries that try to provide 'large enough' fixed-size arrays? One can't stop people from doing silly things. One *can* try to stop these silly things from interfering with non-silly things.) This thread originated with C's library, however, not with Fortran's -- in particular with malloc(), which is defined by the ANSI standard to either return NULL, or usable memory. Killing the program later for trying to use successfully malloc()'d memory (what the standard would call 'undefined behavior') isn't an option. If a program that calls only standard C functions can die when it uses malloc()'d memory then, as I read the document, the system does not support standard C. In standard C, malloc() returns something functionally equivalent to committed memory, or it returns NULL. On a system that has a writable file system, a standard-conforming version of malloc() could presumably be written by having malloc() create a file, make it non-sparse, and mmap() it. This seems unattractive. >Imagine a reasonably big >program, like Netscape or Emacs, of which you usually just use a >subset of features. There can easily be many megabytes of code and >data in them you never actually use, or you don't _usually_ use >(like the people who use emacs like it was vi :). Without >overcommit, you need to allocate all that memory for the code, no >matter whether you end up using it or not. With overcommit, there is >no such problem. Code, static data, and not-yet-written writable data should be backed by the executable file, not by swap space, so unused code and tables should not be a problem. >And that's not all, either. > >Hell, people, if overcommit was not useful, everybody wouldn't be >doing it. If a program wants uncommitted space, it should request it by means beyond the standard C library functions (e.g. mmap() with MAP_NORESERVE). A program that does so can then deal with the consequences (e.g. a signal when unavailable space is touched), or die, as the case may be. Innocent bystanders should not be hit. (Of course, programs that use standard C functions like malloc() should be prepared to deal with the consequences of *that*, but the consequences should be the ones defined by the C standard, i.e. a NULL return from malloc().) Stack is more interesting. There might be a place for a global overcommit switch. I think I'd be happier with a scheme in which stack the first page or first few pages are committed (so that reasonable programs will never run into trouble) and remaining stack is over-/un-committed by default, along with means for unusual programs to commit (and/or test commitability of) subsequent pages. -- Kevin Schoedel <[EMAIL PROTECTED]> "Sorry, Opus. I only have NetBSD on my computer." "Darn it. There *has* to be a place where I can use some bad software." -- 'User Friendly' 99/06/14 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
:> > :> > Quite true. In the embedded world we preallocate memory and shape :> > the programs to what is available in the system. But if we run out :> > of memory we usually panic and reboot - because the code is designed :> > to NOT run out of memory and thus running out of memory is a catastrophic :> > situation. : :*ACK* This is unacceptable in many 'embedded' systems. Don't confuse a watchdog panic from other conditions. If the embedded system software is supposed to deal with a low-memory condition and can't, the failsafe is all that's left between it and infinity. The statement that the kernel's overcommit methodology somehow prevents one from being able to build embedded systems on top of it is just plain incorrect. The embedded system is perfectly capable of implementing its own memory management to avoid the filesafe provided by the kernel. Most of the embedded work I've done -- mainly remote telemetry units running with flash and a megabyte or so of ram -- panic and reboot if they run out of memory. I have several dozen units in the field each keeping track of several thousand data points on 2 minute intervals which have not ever crashed. The only time we reboot them is when we need to upgrade the OS core. The last time was 4 years ago. *These* units will panic and reboot if they run out of memory because the software is designed not to. It is as simple as that. -Matt To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: docs/12377: doc patch for login_cap.
On Fri, Jul 09, 1999 at 11:39:38AM +0200, Sheldon Hearn wrote: > On Thu, 08 Jul 1999 20:59:58 +0100, Nik Clayton wrote: > > With that in mind, how about this patch (in conjunction with the patch to > > login.conf in the original PR, which just updates a comment)? > > This looks much better. :-) Committing it now. N -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <37514...@cs.colorado.edu> To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :One option is to special-case overcommit the stack. Another is to :set the default stack limits to something more reasonable on a system :where overcommit is disabled. : :-- Jason R. Thorpe Try setting all the resource limits to something reasonable on general principles. It would work as well in an overcommit system as it would in a non-overcommit system. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
[ Trimmed CC list a bit ] > > :* even if you are not willing to pay that price, there _are_ people > > :who are quite willing to pay that price to get the benefits that they > > :see (whether it's a matter of perception or not, from their > > :perspective they may as well be real) of such a scheme. > > > > Quite true. In the embedded world we preallocate memory and shape > > the programs to what is available in the system. But if we run out > > of memory we usually panic and reboot - because the code is designed > > to NOT run out of memory and thus running out of memory is a > > catastrophic > > situation. *ACK* This is unacceptable in many 'embedded' systems. > There's a whole spectrum of embedded devices, and applications that > run on them. That definition works for some of them, but definitely > not all. > Totally agreed. A previous poster brought up the fact that *some* embedded systems are built to deal with 'out of memory' situations, and that the 'total' amount of memory used in the system can be used by other parts of the system. For performance reasons, a particular application may choose to 'cache' data, but in low memory situation it can 'free' up alot of memory. You don't want to put hard-coded limits the process simply because if the memory is there you want it to be able to use it, but you *certainly* don't want to go through a reboot just to get memory back. [ And, I don't want to write my own OS to do this for me. :) ] (However, I agree that for general purpose computing, over-commit is the way to go. But, *BSD is not just for general purpose computing, although that is it's primary market.) Nate To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: a BSD identd
On Tue, Jul 13, 1999 at 11:47:33PM -0400, Brian F. Feldman wrote: > We don't _need_ pidentd anymore. It will load down a system more than > the inetd's implementation of ident will. Therefore, pidentd should be > phased out. Other than that, pidentd should be using > http://www.FreeBSD.org/~green/freebsd4.c and not linking with libkvm. > pidentd supports DES-encrypted tokens as replies instead of the plaintext usernames. As long as your identd (or any other replacement) does not support this, there is still valid use for pidentd. bye, Harold -- Sleep is an abstinence syndrome wich occurs due to lack of caffein. Wed Mar 4 04:53:33 CET 1998 #unix, ircnet To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
misc/12633: CMI8330 chip based integrated sound card produce no output
Hi, I have some minor probem with my CMI8330 chip based integrated sound card on m726 motherboard. Here's the URL where you can find how to fix it: http://www.cslab.vt.edu/~jobaldwi/cmi8330init.tar.gz = cut here = Thank you very much for your problem report. It has the internal identification `misc/12633'. The individual assigned to look at your report is: freebsd-bugs. >Category: misc >Responsible:freebsd-bugs >Synopsis: CMI8330 chip based integrated sound card produce no output >Arrival-Date: Tue Jul 13 15:30:01 PDT 1999 end === To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap subsystem overhead (was Re: Replacement for grep(1) (part 2))
On Tue, Jul 13, 1999 at 05:12:30PM -0700, Matthew Dillon wrote: > Ok, I will be more specific. > > Under FreeBSD-STABLE *AND* FreeBSD-CURRENT, FreeBSD allocates metadata > structures that scale to the amount of swap space assigned to the system. > However, it is not *precisely* the amount of swap space. > Under FreeBSD-stable, just look under "VM pgdata" to see how much > memory is being wired to support the swap subsystem. This usage covers > both the fixed and dynamic allocations. OK, at the risk of reawakening that particular thread -- if people are a little uneasy about Matt committing to src/*, how about letting him commit to doc/* instead? Matt -- some of these messages of yours could probably turn in to great articles for DaemonNews, or the FreeBSD 'zine, if you were that way inclined. . . N -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <37514...@cs.colorado.edu> To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On Thu, 15 Jul 1999 01:59:12 +0900 "Daniel C. Sobral" wrote: > > That's why you make it a switch. No, really, you *can* just make it > > a switch. > > So, enlighten me, please... how do you switch it in NetBSD? When the code to do it is implemented (not that hard, really, and it is in the list of things to do with UVM), a sysctl will enable/disable overcommit checking. There would be like 4 or 5 places in the code where this boolean switch would have to be tested. -- Jason R. Thorpe To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: dbm_* manpages for review
On Thu, Jul 08, 1999 at 10:22:47PM +0100, Nik Clayton wrote: > Tim Singletary has written some man pages for the dbm_* functions in libc, > which are currently undocumented -- we know they are written in terms > of dbopen(), but it's nice to have them documented anyway. > > Could anyone who knows anything about DBM take a look at docs/12557 and > let me know if they are correct? If they are, I'll commit them. I've had one response to this so far, from Mike Pritchard (thanks Mike). He's corrected some of the macro use in the submitted documentation, but said he didn't do a technical review. So I'm still waiting for any DBM hackers out there to take the 10 minutes required to look at this. If there's no response in (say) 3 days, I'll commit Mike's cleaned up version -- if there are any inaccuracies, I'm sure the wider FreeBSD user base will let us know about them. . . N PS: Only half-kidding about that last paragraph. -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <37514...@cs.colorado.edu> To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On Thu, 15 Jul 1999 01:52:11 +0900 "Daniel C. Sobral" wrote: > > ...um, so, make the code that deals with faulting in the stack a bit > > smarter. > > Uh? Like what? Like overcommitting, for instance? The beauty of > overcommitting is that either you do it or you don't. :-) One option is to special-case overcommit the stack. Another is to set the default stack limits to something more reasonable on a system where overcommit is disabled. -- Jason R. Thorpe To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On Wed, 14 Jul 1999 12:43:07 + Niall Smart wrote: > Perhaps it could be an additional flag to mmap, in this way > people wishing to run an overcommited system could do so > but those writing programs which must not overcommit for > certain memory allocations could ensure they did not do so. This has already been mentioned. SVR4 has MAP_NORESERVE specifcally for this purpose. -- Jason R. Thorpe To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> :Returning NULL isn't an error, it's an indication that there is no more > :memory. Don't think if it as an error, think of it as a hint. > > It's only a hint if it is returned due to the process resource limit > being hit. If it is returned due to the system running out of swap, > it would be an error. Not necessarily. > This creates a dependancy between two completely independant processes > possibly owned two totally unrelated users. This is not acceptable, > and is precisely why this sort of behavior does not occur. In embedded systems, there are no such things as 'independant' processes that are given unlimited access to the system. Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :Returning NULL isn't an error, it's an indication that there is no more :memory. Don't think if it as an error, think of it as a hint. It's only a hint if it is returned due to the process resource limit being hit. If it is returned due to the system running out of swap, it would be an error. Look at it this way: Processes run in their own VM address space and should theoretically be unaffected by other processes. If your resource limits allow a process, say, 32MB of ram, the process should be confident in its ability to utilize that much if it needs to without running out of memory. This is what FreeBSD and most UNIX's give to processes. But if you change the rules so that process A can run the system out of swap (or, in the case of a non-overcommit model, out of reservable space), then process B can get allocation errors without even coming close to their allocation limit. This creates a dependancy between two completely independant processes possibly owned two totally unrelated users. This is not acceptable, and is precisely why this sort of behavior does not occur. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Reading CIS from kernel?
In message <19990714185101.09...@goatsucker.org> Scott Mitchell writes: : Ugh. In that case, can someone back out Poul-Henning's changes to the : if_xe.c in the -STABLE tree? That's (I hope) the only thing stopping it : from working. At least that way only my code will be bogus :-) Believe : me, I know it's ugly, but there's no getting around the fact that the : driver needs to read the CIS, and right now there's no clean way to do that : in -STABLE (is there?). Can I get your comments on the following interface? int pccard_map_cis(int slot) Maps the slot's cis into memory. You must call the before any of the following. It returns 0 on success, or an error from /usr/include/sys/errno.h (most likely EBUSY if there are no memory windows available). int pccard_unmap_cis(int slot) Unmaps the CIS. This frees any resource used by the slot to map its CIS. It returns 0 on success, and an errno value if not. vaddr_t pccard_cis_addr(int slot) Return the virtual address of the CIS. The CIS must be mapped before call this function. Drivers may read/write this memory. Reading this memory will get the CIS entries. Drivers are responsible for interpreting the CIS. Writing to CIS locations generally is used to configure the card and does not change the CIS stored on the card. If the card is not mapped, then 0 will be returned. It is not valid to access memory returned by this call after a call to pccard_unmap_cis. Future interfaces may ease the burdon on driver writers, but this interface will be supported for a while. Does this fill your needs? To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
:On Tue, 13 Jul 1999 23:18:58 -0400 (EDT) : John Baldwin wrote: : : > What does that have to do with overcommit? I student administrate a undergrad : > CS lab at a university, and when student's programs misbehaved, they generate a : > fault and are killed. The only machines that reboot on us without be : > explicitly told to are the NT ones, and yes we run FreeBSD. : :What does it have to do with overcommit? Everthing in the world! : :If you have a lot of users, all of which have buggy programs which eat :a lot of memory, per-user swap quotas don't necessarily save your butt. If every single one of your users is trying to crash your machine daily, maybe you should consider throwing them off the system and finding users that are less hostile. This conversation is getting silly. Do you actually believe that an operating system can magically protect itself 100% from armloads of hostile users? Give me a break. You people are crazy. If you have something worthwhile to say i'll listen, but these "the sky is falling!" arguments are idiotic. -Matt To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :> Also, your named is badly misconfigured if it grows to 130MB. We never :> allow ours to grow past 30MB. : :How do you know what kind of name server configuration kre is running? :Here's an example of a name server running *non-recursive*, serving :11.500 zones: : : PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND :27162 root 2070M 57M sleep 271:01 3.27% 3.27% named : :Are you saying that such configurations should be illegal? : :Steinar Haug, Nethelp consulting, sth...@nethelp.no I assumed that since the guy said that his named GREW, that he was running a recurisve/caching named. Obviously if you are running a non-recursive named the static size will depend on the zones you are serving. Duh! It is not generally beneficial to allow a caching named to exceed 30MB or so on a system that is doing other things. If the system starts to page (which this person's system is obviously doing), then it is doubly a bad idea to allow a named to grow that large. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> :Most of the work we've done wouldn't allow this, especially if we were > :using an OS like FreeBSD with a fairly long bootup time. Especially if > :it can be avoided. > : > :Yes, we could (and did) do our own memory management, but it seems to me > :that the kernel has alot more information available to it and would do > :it better than I could. Then again, maybe I'm totally confused about > :how the VM system 'does its thing', and in reality it's no better at it > :than our code, just alot more complex for no reason. :) :) :) > : > > The kernel just isn't the best place to do this. The level of > sophistication required to satisfy the larger set of programming > is much greater then anything discussed here. The last thing that the > kernel should be doing is returning NULL from malloc() to a program > which is operating within its specifications. You and I disagree on this statement. Returning NULL is completely acceptable, and is the 'specification' for many OS's, including FreeBSD if you set user/process limits. > That doesn't help anyone, least of all the programmer who now has > to deal not only with the complexity of the project he is working > on, but must also deal with the potential that the OS will step in > at any time and give him an error that he must deal with in a > sophisticated fashion even when his software is operating > properly. Returning NULL isn't an error, it's an indication that there is no more memory. Don't think if it as an error, think of it as a hint. General purpose computing tends to make one think that out-of-memory condition is an un-acceptable situation, and the program must exit. FreeBSD exacerbates this by rarely returning NULL and randomly killing off processes which may/may not be involved in the memory fracus. > > The same goes for non-embedded work. Why is it that programs generally > exit when they encounter a memory allocation failure? Because programmer's are lazy. Really, and I do it all the time as well. But, that's because on 'general purpose' computing hardware, re-starting the process or having to wait for a reboot is 'acceptable'. But, to many of my collegues who are running simulations that are running for DAYS and WEEKS, they write code that 'saves the state' of the system when they can't get critical memory, *THEN* abort the application. Yes, it's possible that the process of saving state *may* not work if the system is *really* low on memory, but then again it may work. But, more times than not it *does* (appear) to work, and the work is not lost. > with memory allocation errors" provides no solution. 98% of the source > code in the BSD code base will exit if a memory allocation fails, and > I don't know anyone who wants to go through and somehow "fix" it all > (and one could argue what the "fix" should be when something like grep or > a shell has a memory allocation failure). To require that this code be > made twice as sophisticated as it is already in order to deal with a > non-overcommit model reliably is a fantasy. Who said anything about using the bsd code base in an embedded system? Also, you *can* use limits for that stuff. Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :Most of the work we've done wouldn't allow this, especially if we were :using an OS like FreeBSD with a fairly long bootup time. Especially if :it can be avoided. : :Yes, we could (and did) do our own memory management, but it seems to me :that the kernel has alot more information available to it and would do :it better than I could. Then again, maybe I'm totally confused about :how the VM system 'does its thing', and in reality it's no better at it :than our code, just alot more complex for no reason. :) :) :) : :Nate The kernel just isn't the best place to do this. The level of sophistication required to satisfy the larger set of programming is much greater then anything discussed here. The last thing that the kernel should be doing is returning NULL from malloc() to a program which is operating within its specifications. That doesn't help anyone, least of all the programmer who now has to deal not only with the complexity of the project he is working on, but must also deal with the potential that the OS will step in at any time and give him an error that he must deal with in a sophisticated fashion even when his software is operating properly. The same goes for non-embedded work. Why is it that programs generally exit when they encounter a memory allocation failure? Because it is too damn hard to ensure proper operation at every instance where a program's allocation may fail, especially if the program getting the failure is not responsible for the problem. The glib statement that "programs should deal with memory allocation errors" provides no solution. 98% of the source code in the BSD code base will exit if a memory allocation fails, and I don't know anyone who wants to go through and somehow "fix" it all (and one could argue what the "fix" should be when something like grep or a shell has a memory allocation failure). To require that this code be made twice as sophisticated as it is already in order to deal with a non-overcommit model reliably is a fantasy. Due to the side effects it causes, to put the capability in the kernel in this situation will make things even more unstable then the kernel's current operation. A situation made much worse by the magnitude of the resources necessary to bring such a system up to the same performance levels attainable by a standard overcommit system. The only case where this does not occur is also the case where the programmer can fairly easily manage the memory to avoid the overcommit from occuring in the first place. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: tcp windowsize query?
> In article [EMAIL PROTECTED]> you >write: > >>delayed ack sounds interesting > > > >Turning that off disables TCP slow-start. It's a huge performance > >booster for things like SMB service, where you have lots of short-lived > >TCP connections on a local net. > > Mike probably already knows this, but just a general clarification: > > No, it does _not_ turn off slow-start. Normally a TCP implementation > will delay sending an ACK to the peer until the TCP fasttimer expires, > or certain other conditions are met. The goal here is to amortize an > ACK over several incoming data packets. Yeah, brain fart. Slow start != delayed ACK. Blah. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ [EMAIL PROTECTED] \\-- Joseph Merrick \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> :> > Quite true. In the embedded world we preallocate memory and shape > :> > the programs to what is available in the system. But if we run out > :> > of memory we usually panic and reboot - because the code is designed > :> > to NOT run out of memory and thus running out of memory is a catastrophic > :> > situation. > : > :*ACK* This is unacceptable in many 'embedded' systems. > > Don't confuse a watchdog panic from other conditions. If the embedded > system software is supposed to deal with a low-memory condition and can't, > the failsafe is all that's left between it and infinity. > > The statement that the kernel's overcommit methodology somehow prevents > one from being able to build embedded systems on top of it is just plain > incorrect. The embedded system is perfectly capable of implementing its > own memory management to avoid the filesafe provided by the kernel. It is. But, it's not the most effecient/effective way of doing it. Sometimes it's best to let the 'kernel' do the things it does best, and let the users worry about the things they do best. The kernel manages memory alot more effeciently than any userland process can do, so why not let it? > Most of the embedded work I've done -- mainly remote telemetry > units running with flash and a megabyte or so of ram -- panic and > reboot if they run out of memory. Most of the work we've done wouldn't allow this, especially if we were using an OS like FreeBSD with a fairly long bootup time. Especially if it can be avoided. Yes, we could (and did) do our own memory management, but it seems to me that the kernel has alot more information available to it and would do it better than I could. Then again, maybe I'm totally confused about how the VM system 'does its thing', and in reality it's no better at it than our code, just alot more complex for no reason. :) :) :) Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
:> > :> > Quite true. In the embedded world we preallocate memory and shape :> > the programs to what is available in the system. But if we run out :> > of memory we usually panic and reboot - because the code is designed :> > to NOT run out of memory and thus running out of memory is a catastrophic :> > situation. : :*ACK* This is unacceptable in many 'embedded' systems. Don't confuse a watchdog panic from other conditions. If the embedded system software is supposed to deal with a low-memory condition and can't, the failsafe is all that's left between it and infinity. The statement that the kernel's overcommit methodology somehow prevents one from being able to build embedded systems on top of it is just plain incorrect. The embedded system is perfectly capable of implementing its own memory management to avoid the filesafe provided by the kernel. Most of the embedded work I've done -- mainly remote telemetry units running with flash and a megabyte or so of ram -- panic and reboot if they run out of memory. I have several dozen units in the field each keeping track of several thousand data points on 2 minute intervals which have not ever crashed. The only time we reboot them is when we need to upgrade the OS core. The last time was 4 years ago. *These* units will panic and reboot if they run out of memory because the software is designed not to. It is as simple as that. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: docs/12377: doc patch for login_cap.
On Fri, Jul 09, 1999 at 11:39:38AM +0200, Sheldon Hearn wrote: > On Thu, 08 Jul 1999 20:59:58 +0100, Nik Clayton wrote: > > With that in mind, how about this patch (in conjunction with the patch to > > login.conf in the original PR, which just updates a comment)? > > This looks much better. :-) Committing it now. N -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
[ Trimmed CC list a bit ] > > :* even if you are not willing to pay that price, there _are_ people > > :who are quite willing to pay that price to get the benefits that they > > :see (whether it's a matter of perception or not, from their > > :perspective they may as well be real) of such a scheme. > > > > Quite true. In the embedded world we preallocate memory and shape > > the programs to what is available in the system. But if we run out > > of memory we usually panic and reboot - because the code is designed > > to NOT run out of memory and thus running out of memory is a catastrophic > > situation. *ACK* This is unacceptable in many 'embedded' systems. > There's a whole spectrum of embedded devices, and applications that > run on them. That definition works for some of them, but definitely > not all. > Totally agreed. A previous poster brought up the fact that *some* embedded systems are built to deal with 'out of memory' situations, and that the 'total' amount of memory used in the system can be used by other parts of the system. For performance reasons, a particular application may choose to 'cache' data, but in low memory situation it can 'free' up alot of memory. You don't want to put hard-coded limits the process simply because if the memory is there you want it to be able to use it, but you *certainly* don't want to go through a reboot just to get memory back. [ And, I don't want to write my own OS to do this for me. :) ] (However, I agree that for general purpose computing, over-commit is the way to go. But, *BSD is not just for general purpose computing, although that is it's primary market.) Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
: :One option is to special-case overcommit the stack. Another is to :set the default stack limits to something more reasonable on a system :where overcommit is disabled. : :-- Jason R. Thorpe <[EMAIL PROTECTED]> Try setting all the resource limits to something reasonable on general principles. It would work as well in an overcommit system as it would in a non-overcommit system. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: a BSD identd
On Tue, Jul 13, 1999 at 11:47:33PM -0400, Brian F. Feldman wrote: > We don't _need_ pidentd anymore. It will load down a system more than > the inetd's implementation of ident will. Therefore, pidentd should be > phased out. Other than that, pidentd should be using > http://www.FreeBSD.org/~green/freebsd4.c and not linking with libkvm. > pidentd supports DES-encrypted tokens as replies instead of the plaintext usernames. As long as your identd (or any other replacement) does not support this, there is still valid use for pidentd. bye, Harold -- Sleep is an abstinence syndrome wich occurs due to lack of caffein. Wed Mar 4 04:53:33 CET 1998 #unix, ircnet To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap subsystem overhead (was Re: Replacement for grep(1) (part 2))
On Tue, Jul 13, 1999 at 05:12:30PM -0700, Matthew Dillon wrote: > Ok, I will be more specific. > > Under FreeBSD-STABLE *AND* FreeBSD-CURRENT, FreeBSD allocates metadata > structures that scale to the amount of swap space assigned to the system. > However, it is not *precisely* the amount of swap space. > Under FreeBSD-stable, just look under "VM pgdata" to see how much > memory is being wired to support the swap subsystem. This usage covers > both the fixed and dynamic allocations. OK, at the risk of reawakening that particular thread -- if people are a little uneasy about Matt committing to src/*, how about letting him commit to doc/* instead? Matt -- some of these messages of yours could probably turn in to great articles for DaemonNews, or the FreeBSD 'zine, if you were that way inclined. . . N -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On Thu, 15 Jul 1999 01:59:12 +0900 "Daniel C. Sobral" <[EMAIL PROTECTED]> wrote: > > That's why you make it a switch. No, really, you *can* just make it > > a switch. > > So, enlighten me, please... how do you switch it in NetBSD? When the code to do it is implemented (not that hard, really, and it is in the list of things to do with UVM), a sysctl will enable/disable overcommit checking. There would be like 4 or 5 places in the code where this boolean switch would have to be tested. -- Jason R. Thorpe <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: dbm_* manpages for review
On Thu, Jul 08, 1999 at 10:22:47PM +0100, Nik Clayton wrote: > Tim Singletary has written some man pages for the dbm_* functions in libc, > which are currently undocumented -- we know they are written in terms > of dbopen(), but it's nice to have them documented anyway. > > Could anyone who knows anything about DBM take a look at docs/12557 and > let me know if they are correct? If they are, I'll commit them. I've had one response to this so far, from Mike Pritchard (thanks Mike). He's corrected some of the macro use in the submitted documentation, but said he didn't do a technical review. So I'm still waiting for any DBM hackers out there to take the 10 minutes required to look at this. If there's no response in (say) 3 days, I'll commit Mike's cleaned up version -- if there are any inaccuracies, I'm sure the wider FreeBSD user base will let us know about them. . . N PS: Only half-kidding about that last paragraph. -- [intentional self-reference] can be easily accommodated using a blessed, non-self-referential dummy head-node whose own object destructor severs the links. -- Tom Christiansen in <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On Thu, 15 Jul 1999 01:52:11 +0900 "Daniel C. Sobral" <[EMAIL PROTECTED]> wrote: > > ...um, so, make the code that deals with faulting in the stack a bit smarter. > > Uh? Like what? Like overcommitting, for instance? The beauty of > overcommitting is that either you do it or you don't. :-) One option is to special-case overcommit the stack. Another is to set the default stack limits to something more reasonable on a system where overcommit is disabled. -- Jason R. Thorpe <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
On Wed, 14 Jul 1999 12:43:07 + Niall Smart <[EMAIL PROTECTED]> wrote: > Perhaps it could be an additional flag to mmap, in this way > people wishing to run an overcommited system could do so > but those writing programs which must not overcommit for > certain memory allocations could ensure they did not do so. This has already been mentioned. SVR4 has MAP_NORESERVE specifcally for this purpose. -- Jason R. Thorpe <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replacement for grep(1) (part 2)
> Also, your named is badly misconfigured if it grows to 130MB. We never > allow ours to grow past 30MB. How do you know what kind of name server configuration kre is running? Here's an example of a name server running *non-recursive*, serving 11.500 zones: PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND 27162 root 2070M 57M sleep 271:01 3.27% 3.27% named Are you saying that such configurations should be illegal? Steinar Haug, Nethelp consulting, sth...@nethelp.no To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Swap overcommit (was Re: Replacement for grep(1) (part 2))
On Tue, 13 Jul 1999 23:18:58 -0400 (EDT) John Baldwin wrote: > What does that have to do with overcommit? I student administrate a > undergrad > CS lab at a university, and when student's programs misbehaved, they > generate a > fault and are killed. The only machines that reboot on us without be > explicitly told to are the NT ones, and yes we run FreeBSD. What does it have to do with overcommit? Everthing in the world! If you have a lot of users, all of which have buggy programs which eat a lot of memory, per-user swap quotas don't necessarily save your butt. And maybe the individual programs didn't encounter their resource limits. ...but the sheer number of these runaway things caused the overcommit to be a problem. If malloc() or whatever had actually returned NULL at the right time (i.e. as backing store was about to become overcommitted), then these runaway processes would have stopped running away (they would have gotten a SIGSEGV and died). Anyhow, my "lame undergrads" example comes from a time when PCs weren't really powerful enough for the job (or something; anyhow, we didn't have any in the department :-). My example is from a Sequent Balance (16 ns32032 processors, 64M RAM [I think; been a while], 4.2BSD variant). -- Jason R. Thorpe To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Reading CIS from kernel?
In message <[EMAIL PROTECTED]> Scott Mitchell writes: : Ugh. In that case, can someone back out Poul-Henning's changes to the : if_xe.c in the -STABLE tree? That's (I hope) the only thing stopping it : from working. At least that way only my code will be bogus :-) Believe : me, I know it's ugly, but there's no getting around the fact that the : driver needs to read the CIS, and right now there's no clean way to do that : in -STABLE (is there?). Can I get your comments on the following interface? int pccard_map_cis(int slot) Maps the slot's cis into memory. You must call the before any of the following. It returns 0 on success, or an error from /usr/include/sys/errno.h (most likely EBUSY if there are no memory windows available). int pccard_unmap_cis(int slot) Unmaps the CIS. This frees any resource used by the slot to map its CIS. It returns 0 on success, and an errno value if not. vaddr_t pccard_cis_addr(int slot) Return the virtual address of the CIS. The CIS must be mapped before call this function. Drivers may read/write this memory. Reading this memory will get the CIS entries. Drivers are responsible for interpreting the CIS. Writing to CIS locations generally is used to configure the card and does not change the CIS stored on the card. If the card is not mapped, then 0 will be returned. It is not valid to access memory returned by this call after a call to pccard_unmap_cis. Future interfaces may ease the burdon on driver writers, but this interface will be supported for a while. Does this fill your needs? To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
