Re: Overcommit and calloc()
[EMAIL PROTECTED] (Dag-Erling Smorgrav) writes: "Kelly Yancey" [EMAIL PROTECTED] writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an "non-overcomitting malloc", it would return NULL and set errno to ENOMEM, instead of dumping core. It won't dump core. If it isn't the biggest process, it'll simply succeed, but somebody else is killed. If it's the biggest process, it'll die with SIGKILL without dumping core. There *are* systems that kill "random" processes when swap runs out, presumably when they need to actually get pages that aren't available. FreeBSD is not one of them. Overcommit still has nothing to do with malloc. Either the *system* is overcommitted or it isn't - per-process overcommitment is irrelevant, as is the way memory has become overcommitted. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
"Kelly Yancey" [EMAIL PROTECTED] writes: I don't know how many programs make use of calloc() but would not a more efficient algorithm be to better integrate it with malloc() such that if there is a fragment on the heap which can be used, bzero() it and return that, otherwise, simply call sbrk() and return that (since it is already zero'ed). Currently, in the event that malloc() simply returns newly sbrk()'ed memory, we unnecessarily zero it again. I don't see the point. I've seen very few examples of justified calloc() use. For instance, I see a lot of people use calloc() instead of malloc() when allocating strings, just to make sure they'll be terminated after a strncpy(), instead of simply setting the last character to '\0' after the strncpy(). When I allocate memory, I usually intend to put something in it. There's always the odd struct sockaddr_in which I bzero() before filling it in, but they're usually on the stack. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
John-Mark Gurney [EMAIL PROTECTED] writes: Dag-Erling Smorgrav scribbled this message on Jul 20: When I allocate memory, I usually intend to put something in it. There's always the odd struct sockaddr_in which I bzero() before filling it in, but they're usually on the stack. and even then, I don't believe in filling sockaddr_in w/ bzero, I believe in using getsockaddr on it so that you actually get all the fields filled out properly... See? One less reason to use bzero() / calloc() :) DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
Well, bzero could map all memory (outside the boundaries) to a single zeroed page marked copy on write. The statistics you could gather might then point out some grossly broken programs. Peter -- Peter Dufault ([EMAIL PROTECTED]) Realtime development, Machine control, HD Associates, Inc. Safety critical systems, Agency approval To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Overcommit and calloc()
From: Charles Randall I have another post on this list which begs the question: if memory given to us from sbrk() is already zeroed, why zero it again if we don't have too if we make calloc() smarter, we could save come clock cycles. Because the memory returned from malloc() might be from a previous malloc()/free() and may be dirty. Charles Specifically, if there are no blocks on the heap that can be re-used by malloc(), it calls sbrk() to extend the process' address space. sbrk() always returns memory that the kernel has already gone through the effort of zero'ing...why do it again? I am thinking about making calloc() more intellegent like malloc(): if it re-uses a free()'ed memory block, then it has to bzero() it (exactly like it does now), if it has to sbrk() then it can skip the bzero() step and save some clock cycles. I'm expecting that in addition of gaining clock cycles from not zeroing the memory twice, the fact that calloc() won't have to touch all newly allocated pages would reduce the amount of work by the VM system and give another little performance boost. On recent thought though, I seem to recall having read in the 4.4BSD Daemon book that having the kernel zero memory is not the preferred practice, but present because when they tried to stop many progrems dies which assumed memory was initialized to zero. If I am remembering that correctly, then, the only real concern is that one day we may want to kernel to stop zeroing pages, in which case the extra logic in calloc() would be for nought. Really the jist of the changes I'm looking to make are to use the fact that the kernel is already zeroing pages to optimize the calloc() implementation. If this ever changes, then the optimization would go away. Kelly ~[EMAIL PROTECTED]~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Overcommit and calloc()
Hi, At 1:28 pm -0400 20/7/99, Kelly Yancey wrote: [...] On recent thought though, I seem to recall having read in the 4.4BSD Daemon book that having the kernel zero memory is not the preferred practice, but present because when they tried to stop many progrems dies which assumed memory was initialized to zero. Handing out unzeroed memory is a potential security hole. -- Bob Bishop (0118) 977 4017 international code +44 118 [EMAIL PROTECTED]fax (0118) 989 4254 between 0800 and 1800 UK To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: RE: Overcommit and calloc()
:Hi, : :At 1:28 pm -0400 20/7/99, Kelly Yancey wrote: :[...] : On recent thought though, I seem to recall having read in the 4.4BSD :Daemon book that having the kernel zero memory is not the preferred :practice, but present because when they tried to stop many progrems dies :which assumed memory was initialized to zero. : :Handing out unzeroed memory is a potential security hole. : :-- :Bob Bishop (0118) 977 4017 international code +44 118 :[EMAIL PROTECTED]fax (0118) 989 4254 between 0800 and 1800 UK It should also be noted that unless your system is entirely cpu-bound, there is no cost to the kernel to zero memory because it pre-zero's pages in its idle loop. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: RE: Overcommit and calloc()
-Original Message- From: Matthew Dillon [mailto:[EMAIL PROTECTED]] Sent: Tuesday, July 20, 1999 1:53 PM To: Kelly Yancey Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: RE: Overcommit and calloc() I think this would be a waste of time. As I have said, very few large allocations use calloc(). Nearly all small allocations come off the heap. The cost of adding the complexity to calloc to avoid zeroing the data is not going to be worth the minor (and unnoticeable) improvement in performance that you reap from it. Right now malloc/calloc add new blocks to the heap in chunks and while they may know that a page is zero on allocation, once the first small bit has been removed from the heap they have no idea that the remainder is still zero. Making them knowledgeable of that will simply make the critical path more costly and probably end up in reducing your overall performance. -Matt Matthew Dillon [EMAIL PROTECTED] I don't dare claim to know as much as about the system as you do, how about this: I am sure that I can at least just find a spare bit in struct pgfree (/usr/src/lib/libc/stdlib/malloc.c) in which I can store a boolean flag indicating whether or not the run is already zeroed. Then all I add in complexity is setting the flag when a run is added to the list via map_pages/extend_pgdir, and a fairly small calloc() implementation which looks much like the existing calloc() but checking specifically for a run with the bit set. malloc()'s implementation doesn't have to change a bit since it doesn't care whether it's memory is zeroed or not, so the critical path isn't affected. calloc() will then use already zero'ed pages if they are available, otherwise just take what it can get and bzero() them itself. Since there will only ever be at most a single run of pre-zero'ed pages, I'll probably keep a separate pointer directly to that struct pgfree to optimize calloc() further (which would add to the critical path, so I may reconsider). I can't actually make patches, though, until I get home. As far as your other point, you imply that because pages are be pre-zeroed during the idle loop, we are currently no worse off then if they were never zeroed at all. Which is true. But is that not to say that we wouldn't be better off by taking advantage of the work we have already done? Besides, how many of us are running things like rc5 or seti@home which prevent the idle loop from doing any work? Os it not possible that we run out of zeroed pages and then when a process calls sbrk(), the VM system has to zero the memory to give to us (which may be zeroed again by calloc())...in which case a call to calloc() results in zeroing the memory twice *while we wait*. Admittingly, calloc() isn't called nearly as often as malloc(). But malloc() is already pretty darned optimized thanks to phk, I'm just trying to extend his optimizations to include cases where calloc() could further benefit. Don't worry I'll post benchmarks before and after, with and without rc5 running in the background, along with the patches. I really appreciate all the feedback though, Kelly ~[EMAIL PROTECTED]~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
d...@flood.ping.uio.no (Dag-Erling Smorgrav) writes: Kelly Yancey kby...@alcnet.com writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an non-overcomitting malloc, it would return NULL and set errno to ENOMEM, instead of dumping core. It won't dump core. If it isn't the biggest process, it'll simply succeed, but somebody else is killed. If it's the biggest process, it'll die with SIGKILL without dumping core. There *are* systems that kill random processes when swap runs out, presumably when they need to actually get pages that aren't available. FreeBSD is not one of them. Overcommit still has nothing to do with malloc. Either the *system* is overcommitted or it isn't - per-process overcommitment is irrelevant, as is the way memory has become overcommitted. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
Kelly Yancey kby...@alcnet.com writes: I don't know how many programs make use of calloc() but would not a more efficient algorithm be to better integrate it with malloc() such that if there is a fragment on the heap which can be used, bzero() it and return that, otherwise, simply call sbrk() and return that (since it is already zero'ed). Currently, in the event that malloc() simply returns newly sbrk()'ed memory, we unnecessarily zero it again. I don't see the point. I've seen very few examples of justified calloc() use. For instance, I see a lot of people use calloc() instead of malloc() when allocating strings, just to make sure they'll be terminated after a strncpy(), instead of simply setting the last character to '\0' after the strncpy(). When I allocate memory, I usually intend to put something in it. There's always the odd struct sockaddr_in which I bzero() before filling it in, but they're usually on the stack. DES -- Dag-Erling Smorgrav - d...@flood.ping.uio.no To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
Dag-Erling Smorgrav scribbled this message on Jul 20: When I allocate memory, I usually intend to put something in it. There's always the odd struct sockaddr_in which I bzero() before filling it in, but they're usually on the stack. and even then, I don't believe in filling sockaddr_in w/ bzero, I believe in using getsockaddr on it so that you actually get all the fields filled out properly... -- John-Mark Gurney Voice: +1 541 684 8449 Cu Networking P.O. Box 5693, 97405 The soul contains in itself the event that shall presently befall it. The event is only the actualizing of its thought. -- Ralph Waldo Emerson To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
John-Mark Gurney gurne...@efn.org writes: Dag-Erling Smorgrav scribbled this message on Jul 20: When I allocate memory, I usually intend to put something in it. There's always the odd struct sockaddr_in which I bzero() before filling it in, but they're usually on the stack. and even then, I don't believe in filling sockaddr_in w/ bzero, I believe in using getsockaddr on it so that you actually get all the fields filled out properly... See? One less reason to use bzero() / calloc() :) DES -- Dag-Erling Smorgrav - d...@flood.ping.uio.no To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
Well, bzero could map all memory (outside the boundaries) to a single zeroed page marked copy on write. The statistics you could gather might then point out some grossly broken programs. Peter -- Peter Dufault (dufa...@hda.com) Realtime development, Machine control, HD Associates, Inc. Safety critical systems, Agency approval To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
From: Charles Randall I have another post on this list which begs the question: if memory given to us from sbrk() is already zeroed, why zero it again if we don't have too if we make calloc() smarter, we could save come clock cycles. Because the memory returned from malloc() might be from a previous malloc()/free() and may be dirty. Charles Specifically, if there are no blocks on the heap that can be re-used by malloc(), it calls sbrk() to extend the process' address space. sbrk() always returns memory that the kernel has already gone through the effort of zero'ing...why do it again? I am thinking about making calloc() more intellegent like malloc(): if it re-uses a free()'ed memory block, then it has to bzero() it (exactly like it does now), if it has to sbrk() then it can skip the bzero() step and save some clock cycles. I'm expecting that in addition of gaining clock cycles from not zeroing the memory twice, the fact that calloc() won't have to touch all newly allocated pages would reduce the amount of work by the VM system and give another little performance boost. On recent thought though, I seem to recall having read in the 4.4BSD Daemon book that having the kernel zero memory is not the preferred practice, but present because when they tried to stop many progrems dies which assumed memory was initialized to zero. If I am remembering that correctly, then, the only real concern is that one day we may want to kernel to stop zeroing pages, in which case the extra logic in calloc() would be for nought. Really the jist of the changes I'm looking to make are to use the fact that the kernel is already zeroing pages to optimize the calloc() implementation. If this ever changes, then the optimization would go away. Kelly ~kby...@posi.net~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
From: Kelly Yancey kby...@alcnet.com Date: Tue, 20 Jul 1999 13:28:21 -0400 On recent thought though, I seem to recall having read in the 4.4BSD Daemon book that having the kernel zero memory is not the preferred practice, but present because when they tried to stop many progrems dies which assumed memory was initialized to zero. If I am remembering that correctly, then, the only real concern is that one day we may want to kernel to stop zeroing pages, in which case the extra logic in calloc() would be for nought. I'd *think* you'd want to ensure that lack of initializing the data didn't become a way for unintended access to data that should not have been available to the process in question. (Ugh. Too many negatives in there.) Anyway, the process merely reminded me of the ability on a system I used 28+ years ago, where a FORTRAN program could open a file for writing, but read it first... and possibly find some interesting information left over from a previous program (No, that wasn't a UNIX system, let alone FreeBSD. :-}) Cheers, david -- David Wolfskill d...@whistle.comUNIX System Administrator voice: (650) 577-7158 pager: (888) 347-0197 FAX: (650) 372-5915 To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
Hi, At 1:28 pm -0400 20/7/99, Kelly Yancey wrote: [...] On recent thought though, I seem to recall having read in the 4.4BSD Daemon book that having the kernel zero memory is not the preferred practice, but present because when they tried to stop many progrems dies which assumed memory was initialized to zero. Handing out unzeroed memory is a potential security hole. -- Bob Bishop (0118) 977 4017 international code +44 118 r...@gid.co.ukfax (0118) 989 4254 between 0800 and 1800 UK To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: RE: Overcommit and calloc()
I think this would be a waste of time. As I have said, very few large allocations use calloc(). Nearly all small allocations come off the heap. The cost of adding the complexity to calloc to avoid zeroing the data is not going to be worth the minor (and unnoticeable) improvement in performance that you reap from it. Right now malloc/calloc add new blocks to the heap in chunks and while they may know that a page is zero on allocation, once the first small bit has been removed from the heap they have no idea that the remainder is still zero. Making them knowledgeable of that will simply make the critical path more costly and probably end up in reducing your overall performance. -Matt Matthew Dillon dil...@backplane.com : : Specifically, if there are no blocks on the heap that can be re-used by :malloc(), it calls sbrk() to extend the process' address space. sbrk() :always returns memory that the kernel has already gone through the effort of :zero'ing...why do it again? I am thinking about making calloc() more :intellegent like malloc(): if it re-uses a free()'ed memory block, then it :has to bzero() it (exactly like it does now), if it has to sbrk() then it :can skip the bzero() step and save some clock cycles. I'm expecting that in :addition of gaining clock cycles from not zeroing the memory twice, the fact :that calloc() won't have to touch all newly allocated pages would reduce the :amount of work by the VM system and give another little performance boost. : : On recent thought though, I seem to recall having read in the 4.4BSD :Daemon book that having the kernel zero memory is not the preferred :practice, but present because when they tried to stop many progrems dies :which assumed memory was initialized to zero. If I am remembering that :correctly, then, the only real concern is that one day we may want to kernel :to stop zeroing pages, in which case the extra logic in calloc() would be :for nought. : Really the jist of the changes I'm looking to make are to use the fact :that the kernel is already zeroing pages to optimize the calloc() :implementation. If this ever changes, then the optimization would go away. : : Kelly : ~kby...@posi.net~ : FreeBSD - The Power To Serve - http://www.freebsd.org/ : Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: RE: Overcommit and calloc()
:Hi, : :At 1:28 pm -0400 20/7/99, Kelly Yancey wrote: :[...] : On recent thought though, I seem to recall having read in the 4.4BSD :Daemon book that having the kernel zero memory is not the preferred :practice, but present because when they tried to stop many progrems dies :which assumed memory was initialized to zero. : :Handing out unzeroed memory is a potential security hole. : :-- :Bob Bishop (0118) 977 4017 international code +44 118 :r...@gid.co.ukfax (0118) 989 4254 between 0800 and 1800 UK It should also be noted that unless your system is entirely cpu-bound, there is no cost to the kernel to zero memory because it pre-zero's pages in its idle loop. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: RE: Overcommit and calloc()
-Original Message- From: Matthew Dillon [mailto:dil...@apollo.backplane.com] Sent: Tuesday, July 20, 1999 1:53 PM To: Kelly Yancey Cc: crand...@matchlogic.com; freebsd-hackers@FreeBSD.ORG Subject: Re: RE: Overcommit and calloc() I think this would be a waste of time. As I have said, very few large allocations use calloc(). Nearly all small allocations come off the heap. The cost of adding the complexity to calloc to avoid zeroing the data is not going to be worth the minor (and unnoticeable) improvement in performance that you reap from it. Right now malloc/calloc add new blocks to the heap in chunks and while they may know that a page is zero on allocation, once the first small bit has been removed from the heap they have no idea that the remainder is still zero. Making them knowledgeable of that will simply make the critical path more costly and probably end up in reducing your overall performance. -Matt Matthew Dillon dil...@backplane.com I don't dare claim to know as much as about the system as you do, how about this: I am sure that I can at least just find a spare bit in struct pgfree (/usr/src/lib/libc/stdlib/malloc.c) in which I can store a boolean flag indicating whether or not the run is already zeroed. Then all I add in complexity is setting the flag when a run is added to the list via map_pages/extend_pgdir, and a fairly small calloc() implementation which looks much like the existing calloc() but checking specifically for a run with the bit set. malloc()'s implementation doesn't have to change a bit since it doesn't care whether it's memory is zeroed or not, so the critical path isn't affected. calloc() will then use already zero'ed pages if they are available, otherwise just take what it can get and bzero() them itself. Since there will only ever be at most a single run of pre-zero'ed pages, I'll probably keep a separate pointer directly to that struct pgfree to optimize calloc() further (which would add to the critical path, so I may reconsider). I can't actually make patches, though, until I get home. As far as your other point, you imply that because pages are be pre-zeroed during the idle loop, we are currently no worse off then if they were never zeroed at all. Which is true. But is that not to say that we wouldn't be better off by taking advantage of the work we have already done? Besides, how many of us are running things like rc5 or s...@home which prevent the idle loop from doing any work? Os it not possible that we run out of zeroed pages and then when a process calls sbrk(), the VM system has to zero the memory to give to us (which may be zeroed again by calloc())...in which case a call to calloc() results in zeroing the memory twice *while we wait*. Admittingly, calloc() isn't called nearly as often as malloc(). But malloc() is already pretty darned optimized thanks to phk, I'm just trying to extend his optimizations to include cases where calloc() could further benefit. Don't worry I'll post benchmarks before and after, with and without rc5 running in the background, along with the patches. I really appreciate all the feedback though, Kelly ~kby...@posi.net~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: RE: Overcommit and calloc()
Hi again, At 10:54 am -0700 20/7/99, Matthew Dillon wrote: [...] It should also be noted that unless your system is entirely cpu-bound, there is no cost to the kernel to zero memory because it pre-zero's pages in its idle loop. Thanks to distributed.net, SETI. et al, idle cycles are fast going out of fashion. -- Bob Bishop (0118) 977 4017 international code +44 118 r...@gid.co.ukfax (0118) 989 4254 between 0800 and 1800 UK To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: RE: Overcommit and calloc()
:Hi again, : :At 10:54 am -0700 20/7/99, Matthew Dillon wrote: :[...] :It should also be noted that unless your system is entirely cpu-bound, :there is no cost to the kernel to zero memory because it pre-zero's :pages in its idle loop. : :Thanks to distributed.net, SETI. et al, idle cycles are fast going out of :fashion. : :-- :Bob Bishop (0118) 977 4017 international code +44 118 The overhead of running seti - the OS allowing the mintick interval to elapse when it gives seti cpu, that is - is going to be several orders of magnitude greater then any increase in performance that you get from trying to optimize calloc(). -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
"Kelly Yancey" [EMAIL PROTECTED] writes:
I'm afraid my question got lost amongst the see of overcommit messages. :)
I was curious if calloc() was overcommitted also?
Here's our calloc() implementation:
void *
calloc(num, size)
size_t num;
register size_t size;
{
register void *p;
size *= num;
if ( (p = malloc(size)) )
bzero(p, size);
return(p);
}
so the answer is yes, it overcommits, but the bzero() may cause the
system to run out of swap.
DES
--
Dag-Erling Smorgrav - [EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
RE: Overcommit and calloc()
Ahh...but wouldn't the bzero() touch all of the memory just allocated
functionally making it non-overcommit? I realize that technically it isn't
the same since although calloc() would ensure the memory actually was
committed (again, since the bzero() touches all the pages), if there weren't
enough pages free then a random process would still be killed rather than
returning null.
Believe me, I'm not trying to start that war again :) I'm just pointing
out that while technically it is still overcommit, it will touch the pages
ensuring that the memory does in fact exist (is committed). Perhaps this is
why I've always heard we should avoid calloc() because it is 'slow'?
Thanks for the great feedback, I should kick myself for now digging my
hands in the mud myself (I assumed it was more complicated than that :) ).
Kelly
~[EMAIL PROTECTED]~
FreeBSD - The Power To Serve - http://www.freebsd.org/
Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD
-Original Message-
From: Dag-Erling Smorgrav [mailto:[EMAIL PROTECTED]]
Sent: Monday, July 19, 1999 12:07 PM
To: Kelly Yancey
Cc: [EMAIL PROTECTED]
Subject: Re: Overcommit and calloc()
"Kelly Yancey" [EMAIL PROTECTED] writes:
I'm afraid my question got lost amongst the see of overcommit
messages. :)
I was curious if calloc() was overcommitted also?
Here's our calloc() implementation:
void *
calloc(num, size)
size_t num;
register size_t size;
{
register void *p;
size *= num;
if ( (p = malloc(size)) )
bzero(p, size);
return(p);
}
so the answer is yes, it overcommits, but the bzero() may cause the
system to run out of swap.
DES
--
Dag-Erling Smorgrav - [EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
"Kelly Yancey" [EMAIL PROTECTED] writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an "non-overcomitting malloc", it would return NULL and set errno to ENOMEM, instead of dumping core. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
In the last episode (Jul 19), Dag-Erling Smorgrav said: "Kelly Yancey" [EMAIL PROTECTED] writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an "non-overcomitting malloc", it would return NULL and set errno to ENOMEM, instead of dumping core. It should be possible to modify calloc to trap signals, then bzero. If bzero faults, you free the memory and return NULL. No, wait. You can't trap SIGKILL. How about this. mmap() some anonymous memory MAP_SHARED, fork a child to bzero it. If the child dies, unmmap and return NULL. If the child succeeds, use the memory. This memory won't be freeable with malloc(), though. -Dan Nelson [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Overcommit and calloc()
-Original Message- From: Dan Nelson [mailto:[EMAIL PROTECTED]] Sent: Monday, July 19, 1999 12:55 PM To: Dag-Erling Smorgrav Cc: Kelly Yancey; [EMAIL PROTECTED] Subject: Re: Overcommit and calloc() In the last episode (Jul 19), Dag-Erling Smorgrav said: "Kelly Yancey" [EMAIL PROTECTED] writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an "non-overcomitting malloc", it would return NULL and set errno to ENOMEM, instead of dumping core. It should be possible to modify calloc to trap signals, then bzero. If bzero faults, you free the memory and return NULL. No, wait. You can't trap SIGKILL. How about this. mmap() some anonymous memory MAP_SHARED, fork a child to bzero it. If the child dies, unmmap and return NULL. If the child succeeds, use the memory. This memory won't be freeable with malloc(), though. Hrm. I'm not actually trying to get my system to not overcommit memory. I like overcommit. Besides, every process would have to trap SIGKILL (if you even could) to simulate a non-overcommit system because any process could be killed, not just the one requesting the memory, right? Really, I was just on an educational quest. I was curious to exactly how calloc() was implemented and what affect that had on our overcommit policy. DES was nice enough to show me that calloc() is just a malloc()+bzero() so effectively, the memory gets 'committed' because all of the pages are touched immediately after the malloc(). Whether or not a process get's shot is another matter entirely. :) I have another post on this list which begs the question: if memory given to us fro sbrk() is already zeroed, why zero it again if we don't have too if we make calloc() smarter, we could save come clock cycles. The real question is, how many? A quick scan (not exact) of the number of times calloc() is called yields: $ cd /usr/src $ t=0; grep -c -h -R "calloc" * | while read n; do let t+=$n; echo $t; done | tail -1 828 $ t=0; grep -c -h -R "malloc" * | while read n; do let t+=$n; echo $t; done | tail -1 11380 (of course, they each are a little high due to comments and the actual function definitions themselves; cavaet emptor) So calloc() is only used about 1/14th as often as malloc(), and considering many of those calloc() calls would still be serviced the same was a malloc() (reusing memory already on the heap and bzero()'ing it), I'm not 100% sure if the added complexity if worth the performance improvement. If it is, I'de be glad to "whip" some patches up for it (it really shouldn't be too hard). Kelly ~[EMAIL PROTECTED]~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Overcommit and calloc()
From: Kelly Yancey [mailto:[EMAIL PROTECTED]] I have another post on this list which begs the question: if memory given to us fro sbrk() is already zeroed, why zero it again if we don't have too if we make calloc() smarter, we could save come clock cycles. Because the memory returned from malloc() might be from a previous malloc()/free() and may be dirty. Charles To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: RE: Overcommit and calloc()
:From: Kelly Yancey [mailto:[EMAIL PROTECTED]] :I have another post on this list which begs the question: if memory given :to us fro sbrk() is already zeroed, why zero it again if we don't have :too if we make calloc() smarter, we could save come clock cycles. : :Because the memory returned from malloc() might be from a previous :malloc()/free() and may be dirty. : :Charles malloc() uses madvise( ... MADV_FREE) heavily in order to reduce the number of page faults that need to be taken through the course of a program's operation. MADV_FREE is an advisory free that causes FreeBSD to mark the underlying page(s) clean and to free any associated swap backing store. The kernel avoids actually freeing the page until it needs the memory, and the process can re-dirty the page to keep it. No new page-faults occur if the kernel has not reclaimed the page at the time the process reuses it. If the kernel reclaims the page first, the kernel replaces it with zero-fill. If the process reclaims the page first, the page's previous contents (considered to be 'garbage') are retained. Thus, calloc() cannot under normal circumstances assume that the data buffer returned by malloc() is already clear. Since calloc() is not usually used to allocate large chunks of memory, this isn't a problem. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Overcommit and calloc()
Kelly Yancey kby...@alcnet.com writes:
I'm afraid my question got lost amongst the see of overcommit messages. :)
I was curious if calloc() was overcommitted also?
Here's our calloc() implementation:
void *
calloc(num, size)
size_t num;
register size_t size;
{
register void *p;
size *= num;
if ( (p = malloc(size)) )
bzero(p, size);
return(p);
}
so the answer is yes, it overcommits, but the bzero() may cause the
system to run out of swap.
DES
--
Dag-Erling Smorgrav - d...@flood.ping.uio.no
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
Ahh...but wouldn't the bzero() touch all of the memory just allocated
functionally making it non-overcommit? I realize that technically it isn't
the same since although calloc() would ensure the memory actually was
committed (again, since the bzero() touches all the pages), if there weren't
enough pages free then a random process would still be killed rather than
returning null.
Believe me, I'm not trying to start that war again :) I'm just pointing
out that while technically it is still overcommit, it will touch the pages
ensuring that the memory does in fact exist (is committed). Perhaps this is
why I've always heard we should avoid calloc() because it is 'slow'?
Thanks for the great feedback, I should kick myself for now digging my
hands in the mud myself (I assumed it was more complicated than that :) ).
Kelly
~kby...@posi.net~
FreeBSD - The Power To Serve - http://www.freebsd.org/
Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD
-Original Message-
From: Dag-Erling Smorgrav [mailto:d...@flood.ping.uio.no]
Sent: Monday, July 19, 1999 12:07 PM
To: Kelly Yancey
Cc: hack...@freebsd.org
Subject: Re: Overcommit and calloc()
Kelly Yancey kby...@alcnet.com writes:
I'm afraid my question got lost amongst the see of overcommit
messages. :)
I was curious if calloc() was overcommitted also?
Here's our calloc() implementation:
void *
calloc(num, size)
size_t num;
register size_t size;
{
register void *p;
size *= num;
if ( (p = malloc(size)) )
bzero(p, size);
return(p);
}
so the answer is yes, it overcommits, but the bzero() may cause the
system to run out of swap.
DES
--
Dag-Erling Smorgrav - d...@flood.ping.uio.no
To Unsubscribe: send mail to majord...@freebsd.org
with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
Kelly Yancey kby...@alcnet.com writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an non-overcomitting malloc, it would return NULL and set errno to ENOMEM, instead of dumping core. DES -- Dag-Erling Smorgrav - d...@flood.ping.uio.no To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Overcommit and calloc()
In the last episode (Jul 19), Dag-Erling Smorgrav said: Kelly Yancey kby...@alcnet.com writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an non-overcomitting malloc, it would return NULL and set errno to ENOMEM, instead of dumping core. It should be possible to modify calloc to trap signals, then bzero. If bzero faults, you free the memory and return NULL. No, wait. You can't trap SIGKILL. How about this. mmap() some anonymous memory MAP_SHARED, fork a child to bzero it. If the child dies, unmmap and return NULL. If the child succeeds, use the memory. This memory won't be freeable with malloc(), though. -Dan Nelson dnel...@emsphone.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
-Original Message- From: Dag-Erling Smorgrav [mailto:d...@flood.ping.uio.no] Sent: Monday, July 19, 1999 12:39 PM To: Kelly Yancey Cc: Dag-Erling Smorgrav; freebsd-hackers@freebsd.org Subject: Re: Overcommit and calloc() Kelly Yancey kby...@alcnet.com writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an non-overcomitting malloc, it would return NULL and set errno to ENOMEM, instead of dumping core. I guess I was looking at it from the commitment of memory point-of-view. Calloc() calls malloc() to get the memory (which will overcommit); calloc() then touches every page of the memory using bzero(). The memory itself will be committed. The only aspect of non-overcommit that it doesn't do is kindly tell the process if it couldn't touch every page, instead the overcommit model kills a process to reclaim pages so the bzero() will succeed (assuming we're not the unfortunate process :) ). You are absolutely right, calloc() doesn't act like a non-overcommit calloc() would, but it does in fact commit the memory. I don't know how many programs make use of calloc() but would not a more efficient algorithm be to better integrate it with malloc() such that if there is a fragment on the heap which can be used, bzero() it and return that, otherwise, simply call sbrk() and return that (since it is already zero'ed). Currently, in the event that malloc() simply returns newly sbrk()'ed memory, we unnecessarily zero it again. And yes, if the majority of people think this is worthwhile and I'm not just complete wrong here, then I'm glad to submit the patches. Kelly ~kby...@posi.net~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
-Original Message- From: Dan Nelson [mailto:dnel...@emsphone.com] Sent: Monday, July 19, 1999 12:55 PM To: Dag-Erling Smorgrav Cc: Kelly Yancey; freebsd-hackers@FreeBSD.ORG Subject: Re: Overcommit and calloc() In the last episode (Jul 19), Dag-Erling Smorgrav said: Kelly Yancey kby...@alcnet.com writes: Ahh...but wouldn't the bzero() touch all of the memory just allocated functionally making it non-overcommit? No. If it were an non-overcomitting malloc, it would return NULL and set errno to ENOMEM, instead of dumping core. It should be possible to modify calloc to trap signals, then bzero. If bzero faults, you free the memory and return NULL. No, wait. You can't trap SIGKILL. How about this. mmap() some anonymous memory MAP_SHARED, fork a child to bzero it. If the child dies, unmmap and return NULL. If the child succeeds, use the memory. This memory won't be freeable with malloc(), though. Hrm. I'm not actually trying to get my system to not overcommit memory. I like overcommit. Besides, every process would have to trap SIGKILL (if you even could) to simulate a non-overcommit system because any process could be killed, not just the one requesting the memory, right? Really, I was just on an educational quest. I was curious to exactly how calloc() was implemented and what affect that had on our overcommit policy. DES was nice enough to show me that calloc() is just a malloc()+bzero() so effectively, the memory gets 'committed' because all of the pages are touched immediately after the malloc(). Whether or not a process get's shot is another matter entirely. :) I have another post on this list which begs the question: if memory given to us fro sbrk() is already zeroed, why zero it again if we don't have too if we make calloc() smarter, we could save come clock cycles. The real question is, how many? A quick scan (not exact) of the number of times calloc() is called yields: $ cd /usr/src $ t=0; grep -c -h -R calloc * | while read n; do let t+=$n; echo $t; done | tail -1 828 $ t=0; grep -c -h -R malloc * | while read n; do let t+=$n; echo $t; done | tail -1 11380 (of course, they each are a little high due to comments and the actual function definitions themselves; cavaet emptor) So calloc() is only used about 1/14th as often as malloc(), and considering many of those calloc() calls would still be serviced the same was a malloc() (reusing memory already on the heap and bzero()'ing it), I'm not 100% sure if the added complexity if worth the performance improvement. If it is, I'de be glad to whip some patches up for it (it really shouldn't be too hard). Kelly ~kby...@posi.net~ FreeBSD - The Power To Serve - http://www.freebsd.org/ Join Team FreeBSD - http://www.posi.net/freebsd/Team-FreeBSD To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Overcommit and calloc()
From: Kelly Yancey [mailto:kby...@alcnet.com] I have another post on this list which begs the question: if memory given to us fro sbrk() is already zeroed, why zero it again if we don't have too if we make calloc() smarter, we could save come clock cycles. Because the memory returned from malloc() might be from a previous malloc()/free() and may be dirty. Charles To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: RE: Overcommit and calloc()
:From: Kelly Yancey [mailto:kby...@alcnet.com] :I have another post on this list which begs the question: if memory given :to us fro sbrk() is already zeroed, why zero it again if we don't have :too if we make calloc() smarter, we could save come clock cycles. : :Because the memory returned from malloc() might be from a previous :malloc()/free() and may be dirty. : :Charles malloc() uses madvise( ... MADV_FREE) heavily in order to reduce the number of page faults that need to be taken through the course of a program's operation. MADV_FREE is an advisory free that causes FreeBSD to mark the underlying page(s) clean and to free any associated swap backing store. The kernel avoids actually freeing the page until it needs the memory, and the process can re-dirty the page to keep it. No new page-faults occur if the kernel has not reclaimed the page at the time the process reuses it. If the kernel reclaims the page first, the kernel replaces it with zero-fill. If the process reclaims the page first, the page's previous contents (considered to be 'garbage') are retained. Thus, calloc() cannot under normal circumstances assume that the data buffer returned by malloc() is already clear. Since calloc() is not usually used to allocate large chunks of memory, this isn't a problem. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
