On 28 June 2012 09:28, Glenn Fowler <g...@research.att.com> wrote: > > On Wed, 27 Jun 2012 12:55:56 +0200 Cedric Blancher wrote: >> On 21 June 2012 08:18, Roland Mainz <roland.ma...@nrubsig.org> wrote: >> > Hi! >> > >> > ---- >> > >> > Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo >> > which shows a problem in ast-ksh.2012-06-12 with memory usage for >> > str="$( < bigfile )" (and similar usage). The problem seems to be that >> > a larger input file size (for example 32MB) triggers excessive memory >> > usage, usually well beyond 512MB on Solaris 11/AMD64. >> > >> > Running the demo triggers the following error (this comes from the >> > limits defined via "ulimit" in the test (e.g. we assume that a file >> > size of 32MB will not cause more than 64MB of extra memory usage in >> > ksh93 (ksh93 itself is granted 16MB for itself&&running the script, >> > e.g. the ulimit limits are 80MB total in this case. However even that >> > is very very generous)): >> > -- snip -- >> > $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh >> > -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile >> > bigfilecommandsubstitution001.sh: line 36: storage allocator out of >> > space on 8486912 byte request ( region 545751040 segments 133 busy >> > 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space] >> > -- snip -- >> > >> > >> > ---- >> > >> > Bye, >> > Roland >> > >> > -- >> > __ . . __ >> > (o.\ \/ /.o) roland.ma...@nrubsig.org >> > \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer >> > /O /==\ O\ TEL +49 641 3992797 >> > (;O/ \/ \O;) >> > >> > _______________________________________________ >> > ast-developers mailing list >> > ast-developers@research.att.com >> > https://mailman.research.att.com/mailman/listinfo/ast-developers >> > > >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO >> showstopper because it prevents the use of larger texts in command >> substitutions: > >> ksh bigfile.sh >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free >> 55:63535184:2850720 ) [Cannot allocate memory] > > this is a vmalloc fragmentation issue when mmap() is used to allocate region > blocks > coalescing adjacent mmap() blocks is not a portable op across mmap() > implementations > a side effect of this is fragmentation, sometimes very bad
BTW: Linux has mremap(), but that is mostly used for IO and not for allocations. > > using mmap() instead of sbrk() by default started with the first thread-safe > vmalloc > implementation earlier this year; the ast beta 2012-06-28, to be posted later > today, > works around this by detecting if the current application requires > thread-safety > (via the ast aso(3) api); if it does require thread-safe then mmap() is used, > otherwise it uses sbrk() (this is the case for all current ast commands), > falling > back to mmap() only if sbrk() fails > > the next iteration of thread-safe vmalloc is currently in test phase > it will use sbrk() by default in a thread-safe way (only restriction - no api > but vmalloc(3) may call sbrk()), and the workaround above will not be needed Good lord. Glenn, please do not use sbrk() in a shared library. This is not going to work well and will have serious consequences for stability if *ANYTHING* else uses sbrk(). We already had such issues in other shared libraries and they were almost impossible to find and were always not readily reproducible. Lionel FYI, we always patched ast-open and removed the sbrk() code (no, we did not use #if/#endif, we always removed the whole code to avoid having the slightest risk of running into bugs caused by multiple wannabe-tyrant owners of the sbrk() pool.) _______________________________________________ ast-developers mailing list ast-developers@research.att.com https://mailman.research.att.com/mailman/listinfo/ast-developers
