On Thu, 28 Jun 2012 11:12:21 +0200 Roland Mainz wrote: > On Thu, Jun 28, 2012 at 9:28 AM, Glenn Fowler <g...@research.att.com> wrote: > > > > On Wed, 27 Jun 2012 12:55:56 +0200 Cedric Blancher wrote: > >> On 21 June 2012 08:18, Roland Mainz <roland.ma...@nrubsig.org> wrote: > >> > Hi! > >> > > >> > ---- > >> > > >> > Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo > >> > which shows a problem in ast-ksh.2012-06-12 with memory usage for > >> > str="$( < bigfile )" (and similar usage). The problem seems to be that > >> > a larger input file size (for example 32MB) triggers excessive memory > >> > usage, usually well beyond 512MB on Solaris 11/AMD64. > >> > > >> > Running the demo triggers the following error (this comes from the > >> > limits defined via "ulimit" in the test (e.g. we assume that a file > >> > size of 32MB will not cause more than 64MB of extra memory usage in > >> > ksh93 (ksh93 itself is granted 16MB for itself&&running the script, > >> > e.g. the ulimit limits are 80MB total in this case. However even that > >> > is very very generous)): > >> > -- snip -- > >> > $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh > >> > -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile > >> > bigfilecommandsubstitution001.sh: line 36: storage allocator out of > >> > space on 8486912 byte request ( region 545751040 segments 133 busy > >> > 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space] > >> > -- snip -- > >> > > >> > > >> > ---- > >> > > >> > Bye, > >> > Roland > >> > > >> > -- > >> > __ . . __ > >> > (o.\ \/ /.o) roland.ma...@nrubsig.org > >> > \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer > >> > /O /==\ O\ TEL +49 641 3992797 > >> > (;O/ \/ \O;) > >> > > >> > _______________________________________________ > >> > ast-developers mailing list > >> > ast-developers@research.att.com > >> > https://mailman.research.att.com/mailman/listinfo/ast-developers > >> > > > > >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO > >> showstopper because it prevents the use of larger texts in command > >> substitutions: > > > >> ksh bigfile.sh > >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile > >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte > >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free > >> 55:63535184:2850720 ) [Cannot allocate memory] > > > > this is a vmalloc fragmentation issue when mmap() is used to allocate > > region blocks > > coalescing adjacent mmap() blocks is not a portable op across mmap() > > implementations > > a side effect of this is fragmentation, sometimes very bad > > > > using mmap() instead of sbrk() by default started with the first > > thread-safe vmalloc > > implementation earlier this year; the ast beta 2012-06-28, to be posted > > later today, > > works around this by detecting if the current application requires > > thread-safety > > (via the ast aso(3) api); if it does require thread-safe then mmap() is > > used, > > otherwise it uses sbrk() (this is the case for all current ast commands), > > falling > > back to mmap() only if sbrk() fails > > > > the next iteration of thread-safe vmalloc is currently in test phase > > it will use sbrk() by default in a thread-safe way (only restriction - no > > api > > but vmalloc(3) may call sbrk()), and the workaround above will not be needed > > > > 2012-06-26 beta vmalloc (and thus malloc()) can be forced to use sbrk() > > first by > > export VMALLOC_OPTIONS=break > > use this workaround until the next beta is posted
> Erm... this does not help with the following issue: > -- snip -- > $ dash -c 'x=$( bzcat </tmp/freedesktop.org.xml.bz2 ) ; printf "%s\n" > "$x" | wc -c -w -l' > 31361 102734 1713511 > $ bash -c 'x=$( bzcat </tmp/freedesktop.org.xml.bz2 ) ; printf "%s\n" > "$x" | wc -c -w -l' > 31361 102734 1713511 > $ ~/bin/ksh -c 'x=$( bzcat </tmp/freedesktop.org.xml.bz2 ) ; printf > "%s\n" "$x" | wc -c -w -l' > 9434 31534 524289 correct, I didn't say it did if you do an strace/truss something strange is going on with read() it gets to a point of doing 1 byte read and then gets to succesive reads with 0 bytes returned _______________________________________________ ast-developers mailing list ast-developers@research.att.com https://mailman.research.att.com/mailman/listinfo/ast-developers
