On Thu, 28 Jun 2012 13:43:52 +0200 Irek Szczesniak wrote: > On Thu, Jun 28, 2012 at 9:28 AM, Glenn Fowler <g...@research.att.com> wrote: > > > > On Wed, 27 Jun 2012 12:55:56 +0200 Cedric Blancher wrote: > >> On 21 June 2012 08:18, Roland Mainz <roland.ma...@nrubsig.org> wrote: > >> > Hi! > >> > > >> > ---- > >> > > >> > Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo > >> > which shows a problem in ast-ksh.2012-06-12 with memory usage for > >> > str="$( < bigfile )" (and similar usage). The problem seems to be that > >> > a larger input file size (for example 32MB) triggers excessive memory > >> > usage, usually well beyond 512MB on Solaris 11/AMD64. > >> > > >> > Running the demo triggers the following error (this comes from the > >> > limits defined via "ulimit" in the test (e.g. we assume that a file > >> > size of 32MB will not cause more than 64MB of extra memory usage in > >> > ksh93 (ksh93 itself is granted 16MB for itself&&running the script, > >> > e.g. the ulimit limits are 80MB total in this case. However even that > >> > is very very generous)): > >> > -- snip -- > >> > $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh > >> > -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile > >> > bigfilecommandsubstitution001.sh: line 36: storage allocator out of > >> > space on 8486912 byte request ( region 545751040 segments 133 busy > >> > 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space] > >> > -- snip -- > >> > > >> > > >> > ---- > >> > > >> > Bye, > >> > Roland > >> > > >> > -- > >> > __ . . __ > >> > (o.\ \/ /.o) roland.ma...@nrubsig.org > >> > \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer > >> > /O /==\ O\ TEL +49 641 3992797 > >> > (;O/ \/ \O;) > >> > > >> > _______________________________________________ > >> > ast-developers mailing list > >> > ast-developers@research.att.com > >> > https://mailman.research.att.com/mailman/listinfo/ast-developers > >> > > > > >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO > >> showstopper because it prevents the use of larger texts in command > >> substitutions: > > > >> ksh bigfile.sh > >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile > >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte > >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free > >> 55:63535184:2850720 ) [Cannot allocate memory] > > > > this is a vmalloc fragmentation issue when mmap() is used to allocate > > region blocks > > coalescing adjacent mmap() blocks is not a portable op across mmap() > > implementations > > a side effect of this is fragmentation, sometimes very bad > > > > using mmap() instead of sbrk() by default started with the first > > thread-safe vmalloc > > implementation earlier this year; the ast beta 2012-06-28, to be posted > > later today, > > works around this by detecting if the current application requires > > thread-safety > > (via the ast aso(3) api); if it does require thread-safe then mmap() is > > used, > > otherwise it uses sbrk() (this is the case for all current ast commands), > > falling > > back to mmap() only if sbrk() fails
> So how is this going to work for applications try to use libast but > are not ksh93? sbrk() is a poor choice if more than one allocator > tries to be in charge of it, which is alas the majority of today's > applications which depend on more than one toolkit (of which libast is > only one, but a major one). For example you can't use libast in > single-threaded Gnome or KDE applications when libast uses the sbrk() > allocator, they'll just crash and burn sooner or later. up to 2012 ast had been using sbrk() if mmap() implementations can't support coalescing efficiently (or at all) then we have to fall back to something that works > > the next iteration of thread-safe vmalloc is currently in test phase > > it will use sbrk() by default in a thread-safe way (only restriction - no > > api > > but vmalloc(3) may call sbrk()), and the workaround above will not be needed > That's IMO both useless and the wrong way to go. It might work for > ksh93 and the AST tools, but using sbrk() in any other context is not > going to work because GNU libc and other libc implementations assume > they are the sole users of sbrk(). Forget that very quickly or you'll > drown in bug reports for the rest of the decade. if -last is in the mix we expect that the ast vmalloc malloc family is the allocator modulo the thread-safe growing pains this year this has served ast and its consmers well for gnu we use the gnu malloc hooks if the hooks are implemented correctly by gnu and used correctly by ast then this will not be a problem on other systems similar hacks/intercepts are employed basically once the first ast vmalloc/malloc call is hit all subsequent calls are expected to be ast calls -- this mechanism even works on windows where they pull in a dll just to do 1+1=2 its not pretty, but the whole reason for bringing in -last is to take advantage of the flexibility of vmalloc and sfio and the other discipline/method apis -- if one of those apis starts misbehaving in un-planned-for circumstances we don't have to rip -last out -- we study the problem and either tweak the existing methods or provide a new method to attack the problem -- all with out having to recompile (and with shared/dll -last relink) user apps > > 2012-06-26 beta vmalloc (and thus malloc()) can be forced to use sbrk() > > first by > > export VMALLOC_OPTIONS=break > > use this workaround until the next beta is posted > We tried that this morning. The warnings go away but the problems > remain. Also we've noticed a sever degradation in performance, i.e. a > simple quicksort testcase implemented in ksh runs without > VMALLOC_OPTIONS=break in less then 5 seconds but takes almost a minute > with VMALLOC_OPTIONS=break. vmstate is a -lcmd builtin that lists the ast vmalloc (ast malloc uses vmalloc) state for all regions -- run that at the end of the script that's misbehaving the size=* value is the total number of bytes in all active regions see how that value relates to you measurement run strace/truss -c and -t to see where the time is going share the test case so we can reproduce what you are seeing > > note that some of the tests with mem ulimits may be set low because of > > over-optimistic > > expectations on how the shell works internally; the tests contain > > > > x=$(<bigfile) > > > > shell execution sees this as two operations > > (1) $(<bigfile) is read in chunks on to the shell's internal stack > > (2) then the value on the stack is assigned to x by making a > > non-stack copy of the value and assiging it to x > > so the test ulimits need to be ( initial-mem-usage + 2 * bytes-in-bigfile ) > The math seems to be wrong. It doesn't explain why 32MB of input data > require multiple gigabytes to process. Using your formula I'd expect > (8MB + 2*32MB)=72MB. OK, lets be conservative and use (128MB + > 8*32MB)=384MB, but this is nowhere of the real 2GB+ usage we see right > now. again we are not seeing those number based on the vmstate measurements that report the exact memory allocated by vmalloc how are you measuring memory consumption? there may be a disconnect between the allocations requested and what was physically allocated by the system _______________________________________________ ast-developers mailing list ast-developers@research.att.com https://mailman.research.att.com/mailman/listinfo/ast-developers
