On Thu, Jun 28, 2012 at 9:28 AM, Glenn Fowler <[email protected]> wrote:
>
> On Wed, 27 Jun 2012 12:55:56 +0200 Cedric Blancher wrote:
>> On 21 June 2012 08:18, Roland Mainz <[email protected]> wrote:
>> > Hi!
>> >
>> > ----
>> >
>> > Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo
>> > which shows a problem in ast-ksh.2012-06-12 with memory usage for
>> > str="$( < bigfile )" (and similar usage). The problem seems to be that
>> > a larger input file size (for example 32MB) triggers excessive memory
>> > usage, usually well beyond 512MB on Solaris 11/AMD64.
>> >
>> > Running the demo triggers the following error (this comes from the
>> > limits defined via "ulimit" in the test (e.g. we assume that a file
>> > size of 32MB will not cause more than 64MB of extra memory usage in
>> > ksh93 (ksh93 itself is granted 16MB for itself&&running the script,
>> > e.g. the ulimit limits are 80MB total in this case. However even that
>> > is very very generous)):
>> > -- snip --
>> > $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh
>> > -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile
>> > bigfilecommandsubstitution001.sh: line 36: storage allocator out of
>> > space on 8486912 byte request ( region 545751040 segments 133 busy
>> > 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space]
>> > -- snip --
>> >
>> >
>> > ----
>> >
>> > Bye,
>> > Roland
>> >
>> > --
>> > __ . . __
>> > (o.\ \/ /.o) [email protected]
>> > \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
>> > /O /==\ O\ TEL +49 641 3992797
>> > (;O/ \/ \O;)
>> >
>> > _______________________________________________
>> > ast-developers mailing list
>> > [email protected]
>> > https://mailman.research.att.com/mailman/listinfo/ast-developers
>> >
>
>> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO
>> showstopper because it prevents the use of larger texts in command
>> substitutions:
>
>> ksh bigfile.sh
>> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile
>> bigfile.sh: line 35: storage allocator out of space on 2981888 byte
>> request ( region 66813952 segments 49 busy 169:3271264:2916256 free
>> 55:63535184:2850720 ) [Cannot allocate memory]
>
> this is a vmalloc fragmentation issue when mmap() is used to allocate region
> blocks
> coalescing adjacent mmap() blocks is not a portable op across mmap()
> implementations
> a side effect of this is fragmentation, sometimes very bad
>
> using mmap() instead of sbrk() by default started with the first thread-safe
> vmalloc
> implementation earlier this year; the ast beta 2012-06-28, to be posted later
> today,
> works around this by detecting if the current application requires
> thread-safety
> (via the ast aso(3) api); if it does require thread-safe then mmap() is used,
> otherwise it uses sbrk() (this is the case for all current ast commands),
> falling
> back to mmap() only if sbrk() fails
>
> the next iteration of thread-safe vmalloc is currently in test phase
> it will use sbrk() by default in a thread-safe way (only restriction - no api
> but vmalloc(3) may call sbrk()), and the workaround above will not be needed
>
> 2012-06-26 beta vmalloc (and thus malloc()) can be forced to use sbrk() first
> by
> export VMALLOC_OPTIONS=break
> use this workaround until the next beta is posted
Erm... this does not help with the following issue:
-- snip --
$ dash -c 'x=$( bzcat </tmp/freedesktop.org.xml.bz2 ) ; printf "%s\n"
"$x" | wc -c -w -l'
31361 102734 1713511
$ bash -c 'x=$( bzcat </tmp/freedesktop.org.xml.bz2 ) ; printf "%s\n"
"$x" | wc -c -w -l'
31361 102734 1713511
$ ~/bin/ksh -c 'x=$( bzcat </tmp/freedesktop.org.xml.bz2 ) ; printf
"%s\n" "$x" | wc -c -w -l'
9434 31534 524289
$ VMALLOC_OPTIONS=break ~/bin/ksh -c 'x=$( bzcat
</tmp/freedesktop.org.xml.bz2 ) ; printf "%s\n" "$x" | wc -c -w -l'
9434 31534 524289
-- snip --
... I have the feeling we have two issues... first the
|mmap(...,MAP_ANON,...)|-allocator going "mad" and a 2nd which causes
silent cut-offs, e.g. loss of data in a command substitution...
... I dug a bit deeper and found that using VMALLOC_OPTIONS=break
somehow reveals a couple of new valgrind hits which may indicate the
cause of the issue:
-- snip --
> VMALLOC_OPTIONS=break valgrind ~/bin/ksh -c 'x=$( bzcat
> </tmp/freedesktop.org.xml.bz2 ) ; printf "%s\n" "$x" | wc -c -w -l'
==21409== Memcheck, a memory error detector
==21409== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==21409== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
[snip]
==21409== Conditional jump or move depends on uninitialised value(s)
==21409== at 0x4566B8: comsubst (macro.c:2205)
==21409== by 0x4521B1: varsub (macro.c:1168)
==21409== by 0x45079F: copyto (macro.c:633)
==21409== by 0x44EBD1: sh_mactrim (macro.c:183)
==21409== by 0x458AB5: nv_setlist (name.c:339)
==21409== by 0x47FF94: sh_exec (xec.c:1208)
==21409== by 0x483B04: sh_exec (xec.c:2195)
==21409== by 0x418E2B: exfile (main.c:581)
==21409== by 0x417FFE: sh_main (main.c:353)
==21409== by 0x417128: main (pmain.c:45)
==21409==
==21409== Conditional jump or move depends on uninitialised value(s)
==21409== at 0x4C29F18: strlen (mc_replace_strmem.c:390)
==21409== by 0x45DEDF: nv_putval (name.c:1882)
==21409== by 0x45CFEC: nv_open (name.c:1547)
==21409== by 0x459C8E: nv_setlist (name.c:616)
==21409== by 0x47FF94: sh_exec (xec.c:1208)
==21409== by 0x483B04: sh_exec (xec.c:2195)
==21409== by 0x418E2B: exfile (main.c:581)
==21409== by 0x417FFE: sh_main (main.c:353)
==21409== by 0x417128: main (pmain.c:45)
[snip]
9434 31534 524289
==21409==
==21409== HEAP SUMMARY:
-- snip --
----
Bye,
Roland
--
__ . . __
(o.\ \/ /.o) [email protected]
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
_______________________________________________
ast-developers mailing list
[email protected]
https://mailman.research.att.com/mailman/listinfo/ast-developers
