Pádraig Brady wrote:
> On 16/03/11 15:32, Jim Meyering wrote:
>> Pádraig Brady wrote:
>>>
>>> With SUBTHREAD_LINES_HEURISTIC=128k and -S1M option to sort we get
>>> no threads as
>>> nlines never gets above 12787 (there looks to be around 80 bytes
>>> overhead per line?).
>>> Only when -S >= 12M do we get nlines high enough to create threads.
>>
>> Thanks for pursuing this.
>> Here's a proposed patch to address the other problem.
>> It doesn't have much of an effect (any?) on your
>> issue when using very little memory, but when a sort user
>> specifies -S1M, I think they probably want to avoid the
>> expense (memory) of going multi-threaded.
>>
>> What do you think?
>>
>> -#define INPUT_FILE_SIZE_GUESS (1024 * 1024)
>> +#define INPUT_FILE_SIZE_GUESS (128 * 1024)
>
> This does seem a bit like whack-a-mole
> but at least we're lining them up better.
>
> The above gives reasonable threading by default,
> while reducing the large upfront malloc.
>
> $ for len in 1 79; do
> for i in $(seq 22); do
> lines=$((2<<$i))
> yes "$(printf %${len}s)"| head -n$lines > t.sort
> strace -f -c -e clone ./sort --parallel=16 t.sort -o /dev/null 2>&1 |
> join --nocheck-order -a1 -o1.4,1.5 - /dev/null |
> sed -n "s/\([0-9]*\) clone/$lines\t\1/p"
> done
> done
>
> #lines threads (2 byte lines)
> ------------------------------
> 131072 1
> 262144 3
> 524288 7
> 1048576 15
> 2097152 15
> 4194304 15
> 8388608 15
>
> #lines threads (80 byte lines)
> ------------------------------
> 131072 1
> 262144 3
> 524288 7
> 1048576 15
> 2097152 15
> 4194304 22
> 8388608 60
Thanks for testing and review.
Pushed.
