Re: Taking advantage of L1 and L2 cache in sort

2012-06-18 Thread Chen Guo
, but either way if desired I can make the change to read MALLOC_PERTURB_ from env[] and set the buffer size accordingly based on that. On Thu, Jun 7, 2012 at 5:14 PM, Pádraig Brady wrote: > On 03/03/2010 01:06 AM, Pádraig Brady wrote: > > On 02/03/10 18:20, Chen Guo wrote: > >> Th

Re: [coreutils] Memory usage of parallel sort

2010-12-17 Thread Chen Guo
Hi Assaf, > On 17/12/10 19:29, Assaf Gordon wrote: >> Hello, >> >> A question regarding the memory usage requirements of the parallel sort: >> It seems that the memory usage (resident size) increases with the number of >> threads used. >> >> It also seems to me (but not verified) that the increase

[coreutils] Re: bug#7597: multi-threaded sort can segfault (unrelated to the sort -u segfault)

2010-12-10 Thread Chen Guo
> +  size_t nlines = (lo_child)? node->parent->nlo : node->parent->nhi; Small change... Remove unnecessary branch and redirection in sortlines: size_t nlines = node->nlo + node->nhi; >From c691813ecbfce60c207960fd8bd327cc7d99fe39 Mon Sep 17 00:00:00 2001 From: Che

[coreutils] Re: bug#7597: multi-threaded sort can segfault (unrelated to the sort -u segfault)

2010-12-10 Thread Chen Guo
7 00:00:00 2001 From: Chen Guo Date: Fri, 10 Dec 2010 13:13:36 -0800 Subject: [PATCH] sort: preallocate merge tree nodes to heap. * src/sort.c: (merge_tree_init) New function. Allocates memory for merge tree nodes. (merge_tree_destory) New function. (init_node) New function. (sortlines) Refactor n

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-07 Thread Chen Guo
Hi Jim, On Tue, Dec 7, 2010 at 3:24 AM, Jim Meyering wrote: > Hi Chen, > > Thanks.  What does your input file look like? > I've been unable to reproduce the failure using the output of > seq 100.  I've tried a few different -S ... options, in case > the amount of available memory is a factor:

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-06 Thread Chen Guo
Hi Professor Eggert, On Sun, Dec 5, 2010 at 11:01 PM, Paul Eggert wrote: > On 12/05/2010 09:16 PM, Chen Guo wrote: >> Before saying anything else, I should note that for mutexes, on 4 >> threads 20% of the time there's a segfault on a seemingly innocuous >> line in

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-05 Thread Chen Guo
Hi Professor Eggert, On Fri, Dec 3, 2010 at 1:10 PM, Paul Eggert wrote: > On 12/03/10 12:18, Chen Guo wrote: > Either option (either switch to mutexes everywhere, or have the top-level > merge go to memory) should work.  Perhaps we should try both and benchmark > them. Test

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-03 Thread Chen Guo
Thanks Jim, that helped a lot. I'll try out Professor Eggert's suggestion, of switching to mutexes only at the top level merge. Of the following approaches, which would you guys consider better practice? 1) void pointer, cast as either mutex or spinlock in lock function 2) union of mutex and spin

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-02 Thread Chen Guo
Hi Professor Eggert, On Mon, Nov 29, 2010 at 11:16 AM, Paul Eggert wrote: >  (for i in $(seq 12); do read line; echo $i; sleep .1; done >  cat > /dev/null) < fifo & >  (ulimit -t 1; ./sort in > fifo \ >  || echo killed via $(env kill -l $(expr $? - 128))) I ran this 10 times or so on an i7 and co

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-01 Thread Chen Guo
Hi guys, Is something up with Savannah? I just tried a git clone and got connection time out; I cant even reach git.sv.gnu.org via ping. I'll try again at work tomorrow.

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-01 Thread Chen Guo
On Tue, Nov 30, 2010 at 2:22 PM, Paul Eggert wrote: Hi Professor Eggert, > Anyway, perhaps Chen can review them (I don't have time > to test them right now). I'll look at it as soon as Savannah's back up; I never actually pulled from coreutils after the original patch was submitted. Professor Egge

Re: bug#7489: [coreutils] over aggressive threads in sort

2010-12-01 Thread Chen Guo
Hi all, On Mon, Nov 29, 2010 at 11:16 AM, Paul Eggert wrote: > entirely and use mutexes instead.  Perhaps a better fix would be to > use mutexes at the top level (where threads can write to a file and > therefore can wait) and to use spin locks at lower levels (where > threads are merely storing