Hi Glen, * glen lenker wrote on Fri, Jun 19, 2009 at 01:03:10PM CEST: > On Fri, Jun 12, 2009 at 12:54 AM, Ralf Wildenhues wrote: > > > > No, I did not specify the amount of RAM. The system I tested on has > > plenty of RAM, way more than is needed for the sort. Specifying > > something like 2G of RAM does not make any visible difference. > > When testing on gcc16, 16G of RAM, sort chooses to use only 1G of > RAM for a 256M file, and ends up hitting the external merge code. It > was through experimentation with gprof that I found that anything less > than 2G causes sort to do an external sort on gcc16.
Hmm, ok. I did retest with 2G at the time, but I didn't check logs for whether the external merge code was used. I can try to recheck sometime (systems are busy ATM). > > > > 160.22user 1.34system 2:41.61elapsed 99%CPU > > > > 159.83user 1.45system 1:27.12elapsed 185%CPU > > > > 159.84user 1.56system 0:52.26elapsed 308%CPU > > > > 160.67user 1.53system 0:36.26elapsed 447%CPU > > > > > > > > This seems to be what I would expect from a good implementation. > > > > Yes, 56% efficiency going from 1 to 8 threads sounds like a much better > > number, also your system overhead looks very sane compared to what I > > saw. Seems like it's a system-specific issue after all. Which Linux > > kernel version and which pthread (glibc) version are you using? > > This was on gcc16. > Linux gcc16 2.6.18-6-vserver-amd64 #1 SMP Thu Dec 25 21:35:11 UTC 2008 > x86_64 GNU/Linux > glibc 2.3.6.ds1-13etch9 here: Linux 2.6.24-23-generic GNU C Library stable release version 2.7, by Roland McGrath et al. > > Glen, could you look at this, too? I mean just timings of 1 vs 2 > > threads for different file sizes? Thanks. > > The data was again cat'ed instances of the dictionary, put onto tmpfs. I > made sure that each time it had enough memory. > > 1st run > memory 1 thread 2 threads speedup > 16M 7.27 4.04 ~1.8 > 32M 16.15 9.19 ~1.75 > 64M 35.06 19.61 ~1.78 > 128M 75.47 40.99 ~1.84 > 256M 162.30 87.27 ~1.86 > > 2nd run > memory 1 thread 2 threads speedup > 16M 7.27 4.05 ~1.8 > 32M 16.15 8.86 ~1.82 > 64M 35.07 19.18 ~1.83 > 128M 75.22 41.28 ~1.82 > 256M 161.76 87.77 ~1.84 Well, that is a small increase and probably below noise level, but at least there is no decrease, and it looks fairly consistent, so I think that is good. > On this subject, what exactly is the motivation of sort's > default_sort_size()? > Shouldn't it try to do an internal sort if possible? Reading the change logs, I think it was added to avoid overcommitting memory (and thus hitting the chance of being killed later by an OOM situation). Cheers, Ralf _______________________________________________ Bug-coreutils mailing list [email protected] http://lists.gnu.org/mailman/listinfo/bug-coreutils
