On 12/6/07, Nicolas Pitre <[EMAIL PROTECTED]> wrote: > On Thu, 6 Dec 2007, Jon Smirl wrote: > > > On 12/6/07, Nicolas Pitre <[EMAIL PROTECTED]> wrote: > > > > When I lasted looked at the code, the problem was in evenly dividing > > > > the work. I was using a four core machine and most of the time one > > > > core would end up with 3-5x the work of the lightest loaded core. > > > > Setting pack.threads up to 20 fixed the problem. With a high number of > > > > threads I was able to get a 4hr pack to finished in something like > > > > 1:15. > > > > > > But as far as I know you didn't try my latest incarnation which has been > > > available in Git's master branch for a few months already. > > > > I've deleted all my giant packs. Using the kernel pack: > > 4GB Q6600 > > > > Using the current thread pack code I get these results. > > > > The interesting case is the last one. I set it to 15 threads and > > monitored with 'top'. > > For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and > > 74-100% was 100% CPU. It never used all for cores. The only other > > things running were top and my desktop. This is the same load > > balancing problem I observed earlier. > > Well, that's possible with a window 25 times larger than the default.
Why did it never use more than three cores? > > The load balancing is solved with a master thread serving relatively > small object list segments to any work thread that finished with its > previous segment. But the size for those segments is currently fixed to > window * 1000 which is way too large when window == 250. > > I have to find a way to auto-tune that segment size somehow. > > But with the default window size there should not be any such noticeable > load balancing problem. > > Note that threading only happens in the compression phase. The count > and write phase are hardly paralleled. > > > Nicolas > -- Jon Smirl [EMAIL PROTECTED]
