On Thu, Jan 25, 2018 at 11:26:24AM -0500, Joe Sapp wrote: > Hi depesz, > > On Thu, Jan 25, 2018 at 10:33 AM, hubert depesz lubaczewski > <[email protected]> wrote: > [snip] > > But it looks that parallel itself is consumming HUGE amount of memory > > - comparable with size of /some/directory itself. > > > > Server that I run it on has 64GB of ram, and the script gets killed > > after ~ 3 minutes with "Out of memory!" error. > > Have you tried using --memfree? That will limit parallel from > spawning new processes
OK, I read the docs, and I don't get it. Why would I want to limit parallel from spawning new processes? The problem, in my opinion, is that parallel is using memory for some kind of buffer when splitting data, instead of operating in some small block size, like 8kb, or whatever to copy data form input to one of the outputs. The final scripts do not use lots of ram. It's just parallel that is using over 10GB of ram to split data and put it into multiple pipes. This doesn't look OK to me - isn't it a bug? Best regards, depesz
