On Mon, Dec 3, 2012 at 11:36 PM, Harry B <[email protected]> wrote: > Hello, > > I tried the parallel grep example on a sample file (1.4G and 3,153,199 > lines). The parallel grep > (http://www.gnu.org/software/parallel/man.html#example__parallel_grep) is > considerably slower - I am trying to see where the bottle neck is. Is there > an easy way for me to guess which way would be better based on the size of > the file?
GNU Parallel is the bottleneck. It can only move 50-100 MB/s in --pipe mode. So while the example without --pipe may speed up your grep, then the second part with --pipe most likely will not. That should probably be stressed in the documentation that the second part is only to show how it can be done, and not that it will actually run faster. /Ole
