Hi Joe,
Please see the following line
https://github.com/apache/nutch/blob/2.x/src/bin/crawl#L65\
and also
https://github.com/apache/nutch/blob/2.x/src/bin/crawl#L91
then
https://github.com/apache/nutch/blob/2.x/src/bin/crawl#L158
reduce tasks are directly aligned with nodes/2
What kind of set up it is that you are using? Apologies for late reply,
this digest got lot.
Lewis

On Wed, May 18, 2016 at 11:46 AM, <user-digest-h...@nutch.apache.org> wrote:

>
>
> From: Joseph Obernberger <joseph.obernber...@gmail.com>
> To: user@nutch.apache.org, tien.nguyenm...@gmail.com
> Cc:
> Date: Mon, 16 May 2016 14:53:17 -0400
> Subject: Re: Nutch 2.3.1 - Fetch Phase - Only 2 Reducers
> Thank you Nguyen.  I tried modifying the yarn settings to no avail.  Other
> jobs, like the generate phase, create 128 reducers no problem.  It's only
> the fetch phase that always makes 2 reducers.  Running multiple jobs does
> work, and does increase throughput, but I'm still curious as to why I'm
> only getting 2 reducers in the fetch phase only.
> Thanks!
>
> -Joe
>
>

Reply via email to