As always, Scott provided lots of great advice below. One note to be aware
of:
The fair scheduler "assignmultiple" feature in 0.20 doesn't do quite what
you think it might. It gives the ability to assign one map and one reduce
per TT heartbeat, but doesn't assign multiple map tasks in a single
hea
On Wed, Dec 9, 2009 at 2:00 PM, Scott Carey wrote:
>
>
>
> On 12/8/09 1:24 PM, "Vasilis Liaskovitis" wrote:
>
>> Hi Scott,
>>
>> thanks for the extra tips, these are very helpful.
>>
>>>
>>
>> I think the slots are being highly utilized, but I seem to have
>> forgotten which option in the web UI
On 12/8/09 1:24 PM, "Vasilis Liaskovitis" wrote:
> Hi Scott,
>
> thanks for the extra tips, these are very helpful.
>
>>
>
> I think the slots are being highly utilized, but I seem to have
> forgotten which option in the web UI allows you to look at the slot
> allocations during runtime on
Hi Scott,
thanks for the extra tips, these are very helpful.
On Mon, Dec 7, 2009 at 3:57 PM, Scott Carey wrote:
>
>>
>> I am using hadoop-0.20.1 to run terasort and randsort benchmarking
>> tests on a small 8-node linux cluster. Most runs consist of usually
>> low (<50%) core utilizations in the
On 12/2/09 12:22 PM, "Vasilis Liaskovitis" wrote:
> Hi,
>
> I am using hadoop-0.20.1 to run terasort and randsort benchmarking
> tests on a small 8-node linux cluster. Most runs consist of usually
> low (<50%) core utilizations in the map and reduce phase, as well as
> heavy I/O phases . There
On Wed, Dec 2, 2009 at 4:37 PM, Vasilis Liaskovitis wrote:
> Hi Todd,
>
> thanks for the reply.
>
> >
> > This is seen reasonably often, and could be partly due to missed
> > configuration changes. A few things to check:
> >
> > - Did you increase the number of tasks per node from the default? If
Hi Todd,
thanks for the reply.
>
> This is seen reasonably often, and could be partly due to missed
> configuration changes. A few things to check:
>
> - Did you increase the number of tasks per node from the default? If you
> have a reasonable number of disks/cores, you're going to want to run a
Hi Vasilis,
This is seen reasonably often, and could be partly due to missed
configuration changes. A few things to check:
- Did you increase the number of tasks per node from the default? If you
have a reasonable number of disks/cores, you're going to want to run a lot
more than 2 map and 2 redu
Hi,
I am using hadoop-0.20.1 to run terasort and randsort benchmarking
tests on a small 8-node linux cluster. Most runs consist of usually
low (<50%) core utilizations in the map and reduce phase, as well as
heavy I/O phases . There is usually a large fraction of runtime for
which cores are idling
