Hi,
I am seeing some strange behavior in Hadoop - I am running a small test
cluster with a capacity of 18 mappers and 18 reducers. I fire a lot of jobs
simultaneously and over time I have observed Hadoop is not utilizing all the
18 slots for the reducers.
And now even if I run just one job (no
other reducers had
finished.
On Wed, Aug 18, 2010 at 2:44 PM, Tarandeep Singh tarand...@gmail.comwrote:
Hi,
I am seeing some strange behavior in Hadoop - I am running a small test
cluster with a capacity of 18 mappers and 18 reducers. I fire a lot of jobs
simultaneously and over time I have
On Mon, Jan 18, 2010 at 2:52 AM, Steve Loughran ste...@apache.org wrote:
Tarandeep Singh wrote:
Hi,
I am running a MR job that requires usage of some java.awt.* classes, that
can't be run in headless mode.
Right now, I am running Hadoop in a single node cluster (my laptop) which
has X11
Hi,
I am running a MR job that requires usage of some java.awt.* classes, that
can't be run in headless mode.
Right now, I am running Hadoop in a single node cluster (my laptop) which
has X11 server running. I have set up my ssh server and client to do X11
forwarding.
I ran the following java
either some permission
issues or what)
On Jan 18, 2010, at 12:41 AM, Tarandeep Singh wrote:
Hi,
I am running a MR job that requires usage of some java.awt.* classes,
that
can't be run in headless mode.
Right now, I am running Hadoop in a single node cluster (my laptop) which
has
The output of mappers is partitioned, each partition is given a number
starting from 0 and a reducer works on one of these partitions. In the
configure method of your reducer code, you can get the partition number by-
jobConf.getInt( mapred.task.partition, 0);
If you use the default output
A similar question-
If in an N node cluster, a file's replication is set to N (replicate on each
node) and later if a node goes down, will HDFS throw an exception since the
file's replication has gone down below the specified number ?
Thanks,
Tarandeep
On Wed, Aug 12, 2009 at 12:11 PM,
You can put compress data on HDFS and run Map Reduce job on it. But you
should use a codec that supports file splitting, otherwise whole file will
be read by one mapper. If you have read about Map reduce architecture, you
would understand that a map function processes chunk of data (called split).
