Doubt: Regarding running Hadoop on a cluster with shared disk.

2010-05-05 Thread Udaya Lakshmi
Hi,
   I have an account on a cluster which is having a file system similar to
NFS. If I create a file on one machine it is being shown on all the machines
in the cluster. But hadoop will work on a cluster of machines, where in ,
each machine has a disk of its own. Can someone please help me use hadoop on
my cluster.
Thanks,
Udaya.


RE: Doubt: Regarding running Hadoop on a cluster with shared disk.

2010-05-05 Thread Michael Segel

Udaya,

You can use non-local disk on your hadoop cloud, however it will have 
sub-optimal performance, and you will have to tune accordingly.

If its a shared drive on all of your nodes, you need to create different 
directories for each machine.

Suppose your shared drive is /foo  then you would need to set up a /foo/name 
of node/data for each machine in your cluster.

The drawback is not only I/O traffic and constraints but you'll have to tune ZK 
and watch out for timing issues as your disk i/o is your constraint.

Definitely not recommended.


 Date: Wed, 5 May 2010 15:52:11 +0530
 Subject: Doubt: Regarding running Hadoop on a cluster with shared disk.
 From: udaya...@gmail.com
 To: common-user@hadoop.apache.org
 
 Hi,
I have an account on a cluster which is having a file system similar to
 NFS. If I create a file on one machine it is being shown on all the machines
 in the cluster. But hadoop will work on a cluster of machines, where in ,
 each machine has a disk of its own. Can someone please help me use hadoop on
 my cluster.
 Thanks,
 Udaya.
  
_
Hotmail is redefining busy with tools for the New Busy. Get more from your 
inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2

The value of Map output records

2010-05-05 Thread Dan Fundatureanu
Is there a way to get the value of Map output records from within the
Reducer ?
I want to know the total number of the Map output records while the
Reducer is running and I've noticed this value in the web interface shown
for each Map.
Or is there a way to read in Reducer the value of Reduce input groups ?

Thanks,
Dan