Doubt: Regarding running Hadoop on a cluster with shared disk.

2010-05-05 Thread Udaya Lakshmi
Hi, I have an account on a cluster which is having a file system similar to NFS. If I create a file on one machine it is being shown on all the machines in the cluster. But hadoop will work on a cluster of machines, where in , each machine has a disk of its own. Can someone please help me use ha

RE: Doubt: Regarding running Hadoop on a cluster with shared disk.

2010-05-05 Thread Michael Segel
Udaya, You can use non-local disk on your hadoop cloud, however it will have sub-optimal performance, and you will have to tune accordingly. If its a shared drive on all of your nodes, you need to create different directories for each machine. Suppose your shared drive is /foo then you would

The value of "Map output records"

2010-05-05 Thread Dan Fundatureanu
Is there a way to get the value of "Map output records" from within the Reducer ? I want to know the total number of the "Map output records" while the Reducer is running and I've noticed this value in the web interface shown for each Map. Or is there a way to read in Reducer the value of "Reduce i