finding the input file of a failed map task

In the JobTracker website, when I click on a JobId, there is a listing of completed maps and killed maps. When I click on the number under the column completed or killed, there is a table with columns as mentioned below. Task, Complete, Status, Start Time, Finish Time, Errors Status column is

IO Exception in Map Tasks

Hi, In one of the map tasks, i get the following exception: java.io.IOException: Task process exit with nonzero status of 255. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:424) java.io.IOException: Task process exit with nonzero status of 255. at

Re: Storing data-node content to other machine

There is no requirement that your hdfs and mapred clusters share an installation directory, it is just done that way because it is simple and most people have a datanode and tasktracker on each slave node. Simply have 2 configuration directories on your cluster machines, and us the

Re: IO Exception in Map Tasks

The jvm had a hard failure and crashed On Sun, Apr 26, 2009 at 11:34 PM, Rakhi Khatwani rakhi.khatw...@gmail.comwrote: Hi, In one of the map tasks, i get the following exception: java.io.IOException: Task process exit with nonzero status of 255. at

Re: IO Exception in Map Tasks

Thanks Jason, is there any way we can avoid this exception?? Thanks, Raakhi On Mon, Apr 27, 2009 at 1:20 PM, jason hadoop jason.had...@gmail.comwrote: The jvm had a hard failure and crashed On Sun, Apr 26, 2009 at 11:34 PM, Rakhi Khatwani rakhi.khatw...@gmail.comwrote: Hi,

Balancing datanodes - Running hadoop 0.18.3

Hi, I had sent out an email yesterday asking about how to balance the cluster after setting the replication level to 2. I have 4 datanodes and one namenode in my setup. Using the -R switch with -setrep did the trick but one of my nodes became under utilized. I then ran hadoop balancer and it

write a large file to HDFS?

hi, If I write a large file to HDFS, will it be split into blocks and multi-blocks are written to HDFS at the same time? Or HDFS can only write block by block? Thanks. -- View this message in context: http://www.nabble.com/write-a-large-file-to-HDFS--tp23252754p23252754.html Sent from the

Blocks replication in downtime even

Hi. I have a question: If I have N of DataNodes, and one or several of the nodes have become unavailable, would HDFS re-synchronize the blocks automatically, according to replication level set? And if yes, when? As soon as the offline node was detected, or only on file access? Regards.

Re: Balancing datanodes - Running hadoop 0.18.3

Hi, The balancer works with the average utilization of all the nodes in the cluster - in your case it's about 13%. Only nodes that are +/- 10% off the average will be rebalanced. Node 4 isn't under-utilized because 13-10=3 which is less than 4%. You can use a different threshold than the default

Re: Balancing datanodes - Running hadoop 0.18.3

Hi Tamir, Thanks for the info, makes sense now :). Cheers, Usman Hi, The balancer works with the average utilization of all the nodes in the cluster - in your case it's about 13%. Only nodes that are +/- 10% off the average will be rebalanced. Node 4 isn't under-utilized because 13-10=3 which

.20.0, Partitioners?

Is there some magic to get a Partitioner working on .20.0? Setting the partitioner class on the Job object doesn't take, hadoop always uses the HashPartitioner. Looking through the source code, it looks like the MapOutputBuffer in MapTask only ever fetches the mapred.partitioner.class, and

ANN: R and Hadoop = RHIPE 0.1

Hello, I'd like to announce the release of the 0.1 version of RHIPE -R and Hadoop Integrated Processing Environment. Using RHIPE, it is possible to write map-reduce algorithms using the R language and start them from within R. RHIPE is built on Hadoop and so benefits from Hadoop's fault tolerance,

Re: Can't start fully-distributed operation of Hadoop in Sun Grid Engine

I have contacted with the administor of our cluster and he gave me the access. Now my program can work under full distributed mode. Thanks a lot. Jasmine - Original Message - From: jason hadoop jason.had...@gmail.com To: core-user@hadoop.apache.org Sent: Sunday, April 26, 2009 12:13

RE: Blocks replication in downtime even

http://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa ilure%2C+Heartbeats+and+Re-Replication hope this helps. Koji -Original Message- From: Stas Oskin [mailto:stas.os...@gmail.com] Sent: Monday, April 27, 2009 4:11 AM To: core-user@hadoop.apache.org Subject: Blocks

Re: Datanode Setup

bump* Any suggestions? -- View this message in context: http://www.nabble.com/Datanode-Setup-tp23064660p23259364.html Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: IO Exception in Map Tasks

You will need to figure out why your task crashed, Check the task logs, there may be some messages there, that give you a hint as to what is going on. you can enable saving failed task logs and then run the task standalone in the isolation runner. chapter 7 of my book (alpha available) provides

Re: write a large file to HDFS?

block by block. open multiple connections and write multiple files if you are not saturating your network connection. Generally a single file writer writing large blocks rapidly will do a decent job of saturating things. On Mon, Apr 27, 2009 at 2:22 AM, Xie, Tao xietao1...@gmail.com wrote: hi,

Re: .20.0, Partitioners?

Ryan, I observed this behavior too -- Partitioner does not seems to work with the new API exactly for the reason you have mentioned. Till this gets fixed, you probably need to use the old API. Jothi On 4/27/09 7:14 PM, Ryan Farris farri...@gmail.com wrote: Is there some magic to get a

Re: .20.0, Partitioners?

I created https://issues.apache.org/jira/browse/HADOOP-5750 to follow this up. Thanks Jothi On 4/27/09 10:10 PM, Jothi Padmanabhan joth...@yahoo-inc.com wrote: Ryan, I observed this behavior too -- Partitioner does not seems to work with the new API exactly for the reason you have

Rescheduling of already completed map/reduce task

Hi, The job froze after the filesystem hung on a machine which had successfully completed a map task. Is there a flag to enable the re scheduling of such a task ? Jstack of job tracker SocketListener0-2 prio=10 tid=0x08916000 nid=0x4a4f runnable [0x4d05c000..0x4d05ce30]

Re: Blocks replication in downtime even

Thanks. 2009/4/27 Koji Noguchi knogu...@yahoo-inc.com http://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa ilure%2C+Heartbeats+and+Re-Replicationhttp://hadoop.apache.org/core/docs/current/hdfs_design.html#Data+Disk+Fa%0Ailure%2C+Heartbeats+and+Re-Replication hope this

Re: How to set System property for my job

I think what you want is the section Task Execution Environment in http://hadoop.apache.org/core/docs/current/mapred_tutorial.html http://hadoop.apache.org/core/docs/current/mapred_tutorial.html . Here is a sample from that document: property namemapred.child.java.opts/name value

Debian support for Cloudera's Distribution

Hey Hadoop fans, just wanted to drop a quick note to let you know that we now have debian packages for our distribution in addition to RPMs. We will continue to support both platforms going forward. Todd Lipcon put in many late nights for this, so next time you see him, but him a beer :-)

Hadoop Training, May 15th: SF Bay Area with Online Participation Available

OK, last announcement from me today :-) We're hosting a training session in the SF bay area (at the Cloudera office) on Friday, May 15th. We're doing two things differently: 1) We've allocated a chunk of discounted early bird registrations - first come first serve until May 1st, at which point,