Could not obtain block blk ...

I am using hadoop 18.3, I have a single datanode and it appears to be up and running fine. I'm able to read/write data to it. However, when I try to spawn a map/reduce job it fails with "Could not obtain block: blk_3263745172951227264_1155 file =/opt/kindsight/hadoop/data/mapred/system/job_2

Reducing Hadoop Logs

I'm using hadoop 0.17.0. Unfortunately I cant upgrade to 0.19.0 just yet. I'm trying to control the amount of extraneous files. I noticed there are the following log files produced by hadoop; On Slave - userlogs (for each map/reduce job) - stderr

RE: Running mapred job from remote machine to a pseudo-distributed hadoop

I'll try again, can anyone tell me should it be possible to run hadoop in a pseudo-distributed mode (i.e. everything on one machine) and then submit a mapred job using the ToolRunner from another machine on that hadoop configuration? Cheers Arv -Original Message- From: Arv M

RE: File Descriptors not cleaned up

to the earlier questions regd your test program (slightly corrected) : "What do you see in your test program and how is it different from what you expect? In addition, why is that a problem?" Raghu. Arv Mistry wrote: > > Raghu, > > This is a real scenario for our appli

Running mapred job from remote machine to a pseudo-distributed hadoop

I have hadoop setup in a pseudo-distributed mode i.e. everything on one machine, And I'm trying to submit a hadoop mapred job from another machine to that hadoop setup. At the point that I run the mapred job I get the following error. Any ideas as to what I'm doing wrong? Is this possible in a p

RE: File Descriptors not cleaned up

dCommentId=12566 250#action_12566250 for more background on design choices made. Raghu. Arv Mistry wrote: > I guess the attachment got stripped, so here it is inline ... > > public class TestFsHadoop { > > public static Configuration conf = null; >

RE: File Descriptors not cleaned up

fis.close(); } catch (Exception e) { e.printStackTrace(); } } } } -Original Message- From: Arv Mistry [mailt

RE: File Descriptors not cleaned up

Raghu Angadi [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 30, 2008 7:36 PM To: core-user@hadoop.apache.org Subject: Re: File Descriptors not cleaned up Arv Mistry wrote: > > Thanks for responding Raghu, > > This code is run every hour, where I open a file ( a different file &

RE: File Descriptors not cleaned up

riginal Message- From: Raghu Angadi [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 30, 2008 4:33 PM To: core-user@hadoop.apache.org Subject: Re: File Descriptors not cleaned up Arv Mistry wrote: > > I've been trying to track down an issue where after some time I get > "Too

File Descriptors not cleaned up

I've been trying to track down an issue where after some time I get "Too many files open " i.e. we're not cleaning up somewhere ... I'm using "lsof -p " to track the open files and I find it's adding 3 file descriptors everytime I do a fs.open() where fs is FileSystem and is a Path object to a

RE: Unknown protocol to name node: JobSubmissionProtocol

supposed to talk a JobTracker. But the stack trace shows that it talked to a namenode. Could you check your configuration to see if the jobtracker port # was set to be the same as the namenode port #. Hairong On 7/30/08 6:56 AM, "Arv Mistry" <[EMAIL PROTECTED]> wrote: > >

Unknown protocol to name node: JobSubmissionProtocol

Can anyone provide any hints as to why this might be happening; I have hadoop running all process' on one machine (for trouble-shooting) and when I go to submit a job from another machine I get the following exception; INFO | jvm 2| 2008/07/30 06:05:05 | 2008-07-30 06:05:05,117 ERROR [Ha

RE: Hadoop warnings in pseudo-distributed mode

Sorry, found the errors of my ways I forgot to add 127.0.0.1 to the master/slave files Cheers Arv -Original Message- From: Arv Mistry Sent: Tuesday, July 29, 2008 8:53 AM To: 'core-user@hadoop.apache.org' Subject: Hadoop warnings in pseudo-distributed mode Could anyo

Hadoop warnings in pseudo-distributed mode

Could anyone tell me, is it normal to get warnings "could only be replicated to 0 nodes, instead of 1" when running in a psudo-distributed mode i.e. everything on one machine? It seems to be writing to the files that I expect, just I get this warning. If it isn't normal, just some background;

RE: Can reducer output multiple files?

Hi, I saw the note at the end of the message below: "Note that MultipleOutputFormat is available in Hadoop-0.17" Is 0.17 out yet? Can we output multiple files another way? Cheers Arv -Original Message- From: Amar Kamat [mailto:[EMAIL PROTECTED] Sent: Thursday, May 08, 2008 4:56 AM

Hadoop Resiliency

Hi folks, I'm new to hadoop and just had a few questions regards resiliency i) Does hadoop support redundant NameNodes? I didn't see any mention of it. ii) In a distributed setup, when you kill a DataNode, should the NameNode restart it automatically? I see the NameNode detects (eventually) th