Hadoop streaming using JAVA

Hello, I am trying to write code for streaming using standard java with out map reduce. I got task to do it that way, so what I did I build executable jar file which does following in main method public static void main(String[] args) throws IOException { BufferedReader in = new Bu

DN utilizing disks of varying sizes

Hi, If I have multiple disks of varying sizes on a given data node, is there to tell HDFS that one disk has more space than others? Last I checked the answer (AFAIR from an email thread I can't find) was 'no', since HDFS uses round robin to write to each disk. Just wondering if that's changed or i

Re: backup namenode setting issue: namenode failed to start

Thanks, Konstantin. I will look at options for mounting the folder. Is there any guide to a successful deployment of this method? Still have this question, does this backup method only work for a fresh cluster (It's my guess the namenode only stores a copy of new data information into folders

Re: Hadoop JobTracker Hanging

There are two issues which were fixed in 0.21.0 and can cause job tracker to run out of memory: https://issues.apache.org/jira/browse/MAPREDUCE-1316 and https://issues.apache.org/jira/browse/MAPREDUCE-841 We've been hit by MAPREDUCE-841 (large jobConf objects with large number of tasks, espec

Re: Hadoop JobTracker Hanging

There was also https://issues.apache.org/jira/browse/MAPREDUCE-1316 whose cause hit clusters at Yahoo! very badly last year. The situation was particularly noticeable in the face of lots of jobs with failed tasks and a specific fix that enabled OutOfBand heartbeats. The latter (i.e. the OOB heartbe

Re: Hadoop JobTracker Hanging

On Jun 22, 2010, at 3:17 AM, Steve Loughran wrote: > > I'm surprised its the JT that is OOM-ing, anecdotally its the NN and 2ary NN > that use more, especially if the files are many and the blocksize small. the > JT should not be tracking that much data over time Pre-0.20.2, there are definite

Re: Namenode issue

On Jun 22, 2010, at 2:29 AM, Pierre ANCELOT wrote: > Fixed! > After tracking down in the source code, I found it tried to read data from > name/current/fstime, which was empty... > I deleted the file and the namenode restarted without an issue. > If this may be of help to anyone... It is conside

Re: Hadoop JobTracker Hanging

+1 for compressed pointers. Sent from my mobile. Please excuse the typos. On 2010-06-22, at 4:18 AM, Steve Loughran wrote: > Bobby Dennett wrote: >> Thanks all for your suggestions (please note that Tan is my co-worker; >> we are both working to try and resolve this issue)... we experienced >

Re: Hadoop JobTracker Hanging

Bobby Dennett wrote: Thanks all for your suggestions (please note that Tan is my co-worker; we are both working to try and resolve this issue)... we experienced another hang this weekend and increased the HADOOP_HEAPSIZE setting to 6000 (MB) as we do periodically see "java.lang.OutOfMemoryError:

Re: backup namenode setting issue: namenode failed to start

Konstantin Shvachko wrote: Looks like the mounted file system /mnt/namenode-backup does not support locking. It should, otherwise hdfs cannot guarantee that only one name-node updates the directory. You might want to check with your sysadmins, may be the mount point is misconfigured. Thanks,

Re: Namenode issue

Fixed! After tracking down in the source code, I found it tried to read data from name/current/fstime, which was empty... I deleted the file and the namenode restarted without an issue. If this may be of help to anyone... -- http://www.neko-consulting.com Ego sum quis ego servo "Je suis ce que je

Re: Can we modify existing file in HDFS?

>>Do I need to remove and re-create the whole file? Simply put, as of now, yes. Append functionality is being made available to users to add to end of file though :) Amogh On 6/22/10 1:56 PM, "elton sky" wrote: hello everyone, I noticed there are 6 operations in HDFS: OP_WRITE_BLOCK OP_READ_

Re: new/old APIs

Thanks Ted, not intuitive, but that'll work for now. job.submit(); String[] trackingUrl = job.getTrackingURL().split("="); BufferedWriter bwriter = new BufferedWriter(new FileWriter(lockFile)); bwriter.write(trackingUrl[1]); bwriter.close(); Alan >On Mon, Jun 21, 2010 at 9:4

Can we modify existing file in HDFS?

hello everyone, I noticed there are 6 operations in HDFS: OP_WRITE_BLOCK OP_READ_BLOCK OP_READ_METADATA OP_REPLACE_BLOCK OP_COPY_BLOCK OP_BLOCK_CHECKSUM and As I know there's no way to modify some arbitrary part in a existing file in HDFS. So what if I create a say, 2 Petabytes, file and like to

Namenode issue

Hi, We had a power failure... after restarting the nodes, I try to start the namenode... Any idea please? Thanks :) $ bin/hadoop namenode 10/06/22 09:24:30 INFO namenode.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MS