Relationship between mapred.child.java.opts, mapred.reduce.child.java.opts, and mapred.map.child.java.opts in Hadoop 0.20.2
Hi everyone, I had lots error in Hadoop 0.20.2 like: Error: java.lang.OutOfMemoryError: Java heap space at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:336) at sun.net.www.http.ChunkedInputStream.readAheadNonBlocking(ChunkedInputStream.java:493) at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:584) at sun.net.www.http.ChunkedInputStream.hurry(ChunkedInputStream.java:734) at sun.net.www.http.ChunkedInputStream.closeUnderlying(ChunkedInputStream.java:198) at sun.net.www.http.ChunkedInputStream.close(ChunkedInputStream.java:715) at java.io.FilterInputStream.close(FilterInputStream.java:155) at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.close(HttpURLConnection.java:2401) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.reserve(ReduceTask.java:1007) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1475) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195) Error: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195) I would like to hear suggestions to overcome that. I tried to increase the "mapred.child.java.opts" settings to 2000M but still get this error. Does that error occur because of the memory shortage of reducers? Should I increase the mapred.reduce.child.java.opts at the same time. I try to figure out the relationship among these three settings: mapred.child.java.opts mapred.reduce.child.java.opts mapred.map.child.java.opts Is the following equations correct? mapred.child.java.opts = mapred.reduce.child.java.opts + mapred.map.child.java.opts So if the mapred.child.java.opts value is fixed, to allow more memory for reducer I should decrease "mapred.map.child.java.opts" and increase "mapred.reduce.child.java.opts"? Or these three are different concepts in different JVMs ? Thanks. Shi
Starting an HDFS node (standalone) programmatically by API
Hi all, hdfs nodes can be started using the sh scripts provided with hadoop. I read that it's all based on script files is it possible to start an HDFS (standalone) from a java application by API? Thanks -- View this message in context: http://hadoop-common.472056.n3.nabble.com/Starting-an-HDFS-node-standalone-programmatically-by-API-tp3075693p3075693.html Sent from the Users mailing list archive at Nabble.com.
Re: HDFS File Appending URGENT
Thanks a lot guys.
Another query for production.
Do we have any way by which we can purge the hdfs job and history logs on time
basis.
For example we want to keep only last 30 days log and its size is increasing a
lot in production.
Thanks again
Regards,
Jagaran
From: "Tsz Wo (Nicholas), Sze"
To: common-user@hadoop.apache.org
Sent: Fri, 17 June, 2011 11:45:22 AM
Subject: Re: HDFS File Appending URGENT
Hi Jagaran,
Short answer: the append feature is not in any release. In this sense, it is
not stable. Below are more details on the Append feature status.
- 0.20.x (includes release 0.20.2)
There are known bugs in append. The bugs may cause data loss.
- 0.20-append
There were effort on fixing the known append bugs but there are no releases. I
heard Facebook was using it (with additional patches?) in production but I did
not have the details.
- 0.21
It has a new append design (HDFS-265). However, the 0.21.0 release is only a
minor release. It has not undergone testing at scale and should not be
considered stable or suitable for production. Also, 0.21 development has been
discontinued. Newly discovered bugs may not be fixed.
- 0.22, 0.23
Not yet released.
Regards,
Tsz-Wo
From: jagaran das
To: common-user@hadoop.apache.org
Sent: Fri, June 17, 2011 11:15:04 AM
Subject: Fw: HDFS File Appending URGENT
Please help me on this.
I need it very urgently
Regards,
Jagaran
- Forwarded Message
From: jagaran das
To: common-user@hadoop.apache.org
Sent: Thu, 16 June, 2011 9:51:51 PM
Subject: Re: HDFS File Appending URGENT
Thanks a lot Xiabo.
I have tried with the below code in HDFS version 0.20.20 and it worked.
Is it not stable yet?
public class HadoopFileWriter {
public static void main (String [] args) throws Exception{
try{
URI uri = new
URI("hdfs://localhost:9000/Users/jagarandas/Work-Assignment/Analytics/analytics-poc/hadoop-0.20.203.0/data/test.dat");
Path pt=new Path(uri);
FileSystem fs = FileSystem.get(new Configuration());
BufferedWriter br;
if(fs.isFile(pt)){
br=new BufferedWriter(new OutputStreamWriter(fs.append(pt)));
br.newLine();
}else{
br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true)));
}
String line = args[0];
System.out.println(line);
br.write(line);
br.close();
}catch(Exception e){
e.printStackTrace();
System.out.println("File not found");
}
}
}
Thanks a lot for your help.
Regards,
Jagaran
From: Xiaobo Gu
To: common-user@hadoop.apache.org
Sent: Thu, 16 June, 2011 8:01:14 PM
Subject: Re: HDFS File Appending URGENT
You can merge multiple files into a new one, there is no means to
append to a existing file.
On Fri, Jun 17, 2011 at 10:29 AM, jagaran das wrote:
> Is the hadoop version Hadoop 0.20.203.0 API
>
> That means still the hadoop files in HDFS version 0.20.20 are immutable?
> And there is no means we can append to an existing file in HDFS?
>
> We need to do this urgently as we have do set up the pipeline accordingly in
> production?
>
> Regards,
> Jagaran
>
>
>
>
> From: Xiaobo Gu
> To: common-user@hadoop.apache.org
> Sent: Thu, 16 June, 2011 6:26:45 PM
> Subject: Re: HDFS File Appending
>
> please refer to FileUtil.CopyMerge
>
> On Fri, Jun 17, 2011 at 8:33 AM, jagaran das wrote:
>> Hi,
>>
>> We have a requirement where
>>
>> There would be huge number of small files to be pushed to hdfs and then use
>>pig
>> to do analysis.
>> To get around the classic "Small File Issue" we merge the files and push a
>> bigger file in to HDFS.
>> But we are loosing time in this merging process of our pipeline.
>>
>> But If we can directly append to an existing file in HDFS we can save this
>> "Merging Files" time.
>>
>> Can you please suggest if there a newer stable version of Hadoop where can go
>> for appending ?
>>
>> Thanks and Regards,
>> Jagaran
>
Re: HDFS File Appending URGENT
Hi Jagaran,
Short answer: the append feature is not in any release. In this sense, it is
not stable. Below are more details on the Append feature status.
- 0.20.x (includes release 0.20.2)
There are known bugs in append. The bugs may cause data loss.
- 0.20-append
There were effort on fixing the known append bugs but there are no releases. I
heard Facebook was using it (with additional patches?) in production but I did
not have the details.
- 0.21
It has a new append design (HDFS-265). However, the 0.21.0 release is only a
minor release. It has not undergone testing at scale and should not be
considered stable or suitable for production. Also, 0.21 development has been
discontinued. Newly discovered bugs may not be fixed.
- 0.22, 0.23
Not yet released.
Regards,
Tsz-Wo
From: jagaran das
To: common-user@hadoop.apache.org
Sent: Fri, June 17, 2011 11:15:04 AM
Subject: Fw: HDFS File Appending URGENT
Please help me on this.
I need it very urgently
Regards,
Jagaran
- Forwarded Message
From: jagaran das
To: common-user@hadoop.apache.org
Sent: Thu, 16 June, 2011 9:51:51 PM
Subject: Re: HDFS File Appending URGENT
Thanks a lot Xiabo.
I have tried with the below code in HDFS version 0.20.20 and it worked.
Is it not stable yet?
public class HadoopFileWriter {
public static void main (String [] args) throws Exception{
try{
URI uri = new
URI("hdfs://localhost:9000/Users/jagarandas/Work-Assignment/Analytics/analytics-poc/hadoop-0.20.203.0/data/test.dat");
Path pt=new Path(uri);
FileSystem fs = FileSystem.get(new Configuration());
BufferedWriter br;
if(fs.isFile(pt)){
br=new BufferedWriter(new OutputStreamWriter(fs.append(pt)));
br.newLine();
}else{
br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true)));
}
String line = args[0];
System.out.println(line);
br.write(line);
br.close();
}catch(Exception e){
e.printStackTrace();
System.out.println("File not found");
}
}
}
Thanks a lot for your help.
Regards,
Jagaran
From: Xiaobo Gu
To: common-user@hadoop.apache.org
Sent: Thu, 16 June, 2011 8:01:14 PM
Subject: Re: HDFS File Appending URGENT
You can merge multiple files into a new one, there is no means to
append to a existing file.
On Fri, Jun 17, 2011 at 10:29 AM, jagaran das wrote:
> Is the hadoop version Hadoop 0.20.203.0 API
>
> That means still the hadoop files in HDFS version 0.20.20 are immutable?
> And there is no means we can append to an existing file in HDFS?
>
> We need to do this urgently as we have do set up the pipeline accordingly in
> production?
>
> Regards,
> Jagaran
>
>
>
>
> From: Xiaobo Gu
> To: common-user@hadoop.apache.org
> Sent: Thu, 16 June, 2011 6:26:45 PM
> Subject: Re: HDFS File Appending
>
> please refer to FileUtil.CopyMerge
>
> On Fri, Jun 17, 2011 at 8:33 AM, jagaran das wrote:
>> Hi,
>>
>> We have a requirement where
>>
>> There would be huge number of small files to be pushed to hdfs and then use
>>pig
>> to do analysis.
>> To get around the classic "Small File Issue" we merge the files and push a
>> bigger file in to HDFS.
>> But we are loosing time in this merging process of our pipeline.
>>
>> But If we can directly append to an existing file in HDFS we can save this
>> "Merging Files" time.
>>
>> Can you please suggest if there a newer stable version of Hadoop where can go
>> for appending ?
>>
>> Thanks and Regards,
>> Jagaran
>
Fw: HDFS File Appending URGENT
Please help me on this.
I need it very urgently
Regards,
Jagaran
- Forwarded Message
From: jagaran das
To: common-user@hadoop.apache.org
Sent: Thu, 16 June, 2011 9:51:51 PM
Subject: Re: HDFS File Appending URGENT
Thanks a lot Xiabo.
I have tried with the below code in HDFS version 0.20.20 and it worked.
Is it not stable yet?
public class HadoopFileWriter {
public static void main (String [] args) throws Exception{
try{
URI uri = new
URI("hdfs://localhost:9000/Users/jagarandas/Work-Assignment/Analytics/analytics-poc/hadoop-0.20.203.0/data/test.dat");
Path pt=new Path(uri);
FileSystem fs = FileSystem.get(new Configuration());
BufferedWriter br;
if(fs.isFile(pt)){
br=new BufferedWriter(new OutputStreamWriter(fs.append(pt)));
br.newLine();
}else{
br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true)));
}
String line = args[0];
System.out.println(line);
br.write(line);
br.close();
}catch(Exception e){
e.printStackTrace();
System.out.println("File not found");
}
}
}
Thanks a lot for your help.
Regards,
Jagaran
From: Xiaobo Gu
To: common-user@hadoop.apache.org
Sent: Thu, 16 June, 2011 8:01:14 PM
Subject: Re: HDFS File Appending URGENT
You can merge multiple files into a new one, there is no means to
append to a existing file.
On Fri, Jun 17, 2011 at 10:29 AM, jagaran das wrote:
> Is the hadoop version Hadoop 0.20.203.0 API
>
> That means still the hadoop files in HDFS version 0.20.20 are immutable?
> And there is no means we can append to an existing file in HDFS?
>
> We need to do this urgently as we have do set up the pipeline accordingly in
> production?
>
> Regards,
> Jagaran
>
>
>
>
> From: Xiaobo Gu
> To: common-user@hadoop.apache.org
> Sent: Thu, 16 June, 2011 6:26:45 PM
> Subject: Re: HDFS File Appending
>
> please refer to FileUtil.CopyMerge
>
> On Fri, Jun 17, 2011 at 8:33 AM, jagaran das wrote:
>> Hi,
>>
>> We have a requirement where
>>
>> There would be huge number of small files to be pushed to hdfs and then use
>>pig
>> to do analysis.
>> To get around the classic "Small File Issue" we merge the files and push a
>> bigger file in to HDFS.
>> But we are loosing time in this merging process of our pipeline.
>>
>> But If we can directly append to an existing file in HDFS we can save this
>> "Merging Files" time.
>>
>> Can you please suggest if there a newer stable version of Hadoop where can go
>> for appending ?
>>
>> Thanks and Regards,
>> Jagaran
>
Data node check dir storm
Hello. I can see that if data node receives some IO error, this can cause checkDir storm. What I mean: 1) any error produces DataNode.checkDiskError call 2) this call locks volume: java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228) at java.io.File.exists(File.java:733) at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsCheck(DiskChecker.java:65) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:86) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.checkDirTree(FSDataset.java:228) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.checkDirTree(FSDataset.java:232) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.checkDirTree(FSDataset.java:232) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSDir.checkDirTree(FSDataset.java:232) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.checkDirs(FSDataset.java:414) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.checkDirs(FSDataset.java:617) - locked <0x00080a8faec0> (a org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet) at org.apache.hadoop.hdfs.server.datanode.FSDataset.checkDataDir(FSDataset.java:1681) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkDiskError(DataNode.java:745) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkDiskError(DataNode.java:735) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(BlockReceiver.java:202) at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:151) at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:167) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:646) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:352) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:390) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:331) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:111) at java.lang.Thread.run(Thread.java:619) 3) This produces timeouts on other calls, e.g. 2011-06-17 17:35:03,922 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception: java.io.InterruptedIOException at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:260) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at java.io.DataOutputStream.flush(DataOutputStream.java:106) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(BlockReceiver.java:183) at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:151) at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:167) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:646) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:352) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:390) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:331) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:111) at java.lang.Thread.run(Thread.java:619) 4) This, in turn, produces more "dir check calls". 5) All the cluster works very slow because of half-working node. Best regards, Vitalii Tymchyshyn
Re: Upgrading namenode/secondary node hardware
I see it is not so obvious and potentially dangerous so I will be learning & experimenting first. Thx for the tip. 2011/6/17 Steve Loughran > On 16/06/11 14:19, MilleBii wrote: > >> But if my Filesystem is up& running fine... do I have to worry at all or >> will the copy (ftp transfer) of hdfs will be enough. >> >> > I'm not going to make any predictions there as if/when things go wrong > > -you do need to shut down the FS before the move > -you ought to get the edit logs replayed before the move > -you may want to try experimenting with copying the namenode data and > bringing up the namenode (without any datanodes connected to, so it comes up > in safe mode), to make sure everything works. > > I'd also worry that if you aren't familiar with the edit log, you may need > to spend some time learning the subtle details of namenode journalling, > replaying, backup and restoration, and what the secondary namenode does. > It's easy to bring up a cluster and get overconfident that it works, right > up to the moment it stops working. Experiment with your cluster's and teams' > failure handling before you really need it > > > >> 2011/6/16 Steve Loughran >> >> On 15/06/11 15:54, MilleBii wrote: >>> >>> Thx. #1 don't understand the "edit logs" remark. >>> well, that's something you need to work on as its the key to keeping your >>> cluster working. The edit log is the journal of changes made to a >>> namenode, >>> which gets streamed to HDD and your secondary Namenode. After a NN >>> restart, >>> it has to replay all changes since the last checkpoint to get its >>> directory >>> structure up to date. Lose the edit log and you may as well reformat the >>> disks. >>> >>> >> >> >> > -- -MilleBii-
Name node problems
Hello. My environment is: HDFS 0.21, NameNode + BackupNode. After some time Backup node crashed with an exception (stack trace below). Problem #1 - the process did not exit. I've tried to run Secondary node to perform checkout. Got similar crash, but it did exit. Backed up my data and restarted name node. Same crash: 2011-06-17 10:12:39,985 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1765) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1753) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadEditRecords(FSEditLog.java:708) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:411) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:378) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1209) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1019) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:483) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:270) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:433) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) The thing that helped was cleaning edits file with shell printf someone've sent to the list few days ago. But it would be great to have an option either to stop on first replay error or to skip such an errors (by looking into code it tried to apply date to unexisting file). In any case this would give later state then simply cleaning edits file. I have backup of the name node data that crashes, if someone is interested. As for me, this is a bug. I even though to put it to Jira, but found out https://issues.apache.org/jira/browse/HDFS-1864 (see last comment). Is it true that bug reports should not go to Jira? -- Best regards, Vitalii Tymchyshyn
Re: Upgrading namenode/secondary node hardware
On 16/06/11 14:19, MilleBii wrote: But if my Filesystem is up& running fine... do I have to worry at all or will the copy (ftp transfer) of hdfs will be enough. I'm not going to make any predictions there as if/when things go wrong -you do need to shut down the FS before the move -you ought to get the edit logs replayed before the move -you may want to try experimenting with copying the namenode data and bringing up the namenode (without any datanodes connected to, so it comes up in safe mode), to make sure everything works. I'd also worry that if you aren't familiar with the edit log, you may need to spend some time learning the subtle details of namenode journalling, replaying, backup and restoration, and what the secondary namenode does. It's easy to bring up a cluster and get overconfident that it works, right up to the moment it stops working. Experiment with your cluster's and teams' failure handling before you really need it 2011/6/16 Steve Loughran On 15/06/11 15:54, MilleBii wrote: Thx. #1 don't understand the "edit logs" remark. well, that's something you need to work on as its the key to keeping your cluster working. The edit log is the journal of changes made to a namenode, which gets streamed to HDD and your secondary Namenode. After a NN restart, it has to replay all changes since the last checkpoint to get its directory structure up to date. Lose the edit log and you may as well reformat the disks.
