Hadoop Name node startup issue
Hello All, We are facing a problem in starting the hadoop name node instance. We are getting the following error : 2010-10-21 12:09:05,379 ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization failed. java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.Text.readString(Text.java:412) at org.apache.hadoop.fs.permission.PermissionStatus.readFields(PermissionStatus.java:84) at org.apache.hadoop.fs.permission.PermissionStatus.read(PermissionStatus.java:98) at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:483) at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:849) at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675) at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) Any help on this issue is much appreciated. Please refer the attachment for the name node logs. Thanks and Regards, Bala 2010-10-21 12:07:57,515 INFO org.apache.hadoop.dfs.NameNode: STARTUP_MSG: / STARTUP_MSG: Starting NameNode STARTUP_MSG: host = red/192.168.100.2 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.18.3 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250; compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 / 2010-10-21 12:07:57,626 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310 2010-10-21 12:07:57,632 INFO org.apache.hadoop.dfs.NameNode: Namenode up at: red.hoonur.com/192.168.100.2:54310 2010-10-21 12:07:57,635 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2010-10-21 12:07:57,651 INFO org.apache.hadoop.dfs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-10-21 12:07:57,739 INFO org.apache.hadoop.fs.FSNamesystem: fsOwner=hadoop,hadoop 2010-10-21 12:07:57,740 INFO org.apache.hadoop.fs.FSNamesystem: supergroup=supergroup 2010-10-21 12:07:57,740 INFO org.apache.hadoop.fs.FSNamesystem: isPermissionEnabled=false 2010-10-21 12:07:57,755 INFO org.apache.hadoop.dfs.FSNamesystemMetrics: Initializing FSNamesystemMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-10-21 12:07:57,756 INFO org.apache.hadoop.fs.FSNamesystem: Registered FSNamesystemStatusMBean 2010-10-21 12:07:57,900 INFO org.apache.hadoop.dfs.Storage: Number of files = 2988433 2010-10-21 12:09:05,014 INFO org.apache.hadoop.dfs.Storage: Number of files under construction = 49 2010-10-21 12:09:05,315 INFO org.apache.hadoop.dfs.Storage: Image file of size 395864924 loaded in 67 seconds. 2010-10-21 12:09:05,351 INFO org.apache.hadoop.dfs.Storage: Edits file edits of size 22024 edits # 215 loaded in 0 seconds. 2010-10-21 12:09:05,379 ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization failed. java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.Text.readString(Text.java:412) at org.apache.hadoop.fs.permission.PermissionStatus.readFields(PermissionStatus.java:84) at org.apache.hadoop.fs.permission.PermissionStatus.read(PermissionStatus.java:98) at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:483) at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:849) at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675) at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) 2010-10-21 12:09:05,380 INFO org.apache.hadoop.ipc.Server: Stopping server on 54310 2010-10-21 12:09:05,384 ERROR org.apache.hadoop.dfs.NameNode: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at
Re: Hadoop Name node startup issue
On 21/10/10 09:12, Bala.Linux wrote: Hello All, We are facing a problem in starting the hadoop name node instance. We are getting the following error : 2010-10-21 12:09:05,379 ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization failed. java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.Text.readString(Text.java:412) at org.apache.hadoop.fs.permission.PermissionStatus.readFields(PermissionStatus.java:84) at org.apache.hadoop.fs.permission.PermissionStatus.read(PermissionStatus.java:98) at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:483) at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:849) at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675) at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289) at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80) at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294) at org.apache.hadoop.dfs.FSNamesystem.init(FSNamesystem.java:273) at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:193) at org.apache.hadoop.dfs.NameNode.init(NameNode.java:179) at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830) at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839) Any help on this issue is much appreciated. Please refer the attachment for the name node logs. Looks like some problem trying to replay the namenode logs on startup, the last record of the log is missing. You'll probably need to delete that last log entry in the fs edit log (I've never done it, so can't describe how) -steve
Error when try to build mumak
Hello, I am trying to build mumak and have a bite, with ant package in mapred/ dir. I got error: [ivy:resolve] :: problems summary :: [ivy:resolve] WARNINGS [ivy:resolve] module not found: org.apache.hadoop#hadoop-common;0.21.0 [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar: [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar [ivy:resolve] module not found: org.apache.hadoop#hadoop-common-test;0.21.0 [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-common-test;0.21.0!hadoop-common-test.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-common-test;0.21.0!hadoop-common-test.jar: [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common-test/0.21.0/hadoop-common-test-0.21.0.jar [ivy:resolve] module not found: org.apache.hadoop#hadoop-hdfs;0.21.0 [ivy:resolve] apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-hdfs;0.21.0!hadoop-hdfs.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.jar [ivy:resolve] maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.pom [ivy:resolve] -- artifact org.apache.hadoop#hadoop-hdfs;0.21.0!hadoop-hdfs.jar: [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/0.21.0/hadoop-hdfs-0.21.0.jar [ivy:resolve] :: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] :: [ivy:resolve] :: org.apache.hadoop#hadoop-common;0.21.0: not found [ivy:resolve] :: org.apache.hadoop#hadoop-common-test;0.21.0: not found [ivy:resolve] :: org.apache.hadoop#hadoop-hdfs;0.21.0: not found [ivy:resolve] :: [ivy:resolve] [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS BUILD FAILED /data0/hadoop-0.21.0/mapred/build.xml:1861: impossible to resolve dependencies: resolve failed - see output for details any ideas?
Re: Size of Directory in HDFS
It should be doable like this: FileSystem.getContentSummary(FileStatus.getPath()).getLength(); On Thu, Oct 21, 2010 at 1:47 AM, Ananth Sarathy ananth.t.sara...@gmail.com wrote: I am trying to see how what the total size of all the files within a directory in HDFS is. I try Configuration conf = new Configuration(); Path inFile = new Path( /MyDir); FileSystem fs = inFile.getFileSystem(conf); FileStatus status = fs.getFileStatus(inFile); System.out.println(status.isDir()); System.out.println(status.getLen()); But since it's a directory, i get a 0. Anyone have any idea on how I can get this info? I am using apache hadoop 20.2. Secondly, I assume FileSystem.getUsed and FileStatus.getLen is in bytes, correct? Ananth T Sarathy -- Harsh J www.harshj.com
Re: Size of Directory in HDFS
Secondly, I assume FileSystem.getUsed and FileStatus.getLen is in bytes, correct? Yes, this is correct. Lengths of files are returned in bytes. -- Harsh J www.harshj.com
Re: How to stop a mapper within a map-reduce job when you detect bad input
If it occurs eventually as your record reader reads it, then you may use a MapRunner class instead of a Mapper IFace/Subclass. This way, you may try/catch over the record reader itself, and call your map function only on valid next()s. I think this ought to work. You can set it via JobConf.setMapRunnerClass(...). Ref: MapRunner API @ http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote: Hello, I have a simple map-reduce job that reads in zipped files and converts them to lzo compression. Some of the files are not properly zipped which results in Hadoop throwing an java.io.EOFException: Unexpected end of input stream error and causes the job to fail. Is there a way to catch this exception and tell hadoop to just ignore the file and move on? I think the exception is being thrown by the class reading in the Gzip file and not my mapper class. Is this correct? Is there a way to handle this type of error gracefully? Thank you! ~Ed -- Harsh J www.harshj.com
Re: Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException
On Wed, 20 Oct 2010 19:49:09 +0200 Erik Forsberg forsb...@opera.com wrote: Hi! I'm running Cloudera CDH2 update 2 (hadoop-0.20 0.20.1+169.113), and after the upgrade I'm getting the following error in the reducers during the copy phase in one of my larger jobs: 2010-10-20 17:43:22,343 INFO org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 12 segments... 2010-10-20 17:43:22,344 INFO org.apache.hadoop.mapred.Merger: Merging 12 sorted segments 2010-10-20 17:43:22,344 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 12 segments left of total size: 382660295 bytes 2010-10-20 17:43:22,517 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201010201640_0001_r_00_0 Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128) What does a EOFException in this code actually mean? Is it hiding some other error that could tell me more about what's wrong? I'm seeing quite a few of these in my datanode logs: 2010-10-21 10:21:01,149 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.20.11.66:50010, storageID=DS-71308762-10.20.11.66-50010-126995760, infoPort= 50075, ipcPort=50020):Got exception while serving blk_1081044479123523815_4852013 to /10.20.11.88: java.net.SocketTimeoutException: 48 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.20.11.66:50010 remote =/10.20.11.88:41347] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:401) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95) at java.lang.Thread.run(Thread.java:619) Could that be related somehow? I'm also seeing large amounts of mortbay exceptions, but MAPREDUCE-5 says they are harmless. *) Running with and without compressed map output, no difference. *) With -Xmx512m and -Xmx768m, no difference. *) Decreasing number of mappers and reducers on all nodes to decrease overall load. *) Decreasing mapred.reduce.parallel.copies from 16 to 5 (default) Also tried doubling the number of reducers to get each reducer to process less data, but that didn't help either :-( \EF -- Erik Forsberg forsb...@opera.com Developer, Opera Software - http://www.opera.com/
Re: Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException
On Thu, 21 Oct 2010 12:44:24 +0200 Erik Forsberg forsb...@opera.com wrote: attempt_201010201640_0001_r_00_0 Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128) In addition, some reducers first fail with: 010-10-21 10:31:24,696 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201010201640_0186_m_000579_0, compressed len: 2281015, decompressed len: 10368075 2010-10-21 10:31:24,696 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 10368075 bytes (2281015 raw bytes) into RAM from attempt_201010201640_0186_m_000579_0 2010-10-21 10:31:24,744 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2010-10-21 10:31:24,854 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 582879560 bytes 2010-10-21 10:31:27,396 FATAL org.apache.hadoop.mapred.TaskRunner: attempt_201010201640_0186_r_27_2 : Failed to merge in memoryjava.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:119) at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:98) at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:153) at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:122) at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373) at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:139) at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103) at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335) at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350) at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2635) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2576) 2010-10-21 10:31:27,397 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201010201640_0186_r_27_2 Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128) at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373) at org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:144) at org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103) at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335) at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350) at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2529) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:154) at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:122) ... 7 more at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2533) 2010-10-21 10:31:27,714 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting 2010-10-21 10:31:27,717 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined. 2010-10-21 10:31:27,727 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager Then, on second try, they fail with the java.io.EOFException above. \EF -- Erik Forsberg forsb...@opera.com Developer, Opera Software - http://www.opera.com/
info
hello,i would know if there is a routine to store a file on a specific datanode example... if i have 4 datanode i want that my file demo.tar is stored only on the datanode 1 and 2 thanks bye Gaetano
Re: Upgrading Hadoop from CDH3b3 to CDH3
I've got a couple of wines saved up, could be useful for this upgrade. Thanks for the heads-up. On 20 October 2010 17:23, Michael Segel michael_se...@hotmail.com wrote: CDH3B3 is the latest. My suggestion is that you get a large bottle of your favorite alcoholic beverage and enjoy it while you're doing this upgrade. In B3 you have user hadoop going to hdfs and the introduction of user mapred. So your hdfs stuff has to be owned by hdfs:hadoop and mapred stuff owned by mapred:hadoop. (That's group hadoop btw) Then you have to make sure that your unix directories have the correct file permissions because if they don't hadoop fails. Also since you're not using the start-all, start-hbase scripts you have to write your own script that uses the /etc/init.d/hadoop* scripts. Then there are zookeeper issues... But when all said and done... its more stable than hbase-0.20.3 (Cloudera's earlier release of Hbase) and 0.89 seems fairly stable in B3. (We're still testing it and trying to break it.) HTH -Mike Date: Wed, 20 Oct 2010 13:53:11 +0100 Subject: Re: Upgrading Hadoop from CDH3b3 to CDH3 From: abhinay.me...@gmail.com To: common-user@hadoop.apache.org Yes you are right, we obviously have beta2 installed on our cluster too. Thanks. On 20 October 2010 12:35, ed hadoopn...@gmail.com wrote: I don't think there is a stable CDH3 yet although we've been using CDH3B2 and it has been pretty stable for us. (at least I don't see it available on their website and they JUST announced CDH3B3 last week at HadoopWorld. ~Ed On Wed, Oct 20, 2010 at 5:57 AM, Abhinay Mehta abhinay.me...@gmail.com wrote: Hi all, We currently have Cloudera's Hadoop beta 3 installed on our cluster, we would like to upgrade to the latest stable release CDH3. Is there documentation or recommended steps on how to do this? We found some docs on how to upgrade from CDH2 and CDHb2 to CDHb3 here: https://docs.cloudera.com/display/DOC/Hadoop+Upgrade+from+CDH2+or+CDH3b2+to+CDH3b3 Are the same steps recommended to upgrade to CDH3? I'm hoping it's a lot easier to upgrade from beta3 to the latest stable version than that document states? Thank you. Abhinay Mehta
Re: Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException
544（5（56 一 �奈业� iPod �魉统� �M 在 2010/10/21 下午6:44 �r，ErikForsberg forsb...@opera.com 到： 艹 On Wed, 20 Oct 2010 19:49:09 +0200 Erik Forsberg forsb...@opera.com wrote: Hi! I'm running Cloudera CDH2 update 2 (hadoop-0.20 0.20.1+169.113), and after the upgrade I'm getting the following error in the reducers during the copy phase in one of my larger jobs: 2010-10-20 17:43:22,343 INFO org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 12 segments... 2010-10-20 17:43:22,344 INFO org.apache.hadoop.mapred.Merger: Merging 12 sorted segments 2010-10-20 17:43:22,344 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 12 segments left of total size: 382660295 bytes 2010-10-20 17:43:22,517 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201010201640_0001_r_00_0 Merging of the local FS files threw an exception: java.io.IOException: java.lang.RuntimeException: java.io.EOFException at org.apache.hadoop.io.WritableComparator.compare (WritableComparator.java:128) What does a EOFException in this code actually mean? Is it hiding some other error that could tell me more about what's wrong? I'm seeing quite a few of these in my datanode logs: 2010-10-21 10:21:01,149 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.20.11.66:50010, storageID=DS-71308762-10.20.11.66-50010-126995760, infoPort= 50075, ipcPort=50020):Got exception while serving blk_1081044479123523815_4852013 to /10.20.11.88: java.net.SocketTimeoutException: 48 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.20.11.66:50010 remote =/10.20.11.88:41347] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO (SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable (SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.transferToFully (SocketOutputStream.java:198) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks (BlockSender.java:313) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock (BlockSender.java:401) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock (DataXceiver.java:180) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run (DataXceiver.java:95) at java.lang.Thread.run(Thread.java:619) Could that be related somehow? I'm also seeing large amounts of mortbay exceptions, but MAPREDUCE-5 says they are harmless. *) Running with and without compressed map output, no difference. *) With -Xmx512m and -Xmx768m, no difference. *) Decreasing number of mappers and reducers on all nodes to decrease overall load. *) Decreasing mapred.reduce.parallel.copies from 16 to 5 (default) Also tried doubling the number of reducers to get each reducer to process less data, but that didn't help either :-( \EF -- Erik Forsberg forsb...@opera.com Developer, Opera Software - http://www.opera.com/
Re: How to stop a mapper within a map-reduce job when you detect bad input
Hello, The MapRunner classes looks promising. I noticed it is in the deprecated mapred package but I didn't see an equivalent class in the mapreduce package. Is this going to ported to mapreduce or is it no longer being supported? Thanks! ~Ed On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.com wrote: If it occurs eventually as your record reader reads it, then you may use a MapRunner class instead of a Mapper IFace/Subclass. This way, you may try/catch over the record reader itself, and call your map function only on valid next()s. I think this ought to work. You can set it via JobConf.setMapRunnerClass(...). Ref: MapRunner API @ http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote: Hello, I have a simple map-reduce job that reads in zipped files and converts them to lzo compression. Some of the files are not properly zipped which results in Hadoop throwing an java.io.EOFException: Unexpected end of input stream error and causes the job to fail. Is there a way to catch this exception and tell hadoop to just ignore the file and move on? I think the exception is being thrown by the class reading in the Gzip file and not my mapper class. Is this correct? Is there a way to handle this type of error gracefully? Thank you! ~Ed -- Harsh J www.harshj.com
Re: How to stop a mapper within a map-reduce job when you detect bad input
Just checked the Hadoop 0.21.0 API docs (I was looking in the wrong docs before) and it doesn't look like MapRunner is deprecated so I'll try catching the error there and will report back if it's a good solution. Thanks! ~Ed On Thu, Oct 21, 2010 at 11:23 AM, ed hadoopn...@gmail.com wrote: Hello, The MapRunner classes looks promising. I noticed it is in the deprecated mapred package but I didn't see an equivalent class in the mapreduce package. Is this going to ported to mapreduce or is it no longer being supported? Thanks! ~Ed On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.com wrote: If it occurs eventually as your record reader reads it, then you may use a MapRunner class instead of a Mapper IFace/Subclass. This way, you may try/catch over the record reader itself, and call your map function only on valid next()s. I think this ought to work. You can set it via JobConf.setMapRunnerClass(...). Ref: MapRunner API @ http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote: Hello, I have a simple map-reduce job that reads in zipped files and converts them to lzo compression. Some of the files are not properly zipped which results in Hadoop throwing an java.io.EOFException: Unexpected end of input stream error and causes the job to fail. Is there a way to catch this exception and tell hadoop to just ignore the file and move on? I think the exception is being thrown by the class reading in the Gzip file and not my mapper class. Is this correct? Is there a way to handle this type of error gracefully? Thank you! ~Ed -- Harsh J www.harshj.com
Re: How to stop a mapper within a map-reduce job when you detect bad input
On Thu, Oct 21, 2010 at 8:23 AM, ed hadoopn...@gmail.com wrote: Hello, The MapRunner classes looks promising. I noticed it is in the deprecated mapred package but I didn't see an equivalent class in the mapreduce package. Is this going to ported to mapreduce or is it no longer being supported? Thanks! The equivalent functionality is in org.apache.hadoop.mapreduce.Mapper#run. Cheers Tom ~Ed On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.com wrote: If it occurs eventually as your record reader reads it, then you may use a MapRunner class instead of a Mapper IFace/Subclass. This way, you may try/catch over the record reader itself, and call your map function only on valid next()s. I think this ought to work. You can set it via JobConf.setMapRunnerClass(...). Ref: MapRunner API @ http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote: Hello, I have a simple map-reduce job that reads in zipped files and converts them to lzo compression. Some of the files are not properly zipped which results in Hadoop throwing an java.io.EOFException: Unexpected end of input stream error and causes the job to fail. Is there a way to catch this exception and tell hadoop to just ignore the file and move on? I think the exception is being thrown by the class reading in the Gzip file and not my mapper class. Is this correct? Is there a way to handle this type of error gracefully? Thank you! ~Ed -- Harsh J www.harshj.com
Re: BUG: Anyone use block size more than 2GB before?
I thought the petasort benchmark you published used 12.5G block sizes. How did you make that work? On Mon, Oct 18, 2010 at 4:27 PM, Owen O'Malley omal...@apache.org wrote: Block sizes larger than 2**31 are known to not work. I haven't ever tracked down the problem, just set my block size to be smaller than that. -- Owen
Re: How to stop a mapper within a map-reduce job when you detect bad input
Sorry to keep spamming this thread. It looks like the correct way to implement MapRunnable using the new mapreduce classes (instead of the deprecated mapred) is to override the run() method of the mapper class. This is actually nice and convenient since everyone should already be using Mapper class (org.apache.hadoop.mapreduce.MaperKEYIN, VALUEIN, KEYOUT, VALUEOUT for their mappers. ~Ed On Thu, Oct 21, 2010 at 12:14 PM, ed hadoopn...@gmail.com wrote: Just checked the Hadoop 0.21.0 API docs (I was looking in the wrong docs before) and it doesn't look like MapRunner is deprecated so I'll try catching the error there and will report back if it's a good solution. Thanks! ~Ed On Thu, Oct 21, 2010 at 11:23 AM, ed hadoopn...@gmail.com wrote: Hello, The MapRunner classes looks promising. I noticed it is in the deprecated mapred package but I didn't see an equivalent class in the mapreduce package. Is this going to ported to mapreduce or is it no longer being supported? Thanks! ~Ed On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.com wrote: If it occurs eventually as your record reader reads it, then you may use a MapRunner class instead of a Mapper IFace/Subclass. This way, you may try/catch over the record reader itself, and call your map function only on valid next()s. I think this ought to work. You can set it via JobConf.setMapRunnerClass(...). Ref: MapRunner API @ http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote: Hello, I have a simple map-reduce job that reads in zipped files and converts them to lzo compression. Some of the files are not properly zipped which results in Hadoop throwing an java.io.EOFException: Unexpected end of input stream error and causes the job to fail. Is there a way to catch this exception and tell hadoop to just ignore the file and move on? I think the exception is being thrown by the class reading in the Gzip file and not my mapper class. Is this correct? Is there a way to handle this type of error gracefully? Thank you! ~Ed -- Harsh J www.harshj.com
Re: How to stop a mapper within a map-reduce job when you detect bad input
Thanks Tom! Didn't see your post before posting =) On Thu, Oct 21, 2010 at 1:28 PM, ed hadoopn...@gmail.com wrote: Sorry to keep spamming this thread. It looks like the correct way to implement MapRunnable using the new mapreduce classes (instead of the deprecated mapred) is to override the run() method of the mapper class. This is actually nice and convenient since everyone should already be using Mapper class (org.apache.hadoop.mapreduce.MaperKEYIN, VALUEIN, KEYOUT, VALUEOUT for their mappers. ~Ed On Thu, Oct 21, 2010 at 12:14 PM, ed hadoopn...@gmail.com wrote: Just checked the Hadoop 0.21.0 API docs (I was looking in the wrong docs before) and it doesn't look like MapRunner is deprecated so I'll try catching the error there and will report back if it's a good solution. Thanks! ~Ed On Thu, Oct 21, 2010 at 11:23 AM, ed hadoopn...@gmail.com wrote: Hello, The MapRunner classes looks promising. I noticed it is in the deprecated mapred package but I didn't see an equivalent class in the mapreduce package. Is this going to ported to mapreduce or is it no longer being supported? Thanks! ~Ed On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.com wrote: If it occurs eventually as your record reader reads it, then you may use a MapRunner class instead of a Mapper IFace/Subclass. This way, you may try/catch over the record reader itself, and call your map function only on valid next()s. I think this ought to work. You can set it via JobConf.setMapRunnerClass(...). Ref: MapRunner API @ http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote: Hello, I have a simple map-reduce job that reads in zipped files and converts them to lzo compression. Some of the files are not properly zipped which results in Hadoop throwing an java.io.EOFException: Unexpected end of input stream error and causes the job to fail. Is there a way to catch this exception and tell hadoop to just ignore the file and move on? I think the exception is being thrown by the class reading in the Gzip file and not my mapper class. Is this correct? Is there a way to handle this type of error gracefully? Thank you! ~Ed -- Harsh J www.harshj.com
Re: How to stop a mapper within a map-reduce job when you detect bad input
I overwrote the run() method in the mapper with a run() method (below) that
catches the EOFException. The mapper and reducer now complete but the
outputted lzo file from the reducer throws an Unexpected End of File error
when decompressing it indicating something did not clean up properly. I
can't think of why this could be happening as the map() method should only
be called on input that was properly decompressed (anything that can't be
decompressed will throw an Exception that is being caught). The reducer
then should not even know that the mapper hit an EOFException in the input
gzip file, and yet the output lzo file still has the unexpected end of file
problem (I'm using the kevinweil lzo libraries). Is there some call that
needs to be made that will close out the mapper and ensure that the lzo
output from the reducer is formatted properly? Thank you!
@Override
public void run(Context context) throw InterruptedException{
try{
setup(context);
while(context.nextKeyValue()){
map(context.getCurrentKey(), context.getCurrentValue(),
context);
}
cleanup(context);
} catch(EOFException){
logError(context, EOFException: Corrupt gzip file + mFileName);
}
}
On Thu, Oct 21, 2010 at 1:29 PM, ed hadoopn...@gmail.com wrote:
Thanks Tom! Didn't see your post before posting =)
On Thu, Oct 21, 2010 at 1:28 PM, ed hadoopn...@gmail.com wrote:
Sorry to keep spamming this thread. It looks like the correct way to
implement MapRunnable using the new mapreduce classes (instead of the
deprecated mapred) is to override the run() method of the mapper class.
This is actually nice and convenient since everyone should already be using
Mapper class (org.apache.hadoop.mapreduce.MaperKEYIN, VALUEIN, KEYOUT,
VALUEOUT for their mappers.
~Ed
On Thu, Oct 21, 2010 at 12:14 PM, ed hadoopn...@gmail.com wrote:
Just checked the Hadoop 0.21.0 API docs (I was looking in the wrong docs
before) and it doesn't look like MapRunner is deprecated so I'll try
catching the error there and will report back if it's a good solution.
Thanks!
~Ed
On Thu, Oct 21, 2010 at 11:23 AM, ed hadoopn...@gmail.com wrote:
Hello,
The MapRunner classes looks promising. I noticed it is in the
deprecated mapred package but I didn't see an equivalent class in the
mapreduce package. Is this going to ported to mapreduce or is it no longer
being supported? Thanks!
~Ed
On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.comwrote:
If it occurs eventually as your record reader reads it, then you may
use a MapRunner class instead of a Mapper IFace/Subclass. This way,
you may try/catch over the record reader itself, and call your map
function only on valid next()s. I think this ought to work.
You can set it via JobConf.setMapRunnerClass(...).
Ref: MapRunner API @
http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html
On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote:
Hello,
I have a simple map-reduce job that reads in zipped files and
converts them
to lzo compression. Some of the files are not properly zipped which
results
in Hadoop throwing an java.io.EOFException: Unexpected end of input
stream
error and causes the job to fail. Is there a way to catch this
exception
and tell hadoop to just ignore the file and move on? I think the
exception
is being thrown by the class reading in the Gzip file and not my
mapper
class. Is this correct? Is there a way to handle this type of error
gracefully?
Thank you!
~Ed
--
Harsh J
www.harshj.com
Re: BUG: Anyone use block size more than 2GB before?
The block sizes were 2G. The input format made splits that were more than a block because that led to better performance. -- Owen
Re: BUG: Anyone use block size more than 2GB before?
Hmm, this is interesting: how did it manage to keep the blocks local? Why performance was better? On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley omal...@apache.org wrote: The block sizes were 2G. The input format made splits that were more than a block because that led to better performance. -- Owen
Re: Setting num reduce tasks
A little more detail: I have mapred.tasktracker.reduce.tasks.maximum set to 4 and I have 5 nodes. On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary matt.tanqu...@gmail.comwrote: I am using the following to set my number of reduce tasks, however when I run my job it's always using just 1 reducer. conf.setInt(mapred.reduce.tasks, 20); 1 reducer will never finish this job. Please help me to understand why the setting I choose is not used. Thanks, -M@ -- Have you thanked a teacher today? --- http://www.liftateacher.org
Re: Setting num reduce tasks
Hi Matt, it might be that the parameter does not end up in the final configuration for a number of reasons. Can you check the job config xml in jt:/var/log/hadoop/history or in the JT UI and see what the mapred.reduce.tasks setting is? -- Alex K On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary matt.tanqu...@gmail.comwrote: I am using the following to set my number of reduce tasks, however when I run my job it's always using just 1 reducer. conf.setInt(mapred.reduce.tasks, 20); 1 reducer will never finish this job. Please help me to understand why the setting I choose is not used. Thanks, -M@
Re: Setting num reduce tasks
You could also try job.setNumReduceTasks(yourNumber); ~Ed On Thu, Oct 21, 2010 at 4:45 PM, Alex Kozlov ale...@cloudera.com wrote: Hi Matt, it might be that the parameter does not end up in the final configuration for a number of reasons. Can you check the job config xml in jt:/var/log/hadoop/history or in the JT UI and see what the mapred.reduce.tasks setting is? -- Alex K On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary matt.tanqu...@gmail.com wrote: I am using the following to set my number of reduce tasks, however when I run my job it's always using just 1 reducer. conf.setInt(mapred.reduce.tasks, 20); 1 reducer will never finish this job. Please help me to understand why the setting I choose is not used. Thanks, -M@
Re: Setting num reduce tasks
Hi Alex, Yes, I confirmed from those locations that the job is setting the reducers to 1. Thanks On Thu, Oct 21, 2010 at 1:45 PM, Alex Kozlov ale...@cloudera.com wrote: Hi Matt, it might be that the parameter does not end up in the final configuration for a number of reasons. Can you check the job config xml in jt:/var/log/hadoop/history or in the JT UI and see what the mapred.reduce.tasks setting is? -- Alex K On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary matt.tanqu...@gmail.com wrote: I am using the following to set my number of reduce tasks, however when I run my job it's always using just 1 reducer. conf.setInt(mapred.reduce.tasks, 20); 1 reducer will never finish this job. Please help me to understand why the setting I choose is not used. Thanks, -M@ -- Have you thanked a teacher today? --- http://www.liftateacher.org
Re: Setting num reduce tasks
It looks like you do not pass the Configuration object correctly. Do you use old or new (mapreduce) API? Do you have something like Job job = new Job(conf, My job with + conf.get(mapred.reduce.tasks) + reducers); to create the job? Is it OK to share you job creation code? Alex K On Thu, Oct 21, 2010 at 2:25 PM, Matt Tanquary matt.tanqu...@gmail.comwrote: Hi Alex, Yes, I confirmed from those locations that the job is setting the reducers to 1. Thanks On Thu, Oct 21, 2010 at 1:45 PM, Alex Kozlov ale...@cloudera.com wrote: Hi Matt, it might be that the parameter does not end up in the final configuration for a number of reasons. Can you check the job config xml in jt:/var/log/hadoop/history or in the JT UI and see what the mapred.reduce.tasks setting is? -- Alex K On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary matt.tanqu...@gmail.com wrote: I am using the following to set my number of reduce tasks, however when I run my job it's always using just 1 reducer. conf.setInt(mapred.reduce.tasks, 20); 1 reducer will never finish this job. Please help me to understand why the setting I choose is not used. Thanks, -M@ -- Have you thanked a teacher today? --- http://www.liftateacher.org
Re: Setting num reduce tasks
I used the new API. I just tried the job.setNumReduceTasks() method and it seems to be working. Thanks Alex and Ed for helping me! On Thu, Oct 21, 2010 at 2:32 PM, Alex Kozlov ale...@cloudera.com wrote: It looks like you do not pass the Configuration object correctly. Do you use old or new (mapreduce) API? Do you have something like Job job = new Job(conf, My job with + conf.get(mapred.reduce.tasks) + reducers); to create the job? Is it OK to share you job creation code? Alex K On Thu, Oct 21, 2010 at 2:25 PM, Matt Tanquary matt.tanqu...@gmail.com wrote: Hi Alex, Yes, I confirmed from those locations that the job is setting the reducers to 1. Thanks On Thu, Oct 21, 2010 at 1:45 PM, Alex Kozlov ale...@cloudera.com wrote: Hi Matt, it might be that the parameter does not end up in the final configuration for a number of reasons. Can you check the job config xml in jt:/var/log/hadoop/history or in the JT UI and see what the mapred.reduce.tasks setting is? -- Alex K On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary matt.tanqu...@gmail.com wrote: I am using the following to set my number of reduce tasks, however when I run my job it's always using just 1 reducer. conf.setInt(mapred.reduce.tasks, 20); 1 reducer will never finish this job. Please help me to understand why the setting I choose is not used. Thanks, -M@ -- Have you thanked a teacher today? --- http://www.liftateacher.org -- Have you thanked a teacher today? --- http://www.liftateacher.org
LZO Compression Libraries don't appear to work properly with MultipleOutputs
Hello everyone,
I am having problems using MultipleOutputs with LZO compression (could be a
bug or something wrong in my own code).
In my driver I set
MultipleOutputs.addNamedOutput(job, test, TextOutputFormat.class,
NullWritable.class, Text.class);
In my reducer I have:
MultipleOutputsNullWritable, Text mOutput = new
MultipleOutputsNullWritable, Text(context);
public String generateFileName(Key key){
return custom_file_name;
}
Then in the reduce() method I have:
mOutput.write(mNullWritable, mValue, generateFileName(key));
This results in creating LZO files that do not decompress properly (lzop -d
throws the error lzop: unexpected end of file: outputFile.lzo)
If I switch back to the regular context.write(mNullWritable, mValue);
everything works fine.
Am I forgetting a step needed when using MultipleOutputs or is this a
bug/non-feature of using LZO compression in Hadoop.
Thank you!
~Ed
Re: How to stop a mapper within a map-reduce job when you detect bad input
So the overwritten run() method was a red herring. The real problem appears
to be that I use MultipleOutputs (the new mapreduce API version) for my
reducer output. I posted a different thread since it's not really related
to the original question here. For everyone that was curious, it turns our
overriding the run() method and catching the EOFException works beautifully
for processing files that might be corrupt or have errors. Thanks!
~Ed
On Thu, Oct 21, 2010 at 2:07 PM, ed hadoopn...@gmail.com wrote:
I overwrote the run() method in the mapper with a run() method (below) that
catches the EOFException. The mapper and reducer now complete but the
outputted lzo file from the reducer throws an Unexpected End of File error
when decompressing it indicating something did not clean up properly. I
can't think of why this could be happening as the map() method should only
be called on input that was properly decompressed (anything that can't be
decompressed will throw an Exception that is being caught). The reducer
then should not even know that the mapper hit an EOFException in the input
gzip file, and yet the output lzo file still has the unexpected end of file
problem (I'm using the kevinweil lzo libraries). Is there some call that
needs to be made that will close out the mapper and ensure that the lzo
output from the reducer is formatted properly? Thank you!
@Override
public void run(Context context) throw InterruptedException{
try{
setup(context);
while(context.nextKeyValue()){
map(context.getCurrentKey(), context.getCurrentValue(),
context);
}
cleanup(context);
} catch(EOFException){
logError(context, EOFException: Corrupt gzip file +
mFileName);
}
}
On Thu, Oct 21, 2010 at 1:29 PM, ed hadoopn...@gmail.com wrote:
Thanks Tom! Didn't see your post before posting =)
On Thu, Oct 21, 2010 at 1:28 PM, ed hadoopn...@gmail.com wrote:
Sorry to keep spamming this thread. It looks like the correct way to
implement MapRunnable using the new mapreduce classes (instead of the
deprecated mapred) is to override the run() method of the mapper class.
This is actually nice and convenient since everyone should already be using
Mapper class (org.apache.hadoop.mapreduce.MaperKEYIN, VALUEIN, KEYOUT,
VALUEOUT for their mappers.
~Ed
On Thu, Oct 21, 2010 at 12:14 PM, ed hadoopn...@gmail.com wrote:
Just checked the Hadoop 0.21.0 API docs (I was looking in the wrong docs
before) and it doesn't look like MapRunner is deprecated so I'll try
catching the error there and will report back if it's a good solution.
Thanks!
~Ed
On Thu, Oct 21, 2010 at 11:23 AM, ed hadoopn...@gmail.com wrote:
Hello,
The MapRunner classes looks promising. I noticed it is in the
deprecated mapred package but I didn't see an equivalent class in the
mapreduce package. Is this going to ported to mapreduce or is it no
longer
being supported? Thanks!
~Ed
On Thu, Oct 21, 2010 at 6:36 AM, Harsh J qwertyman...@gmail.comwrote:
If it occurs eventually as your record reader reads it, then you may
use a MapRunner class instead of a Mapper IFace/Subclass. This way,
you may try/catch over the record reader itself, and call your map
function only on valid next()s. I think this ought to work.
You can set it via JobConf.setMapRunnerClass(...).
Ref: MapRunner API @
http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/mapred/MapRunner.html
On Wed, Oct 20, 2010 at 4:14 AM, ed hadoopn...@gmail.com wrote:
Hello,
I have a simple map-reduce job that reads in zipped files and
converts them
to lzo compression. Some of the files are not properly zipped which
results
in Hadoop throwing an java.io.EOFException: Unexpected end of input
stream
error and causes the job to fail. Is there a way to catch this
exception
and tell hadoop to just ignore the file and move on? I think the
exception
is being thrown by the class reading in the Gzip file and not my
mapper
class. Is this correct? Is there a way to handle this type of
error
gracefully?
Thank you!
~Ed
--
Harsh J
www.harshj.com
Re: LZO Compression Libraries don't appear to work properly with MultipleOutputs
Hi Ed,
Sounds like this might be a bug, either in MultipleOutputs or in LZO.
Does it work properly with gzip compression? Which LZO implementation
are you using? The one from google code or the more up to date one
from github (either kevinweil's or mine)?
Any chance you could write a unit test that shows the issue?
Thanks
-Todd
On Thu, Oct 21, 2010 at 2:52 PM, ed hadoopn...@gmail.com wrote:
Hello everyone,
I am having problems using MultipleOutputs with LZO compression (could be a
bug or something wrong in my own code).
In my driver I set
MultipleOutputs.addNamedOutput(job, test, TextOutputFormat.class,
NullWritable.class, Text.class);
In my reducer I have:
MultipleOutputsNullWritable, Text mOutput = new
MultipleOutputsNullWritable, Text(context);
public String generateFileName(Key key){
return custom_file_name;
}
Then in the reduce() method I have:
mOutput.write(mNullWritable, mValue, generateFileName(key));
This results in creating LZO files that do not decompress properly (lzop -d
throws the error lzop: unexpected end of file: outputFile.lzo)
If I switch back to the regular context.write(mNullWritable, mValue);
everything works fine.
Am I forgetting a step needed when using MultipleOutputs or is this a
bug/non-feature of using LZO compression in Hadoop.
Thank you!
~Ed
--
Todd Lipcon
Software Engineer, Cloudera
Re: specify number of mappers/reducers per node per job?
Actually I believe this cannot be achieved by current m/r scheduler since as I understand, a scheduler can only control how much resource in total to be assigned to a job but it does not allow external control which node a mapper/reducer will run on, which is reasonable (jobtracker has to base its task assignment on the current status of the cluster, like if a node is blacklisted, then you cannot run any tasks on it). Thanks, Michael --- On Thu, 10/21/10, jiang licht licht_ji...@yahoo.com wrote: From: jiang licht licht_ji...@yahoo.com Subject: Re: specify number of mappers/reducers per node per job? To: common-user@hadoop.apache.org Date: Thursday, October 21, 2010, 12:38 PM Thanks, Harsh. That's my suspect as well. It's part of what a schedule is responsible for, s.a. capacity scheduler. Want to make sure I didn't miss it if there is another node-wise property at job-level to limit resource available to a job in parallel with mapred.tasktracker.*.tasks.maximum. Thanks, Michael --- On Thu, 10/21/10, Harsh J qwertyman...@gmail.com wrote: From: Harsh J qwertyman...@gmail.com Subject: Re: specify number of mappers/reducers per node per job? To: common-user@hadoop.apache.org Date: Thursday, October 21, 2010, 1:03 AM AFAIK there is no way to control this from a job submission perspective. Maybe the scheduler concept in Hadoop MapReduce can help you. -- Harsh J http://www.harshj.com On Oct 21, 2010 6:32 AM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to control maximum number of mappers or reducers per node per job? i.e. say I have a cluster, now I want to run a job such that on each node, no more than 2 mappers are expected to run at the same time (but maximum numbers of m/r slots on a node are some bigger numbers specified by mapred.tasktracker.map.tasks.maximum, mapred.tasktracker.reduce.tasks.maximum). Thanks, Michael
Re: specify number of mappers/reducers per node per job?
Thanks, Harsh. That's my suspect as well. It's part of what a schedule is responsible for, s.a. capacity scheduler. Want to make sure I didn't miss it if there is another node-wise property at job-level to limit resource available to a job in parallel with mapred.tasktracker.*.tasks.maximum. Thanks, Michael --- On Thu, 10/21/10, Harsh J qwertyman...@gmail.com wrote: From: Harsh J qwertyman...@gmail.com Subject: Re: specify number of mappers/reducers per node per job? To: common-user@hadoop.apache.org Date: Thursday, October 21, 2010, 1:03 AM AFAIK there is no way to control this from a job submission perspective. Maybe the scheduler concept in Hadoop MapReduce can help you. -- Harsh J http://www.harshj.com On Oct 21, 2010 6:32 AM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to control maximum number of mappers or reducers per node per job? i.e. say I have a cluster, now I want to run a job such that on each node, no more than 2 mappers are expected to run at the same time (but maximum numbers of m/r slots on a node are some bigger numbers specified by mapred.tasktracker.map.tasks.maximum, mapred.tasktracker.reduce.tasks.maximum). Thanks, Michael
Re: LZO Compression Libraries don't appear to work properly with MultipleOutputs
Hi Todd,
I don't have the code in front of me right but I was looking over the API
docs and it looks like I forgot to call close() on the MultipleOutput. I'll
post back if that fixes the problem. If not I'll put together a unit test.
Thanks!
~Ed
On Thu, Oct 21, 2010 at 6:31 PM, Todd Lipcon t...@cloudera.com wrote:
Hi Ed,
Sounds like this might be a bug, either in MultipleOutputs or in LZO.
Does it work properly with gzip compression? Which LZO implementation
are you using? The one from google code or the more up to date one
from github (either kevinweil's or mine)?
Any chance you could write a unit test that shows the issue?
Thanks
-Todd
On Thu, Oct 21, 2010 at 2:52 PM, ed hadoopn...@gmail.com wrote:
Hello everyone,
I am having problems using MultipleOutputs with LZO compression (could be
a
bug or something wrong in my own code).
In my driver I set
MultipleOutputs.addNamedOutput(job, test, TextOutputFormat.class,
NullWritable.class, Text.class);
In my reducer I have:
MultipleOutputsNullWritable, Text mOutput = new
MultipleOutputsNullWritable, Text(context);
public String generateFileName(Key key){
return custom_file_name;
}
Then in the reduce() method I have:
mOutput.write(mNullWritable, mValue, generateFileName(key));
This results in creating LZO files that do not decompress properly (lzop
-d
throws the error lzop: unexpected end of file: outputFile.lzo)
If I switch back to the regular context.write(mNullWritable, mValue);
everything works fine.
Am I forgetting a step needed when using MultipleOutputs or is this a
bug/non-feature of using LZO compression in Hadoop.
Thank you!
~Ed
--
Todd Lipcon
Software Engineer, Cloudera
Re: BUG: Anyone use block size more than 2GB before?
If a file of say, 12.5 GB were produced by a single task with replication 3, the default replication policy will ensure that the first replica of each block will be created on local datanode. So, there will be one datanode in the cluster that contains one replica of all blocks of that file. Map placement hint specifies that node. It's evil, I know :-) - Milind On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote: Hmm, this is interesting: how did it manage to keep the blocks local? Why performance was better? On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley omal...@apache.org wrote: The block sizes were 2G. The input format made splits that were more than a block because that led to better performance. -- Owen -- Milind Bhandarkar (mailto:mili...@yahoo-inc.com) (phone: 408-203-5213 W)
Re: BUG: Anyone use block size more than 2GB before?
Milind, You are right. But that only happens when your client is one of the data nodes in HDFS. otherwise a random node will be picked up for the first replica. On Fri, Oct 22, 2010 at 3:37 PM, Milind A Bhandarkar mili...@yahoo-inc.comwrote: If a file of say, 12.5 GB were produced by a single task with replication 3, the default replication policy will ensure that the first replica of each block will be created on local datanode. So, there will be one datanode in the cluster that contains one replica of all blocks of that file. Map placement hint specifies that node. It's evil, I know :-) - Milind On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote: Hmm, this is interesting: how did it manage to keep the blocks local? Why performance was better? On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley omal...@apache.org wrote: The block sizes were 2G. The input format made splits that were more than a block because that led to better performance. -- Owen -- Milind Bhandarkar (mailto:mili...@yahoo-inc.com) (phone: 408-203-5213 W)
Re: BUG: Anyone use block size more than 2GB before?
That's correct. That is why teragen, the program that generates data to be sorted in terasort is a MR program :-) - Milind On Oct 21, 2010, at 9:47 PM, elton sky wrote: Milind, You are right. But that only happens when your client is one of the data nodes in HDFS. otherwise a random node will be picked up for the first replica. On Fri, Oct 22, 2010 at 3:37 PM, Milind A Bhandarkar mili...@yahoo-inc.comwrote: If a file of say, 12.5 GB were produced by a single task with replication 3, the default replication policy will ensure that the first replica of each block will be created on local datanode. So, there will be one datanode in the cluster that contains one replica of all blocks of that file. Map placement hint specifies that node. It's evil, I know :-) - Milind On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote: Hmm, this is interesting: how did it manage to keep the blocks local? Why performance was better? On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley omal...@apache.org wrote: The block sizes were 2G. The input format made splits that were more than a block because that led to better performance. -- Owen -- Milind Bhandarkar (mailto:mili...@yahoo-inc.com) (phone: 408-203-5213 W) -- Milind Bhandarkar (mailto:mili...@yahoo-inc.com) (phone: 408-203-5213 W)
