Re: 0.18.1 datanode psuedo deadlock problem
I propose an alternate solution for this. If the block information was managed by having a inotify task (in linux/solaris), and the windows equivalent which I forget, the datanode could be informed each time a file in the dfs tree is created, updated, or deleted. With this information being delivered, it can maintain an accurate block map with only 1 full scan of the datanode blocks, at start time. With this algorithm the data nodes will be able to scale to a much larger number of blocks. The other thing is the way the sync blocks on the FSDataset.FSVolumeSet are held totally aggravates this bug in 0.18.1. I have implemented a pure java version of inotify, using JNA (https://jna.dev.java.net/) and there is a windows version also available, or some simple jni could be written. The ja...@attributor.com address will be going away shortly, I will be switching to jason.had...@gmail.com in the next little bit. Jason Venner wrote: The problem we are having is that datanodes periodically stall for 10-15 minutes and drop off the active list and then come back. What is going on is that a long operation set is holding the lock on on FSDataset.volumes, and all of the other block service requests stall behind this lock. "DataNode: [/data/dfs-video-18/dfs/data]" daemon prio=10 tid=0x4d7ad400 nid=0x7c40 runnable [0x4c698000..0x4c6990d0] java.lang.Thread.State: RUNNABLE at java.lang.String.lastIndexOf(String.java:1628) at java.io.File.getName(File.java:399) at org.apache.hadoop.dfs.FSDataset$FSDir.getGenerationStampFromFile(FSDataset.java:148) at org.apache.hadoop.dfs.FSDataset$FSDir.getBlockInfo(FSDataset.java:181) at org.apache.hadoop.dfs.FSDataset$FSVolume.getBlockInfo(FSDataset.java:412) at org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getBlockInfo(FSDataset.java:511) - locked <0x551e8d48> (a org.apache.hadoop.dfs.FSDataset$FSVolumeSet) at org.apache.hadoop.dfs.FSDataset.getBlockReport(FSDataset.java:1053) at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:708) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2890) at java.lang.Thread.run(Thread.java:619) This is basically taking a stat on every hdfs block on the datanode, which in our case is ~ 2million, and can take 10+ minutes (we may be experiencing problems with our raid controller but have no visibility into it) at the OS level the file system seems fine and operations eventually finish. It appears that a couple of different data structures are being locked with the single object FSDataset$Volume. Then this happens: "org.apache.hadoop.dfs.datanode$dataxcei...@1bcee17" daemon prio=10 tid=0x4da8d000 nid=0x7ae4 waiting for monitor entry [0x459fe000..0x459ff0d0] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getNextVolume(FSDataset.java:473) - waiting to lock <0x551e8d48> (a org.apache.hadoop.dfs.FSDataset$FSVolumeSet) at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:934) - locked <0x54e550e0> (a org.apache.hadoop.dfs.FSDataset) at org.apache.hadoop.dfs.DataNode$BlockReceiver.(DataNode.java:2322) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1187) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1045) at java.lang.Thread.run(Thread.java:619) which locks the FSDataset while waiting on the volume object and now all of the Datanode operations stall waiting on the FSDataset object. -- Our particular installation doesn't use multiple directories for hdfs, so a first simple hack for a local fix would be to modify getNextVolume to just return the single volume and not be synchronized A richer alternative would be to make the locking more fine grained on FSDataset$FSVolumeSet. Of course we are also trying to fix the file system performance and dfs block loading that results in the block report taking a long time. Any suggestions or warnings? Thanks.
RE: Job Tracker/Name Node redundancy
Ryan, >From the MR (JobTracker) side we have a failover support. If a large job is submitted and the JobTracker fails midway then you can start the JobTracker on the same host and resume the job. Look at https://issues.apache.org/jira/browse/HADOOP-3245 for more details. Hope that helps. Amar -Original Message- From: Ryan LeCompte [mailto:lecom...@gmail.com] Sent: Fri 1/9/2009 12:09 PM To: core-user@hadoop.apache.org Subject: Job Tracker/Name Node redundancy Are there any plans to build redundancy/failover support for the Job Tracker and Name Node components in Hadoop? Let's take the current scenario: 1) A data/cpu intensive job is submitted to a Hadoop cluster of 10 machines. 2) Half-way through the job execution, the Job Tracker or Name Node fails. 3) We bring up a new Job Tracker or Name Node manually. -- Will the individual task trackers / data nodes "reconnect" to the new masters? Or will the job have to be resubmitted? If we had failover support, we could setup essentially 3 Job Tracker masters and 3 NameNode masters so that if one dies the other would gracefully take over and start handling results from the children nodes. Thanks! Ryan
Problem with Hadoop and concatenated gzip files
Hi, I'm having trouble with Hadoop (tested with 0.17 and 0.19) not fully processing certain gzipped input files. Basically it only actually reads and processes a first part of the gzipped file, and just ignores the rest without any kind of warning. It affects at least (but is maybe not limited to?) any gzip files that are a result of concatenation (which should be legal to do with gzip format): http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage Repro case, using the "WordCount" example from the hadoop distribution: $ echo 'one two three' > f1 $ echo 'four five six' > f2 $ gzip -c f1 > combined_file.gz $ gzip -c f2 >> combined_file.gz Now, if I run "WordCount" with combined_file.gz as input, it will only find the words 'one', 'two', 'three', but not 'four', 'five', 'six'. It seems Java's GZIPInputStream may have a similar issue: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4691425 Now, if I unzip and re-gzip this 'combined_file.gz' manually, the problem goes away. It's especially dangerous since Hadoop doesn't show any errors or complains in the least. It just ignores this extra input. The only way of noticing is to run one's app with gzipped- and unzipped data side by side and notice the record counts being different. Is anyone else familiar with this problem? Any solutions, workarounds, short of re-gzipping very large amounts of data? Thanks! / Oscar The information transmitted in this email is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and permanently delete the email from any computer.
Re: 0.18.1 datanode psuedo deadlock problem
2M files is excessive. But there is no reason block reports should break. My preference is to make block reports handle this better. DNs dropping in and out of the cluster causes too many other problems. Raghu. Konstantin Shvachko wrote: Hi Jason, 2 million blocks per data-node is not going to work. There were discussions about it previously, please check the mail archives. This means you have a lot of very small files, which HDFS is not designed to support. A general recommendation is to group small files into large ones, introducing some kind of record structure delimiting those small files, and control it in on the application level. Thanks, --Konstantin
Re: 0.18.1 datanode psuedo deadlock problem
The scan required for each block report is well known issue and it can be fixed. It was discussed multiple times (e.g. https://issues.apache.org/jira/browse/HADOOP-3232?focusedCommentId=12587795#action_12587795 ). Earlier, inline 'du' on datanodes used to cause the same problem and they they were moved to a separate thread (HADOOP-3232). block reports can do the same... Though 2M blocks on DN is very large, there is no reason block reports should break things. Once we fix block reports, something else might break.. but that is different issue. Raghu. Jason Venner wrote: The problem we are having is that datanodes periodically stall for 10-15 minutes and drop off the active list and then come back. What is going on is that a long operation set is holding the lock on on FSDataset.volumes, and all of the other block service requests stall behind this lock. "DataNode: [/data/dfs-video-18/dfs/data]" daemon prio=10 tid=0x4d7ad400 nid=0x7c40 runnable [0x4c698000..0x4c6990d0] java.lang.Thread.State: RUNNABLE at java.lang.String.lastIndexOf(String.java:1628) at java.io.File.getName(File.java:399) at org.apache.hadoop.dfs.FSDataset$FSDir.getGenerationStampFromFile(FSDataset.java:148) at org.apache.hadoop.dfs.FSDataset$FSDir.getBlockInfo(FSDataset.java:181) at org.apache.hadoop.dfs.FSDataset$FSVolume.getBlockInfo(FSDataset.java:412) at org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getBlockInfo(FSDataset.java:511) - locked <0x551e8d48> (a org.apache.hadoop.dfs.FSDataset$FSVolumeSet) at org.apache.hadoop.dfs.FSDataset.getBlockReport(FSDataset.java:1053) at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:708) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2890) at java.lang.Thread.run(Thread.java:619) This is basically taking a stat on every hdfs block on the datanode, which in our case is ~ 2million, and can take 10+ minutes (we may be experiencing problems with our raid controller but have no visibility into it) at the OS level the file system seems fine and operations eventually finish. It appears that a couple of different data structures are being locked with the single object FSDataset$Volume. Then this happens: "org.apache.hadoop.dfs.datanode$dataxcei...@1bcee17" daemon prio=10 tid=0x4da8d000 nid=0x7ae4 waiting for monitor entry [0x459fe000..0x459ff0d0] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getNextVolume(FSDataset.java:473) - waiting to lock <0x551e8d48> (a org.apache.hadoop.dfs.FSDataset$FSVolumeSet) at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:934) - locked <0x54e550e0> (a org.apache.hadoop.dfs.FSDataset) at org.apache.hadoop.dfs.DataNode$BlockReceiver.(DataNode.java:2322) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1187) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1045) at java.lang.Thread.run(Thread.java:619) which locks the FSDataset while waiting on the volume object and now all of the Datanode operations stall waiting on the FSDataset object. -- Our particular installation doesn't use multiple directories for hdfs, so a first simple hack for a local fix would be to modify getNextVolume to just return the single volume and not be synchronized A richer alternative would be to make the locking more fine grained on FSDataset$FSVolumeSet. Of course we are also trying to fix the file system performance and dfs block loading that results in the block report taking a long time. Any suggestions or warnings? Thanks.
ec2 ganglia fixing missing graphs
Hi, for the mail archive... I'm using the hadoop ec2 scripts and notice that ganglia actually does not show any graphs. I was able to fix this by adding dejavu-fonts to the packages that are installed via yum in create-hadoop-image-remote.sh. The line looks now like this: yum -y install rsync lynx screen ganglia-gmetad ganglia-gmond ganglia- web dejavu-fonts httpd php Since this effects the hadoop image, it might interesting to fix this and create a new public AMI. Stefan ~~~ Hadoop training and consulting http://www.scaleunlimited.com http://www.101tec.com
Re: Job Tracker/Name Node redundancy
Hey Ryan, Some specific JIRA tickets that will help narrow your search: JT: https://issues.apache.org/jira/browse/HADOOP-4586 NN: https://issues.apache.org/jira/browse/HADOOP-4539 Would love to hear your thoughts there! Regards, Jeff On Fri, Jan 9, 2009 at 12:36 PM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > Yes, there is a JIRA issue for a redundant JobTracker already. > The NN redundancy scenario is mentioned on the Wiki (look for > SecondaryNameNode). > > Otis > > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message > > From: Ryan LeCompte > > To: "core-user@hadoop.apache.org" > > Sent: Friday, January 9, 2009 3:09:13 PM > > Subject: Job Tracker/Name Node redundancy > > > > Are there any plans to build redundancy/failover support for the Job > > Tracker and Name Node components in Hadoop? Let's take the current > > scenario: > > > > 1) A data/cpu intensive job is submitted to a Hadoop cluster of 10 > machines. > > 2) Half-way through the job execution, the Job Tracker or Name Node > fails. > > 3) We bring up a new Job Tracker or Name Node manually. > > > > -- Will the individual task trackers / data nodes "reconnect" to the > > new masters? Or will the job have to be resubmitted? If we had > > failover support, we could setup essentially 3 Job Tracker masters and > > 3 NameNode masters so that if one dies the other would gracefully take > > over and start handling results from the children nodes. > > > > Thanks! > > > > Ryan > >
Re: Job Tracker/Name Node redundancy
Yes, there is a JIRA issue for a redundant JobTracker already. The NN redundancy scenario is mentioned on the Wiki (look for SecondaryNameNode). Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Ryan LeCompte > To: "core-user@hadoop.apache.org" > Sent: Friday, January 9, 2009 3:09:13 PM > Subject: Job Tracker/Name Node redundancy > > Are there any plans to build redundancy/failover support for the Job > Tracker and Name Node components in Hadoop? Let's take the current > scenario: > > 1) A data/cpu intensive job is submitted to a Hadoop cluster of 10 machines. > 2) Half-way through the job execution, the Job Tracker or Name Node fails. > 3) We bring up a new Job Tracker or Name Node manually. > > -- Will the individual task trackers / data nodes "reconnect" to the > new masters? Or will the job have to be resubmitted? If we had > failover support, we could setup essentially 3 Job Tracker masters and > 3 NameNode masters so that if one dies the other would gracefully take > over and start handling results from the children nodes. > > Thanks! > > Ryan
Job Tracker/Name Node redundancy
Are there any plans to build redundancy/failover support for the Job Tracker and Name Node components in Hadoop? Let's take the current scenario: 1) A data/cpu intensive job is submitted to a Hadoop cluster of 10 machines. 2) Half-way through the job execution, the Job Tracker or Name Node fails. 3) We bring up a new Job Tracker or Name Node manually. -- Will the individual task trackers / data nodes "reconnect" to the new masters? Or will the job have to be resubmitted? If we had failover support, we could setup essentially 3 Job Tracker masters and 3 NameNode masters so that if one dies the other would gracefully take over and start handling results from the children nodes. Thanks! Ryan
Re: 0.18.1 datanode psuedo deadlock problem
Hi Jason, 2 million blocks per data-node is not going to work. There were discussions about it previously, please check the mail archives. This means you have a lot of very small files, which HDFS is not designed to support. A general recommendation is to group small files into large ones, introducing some kind of record structure delimiting those small files, and control it in on the application level. Thanks, --Konstantin Jason Venner wrote: The problem we are having is that datanodes periodically stall for 10-15 minutes and drop off the active list and then come back. What is going on is that a long operation set is holding the lock on on FSDataset.volumes, and all of the other block service requests stall behind this lock. "DataNode: [/data/dfs-video-18/dfs/data]" daemon prio=10 tid=0x4d7ad400 nid=0x7c40 runnable [0x4c698000..0x4c6990d0] java.lang.Thread.State: RUNNABLE at java.lang.String.lastIndexOf(String.java:1628) at java.io.File.getName(File.java:399) at org.apache.hadoop.dfs.FSDataset$FSDir.getGenerationStampFromFile(FSDataset.java:148) at org.apache.hadoop.dfs.FSDataset$FSDir.getBlockInfo(FSDataset.java:181) at org.apache.hadoop.dfs.FSDataset$FSVolume.getBlockInfo(FSDataset.java:412) at org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getBlockInfo(FSDataset.java:511) - locked <0x551e8d48> (a org.apache.hadoop.dfs.FSDataset$FSVolumeSet) at org.apache.hadoop.dfs.FSDataset.getBlockReport(FSDataset.java:1053) at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:708) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2890) at java.lang.Thread.run(Thread.java:619) This is basically taking a stat on every hdfs block on the datanode, which in our case is ~ 2million, and can take 10+ minutes (we may be experiencing problems with our raid controller but have no visibility into it) at the OS level the file system seems fine and operations eventually finish. It appears that a couple of different data structures are being locked with the single object FSDataset$Volume. Then this happens: "org.apache.hadoop.dfs.datanode$dataxcei...@1bcee17" daemon prio=10 tid=0x4da8d000 nid=0x7ae4 waiting for monitor entry [0x459fe000..0x459ff0d0] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.dfs.FSDataset$FSVolumeSet.getNextVolume(FSDataset.java:473) - waiting to lock <0x551e8d48> (a org.apache.hadoop.dfs.FSDataset$FSVolumeSet) at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:934) - locked <0x54e550e0> (a org.apache.hadoop.dfs.FSDataset) at org.apache.hadoop.dfs.DataNode$BlockReceiver.(DataNode.java:2322) at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:1187) at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1045) at java.lang.Thread.run(Thread.java:619) which locks the FSDataset while waiting on the volume object and now all of the Datanode operations stall waiting on the FSDataset object. -- Our particular installation doesn't use multiple directories for hdfs, so a first simple hack for a local fix would be to modify getNextVolume to just return the single volume and not be synchronized A richer alternative would be to make the locking more fine grained on FSDataset$FSVolumeSet. Of course we are also trying to fix the file system performance and dfs block loading that results in the block report taking a long time. Any suggestions or warnings? Thanks.
Re: NameNode fatal crash - 0.18.1
Hi, Jonathan. The problem is that the local drive(s) you use for "dfs.name.dir" became unaccessible. So the name-node is not able to persist name-space modifications anymore, and therefore self terminated. The rest are the consequences. This is the core message > 2008-12-15 01:49:31,178 FATAL org.apache.hadoop.fs.FSNamesystem: Fatal Error > : All storage directories are inaccessible. Could you please check the drives. --Konstantin Jonathan Gray wrote: I have a 10+1 node cluster, each slave running DataNode/TaskTracker/HBase RegionServer. At the time of this crash, NameNode and SecondaryNameNode were both hosted on same master. We do a nightly backup and about 95% of the way through, HDFS crashed with... NameNode shows: 2008-12-15 01:49:31,178 ERROR org.apache.hadoop.fs.FSNamesystem: Unable to sync edit log. Fatal Error. 2008-12-15 01:49:31,178 FATAL org.apache.hadoop.fs.FSNamesystem: Fatal Error : All storage directories are inaccessible. 2008-12-15 01:49:31,179 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG: Every single DataNode shows: 2008-12-15 01:49:32,340 WARN org.apache.hadoop.dfs.DataNode: java.io.IOException: Call failed on local exception at org.apache.hadoop.ipc.Client.call(Client.java:718) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at org.apache.hadoop.dfs.$Proxy4.sendHeartbeat(Unknown Source) at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:655) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2888) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:499) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:441) This is virtually all of the information I have. At the same time as the backup, we have normal HBase traffic and our hourly batch MR jobs. So slave nodes were pretty heavily loaded, but don't see anything in DN logs besides this Call failed. There are no space issues or anything else, Ganglia shows high CPU load around this time which has been typical every night, but I don't see any issues in DN's or NN about expired leases/no heartbeats/etc. Is there a way to prevent this failure from happening in the first place? I guess just reduce total load across cluster? Second question is about how to recover once NameNode does fail... When trying to bring HDFS back up, we get hundreds of: 2008-12-15 07:54:13,265 ERROR org.apache.hadoop.dfs.LeaseManager: XXX not found in lease.paths And then 2008-12-15 07:54:13,267 ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization failed. Is there a way to recover from this? As of time of this crash, we had SecondaryNameNode on the same node. Moving it to another node with sufficient memory now, but would that even prevent this kind of FS botching? Also, my SecondaryNameNode is telling me it cannot connect when trying to do a checkpoint: 2008-12-15 09:59:48,017 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint: 2008-12-15 09:59:48,018 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) I changed my masters file to just contain the hostname of the secondarynamenode, this seems to have properly started the NameNode where I launched the ./bin/start-dfs.sh from and started SecondaryNameNode on correct node as well. But it seems to be unable to connect back to primary. I have hadoop-site.xml pointing to fs.default.name of primary, but otherwise there are not links back. Where would I specify to the secondary where primary is located? We're also upgrading to Hadoop 0.19.0 at this time. Thank you for any help. Jonathan Gray
Reducer Not creating part-00000 file
I have three machine setup to run a map/reduce program running the wordcount program in the hadoop-examples.jar. The process completes fine but when I try to find the part-0 file after the processes is done I cannot find , it seems that I did not create any part-0 file, I don't know why. But when I change the total number of machines from 3 to 1 , and re run the program it creates part- file. I don't know what is the problem. I did another test but this time with huge files , 6BG ( the previous one for 50 MB) and ran the program with 3 machine setup, it worked fine, and then I ran with one machine setup and this also worked fine, any idea on what might be the problem ? Regards, Madhu Jahagirdar Philips Medical Systems / Ultrasound Monitoring Bothell, Washington Tel: 425-487-8140 The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Hangs when reduce
The hadoop is 0.19-trunk checked out 2008-10-10 we use this trunck and after run several days, it will alway hangs several reducers when copying at 0.0%。 the log is always like this: 2009-01-09 11:19:38,600 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200901081900_0015_r_00_3 0.0% reduce > copy > 2009-01-09 11:19:41,331 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200901081900_0015/attempt_200901081900_0015_r_00_3/output/file.out in any of the configured local directories I also wonder why it find the reduce output in the local directories? I just know the mapresult is on local disk. -- Best wishes! My Friend~
Re: A reporter thread during the reduce stage for a long running line
On Jan 9, 2009, at 12:09 AM, Saptarshi Guha wrote: Hello, Sorry for the puzzling subject. I have a single long running /statement/ in my reduce method, so the the framework might assume my reduce is not responding and kill it. I solved the problem in the map method by subclassing MapRunner, and running a thread which calls reporter.progress() every minute or so. However the same thread does not run during the reduce (i checked this by setting a status string in the thread which did not appear (on the Jobtracker website) during the reduce stage but did appearing during the map stage). Hadoop v.0.20 appears to solve this by having separate run methods for both Map and Reduce, however I'm using v 0.19. I scanned the Streaming source and it only subclasses MapRunner, so I assume it to has the same limitation (probably wrong, if so can someone point me to the location?) Is there a way around this, /without/ starting a thread in the reduce function? Hadoop v 0.19 If you are _really_ sure you do not want the reducer to get killed regardless of whether it's making progress or not, set mapred.task.timeout to 0 in your JobConf. Please be aware that this will mean the framework cannot detect if your mapper/reducer tasks are hung... Arun
A reporter thread during the reduce stage for a long running line
Hello, Sorry for the puzzling subject. I have a single long running /statement/ in my reduce method, so the the framework might assume my reduce is not responding and kill it. I solved the problem in the map method by subclassing MapRunner, and running a thread which calls reporter.progress() every minute or so. However the same thread does not run during the reduce (i checked this by setting a status string in the thread which did not appear (on the Jobtracker website) during the reduce stage but did appearing during the map stage). Hadoop v.0.20 appears to solve this by having separate run methods for both Map and Reduce, however I'm using v 0.19. I scanned the Streaming source and it only subclasses MapRunner, so I assume it to has the same limitation (probably wrong, if so can someone point me to the location?) Is there a way around this, /without/ starting a thread in the reduce function? Hadoop v 0.19 Many thanks Saptarshi -- Saptarshi Guha - saptarshi.g...@gmail.com
