Reduce hang on small set of SequenceFiles (reproducible)
--------------------------------------------------------
Key: HADOOP-895
URL: https://issues.apache.org/jira/browse/HADOOP-895
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.10.1
Environment: This happens both on a mac and on a Linux x86_64 machine.
Reporter: Doug Judd
I'm running into a problem where a Map/Reduce job hangs in the reduce phase.
It happens on a simple program to dump the keys of a small set of
SequenceFiles. It can be reproduced as follows:
1. Pull a vanilla hadoop-0.10.1 release and untar it
2. Apply the attached patch
3. Modify conf/hadoop-env.sh to have the correct JAVA_HOME
4. Run the following script:
#!/bin/sh
ant zvents
./bin/hadoop namenode -format
mkdir fs-hadoop/dfs/tmp
mkdir fs-hadoop/dfs/data
mkdir -p fs-hadoop/mapred/local
./bin/start-all.sh
sleep 5
./bin/hadoop dfs -mkdir /heritrix
./bin/hadoop dfs -mkdir /heritrix/crawls
./bin/hadoop dfs -mkdir /heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal
data/IAH-20070116061532-00000-doug-judds-computer.local
/heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal
data/IAH-20070116061539-00001-doug-judds-computer.local
/heritrix/crawls/test-4-20070116061502956
./bin/hadoop dfs -copyFromLocal
data/IAH-20070116061539-00002-doug-judds-computer.local
/heritrix/crawls/test-4-20070116061502956
./bin/hadoop com.zvents.hadoop.DumpUrls
/heritrix/crawls/test-4-20070116061502956 /foo
- Doug
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
