youlong chen created MAPREDUCE-7496:
---------------------------------------
Summary: FrameworkUploader throws NoSuchElementException despite
successful file upload
Key: MAPREDUCE-7496
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7496
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.4.1
Reporter: youlong chen
Description: The mapred FrameworkUploader tool throws a NoSuchElementException
when uploading a file to HDFS, even though the file is successfully uploaded
with the specified replication factor. Steps to reproduce: 1. Execute the
following command: \{code} $HADOOP_PREFIX/bin/mapred frameworkuploader \ -input
/test/test.jar \ -fs hdfs://namenode:9000 \ -target hdfs:///test.jar#test_test
\ -initialReplication 3 \ -finalReplication 10 \ -acceptableReplication 9 \
-timeout 60 \{code} Current behavior: 1. The file is successfully uploaded to
HDFS with the specified replication factor (10) 2. The tool throws a
NoSuchElementException: \{code} Exception in thread "main"
java.util.NoSuchElementException at
java.util.HashMap$HashIterator.nextNode(HashMap.java:1471) at
java.util.HashMap$ValueIterator.next(HashMap.java:1498) at
java.util.Collections.min(Collections.java:598) at
org.apache.hadoop.mapred.uploader.FrameworkUploader.getSmallestReplicatedBlockCount(FrameworkUploader.java:243)
at
org.apache.hadoop.mapred.uploader.FrameworkUploader.endUpload(FrameworkUploader.java:260)
at
org.apache.hadoop.mapred.uploader.FrameworkUploader.buildPackage(FrameworkUploader.java:293)
at
org.apache.hadoop.mapred.uploader.FrameworkUploader.run(FrameworkUploader.java:117)
at
org.apache.hadoop.mapred.uploader.FrameworkUploader.main(FrameworkUploader.java:560)
\{code} Expected behavior: The tool should complete successfully without
throwing any exception since the file upload and replication operations were
successful. Analysis: The exception occurs in getSmallestReplicatedBlockCount()
method when trying to find the minimum replication count. This suggests that
the block replication tracking map might be empty when it shouldn't be, causing
the Collections.min() operation to fail. Impact: While the file upload
operation succeeds, the exception prevents the tool from completing normally
and could cause issues in automated workflows that depend on the tool's exit
status. Suggested Fix: Add proper null checking and empty collection handling
in getSmallestReplicatedBlockCount() method to prevent the
NoSuchElementException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
