[
https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635668#comment-13635668
]
Chris Nauroth commented on MAPREDUCE-5161:
------------------------------------------
{quote}
Its not clear whether it reverts the independent fix in branch-1-win that is
mentioned in the description?
{quote}
Yes, this patch reverts that fix so that branch-1 and branch-1-win are
identical for this logic. For reference, I've included a diff below showing
the earlier fix that was made straight to branch-1-win, so you can compare.
The branch-1 version is preferable and includes more tests.
{code}
diff --git src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
index c55df11..c439bad 100644
--- src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
+++ src/mapred/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java
@@ -194,7 +194,7 @@ public abstract class CombineFileInputFormat<K, V>
continue;
}
FileSystem fs = paths[i].getFileSystem(job);
- Path p = new Path(paths[i].toUri().getPath());
+ Path p = new Path(paths[i].toString());
if (onepool.accept(p)) {
myPaths.add(paths[i]); // add it to my output set
paths[i] = null; // already processed
diff --git
src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
index c9fa549..c7929e4 100644
--- src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
+++ src/mapred/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
@@ -211,7 +211,7 @@ public abstract class CombineFileInputFormat<K, V>
// times, one time each for each pool in the next loop.
List<Path> newpaths = new LinkedList<Path>();
for (int i = 0; i < paths.length; i++) {
- Path p = new Path(paths[i].toUri().getPath());
+ Path p = new Path(paths[i].toString());
newpaths.add(p);
}
paths = null;
diff --git
src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
index 8f7c4be..f013bb8 100644
--- src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
+++ src/test/org/apache/hadoop/mapred/lib/TestCombineFileInputFormat.java
@@ -462,7 +462,8 @@ public class TestCombineFileInputFormat extends TestCase{
// returns true if the specified path matches the prefix stored
// in this TestFilter.
public boolean accept(Path path) {
- if (path.toString().indexOf(p.toString()) == 0) {
+ Path uriPath = new Path(path.toUri().getPath());
+ if (uriPath.toString().indexOf(p.toString()) == 0) {
return true;
}
return false;
diff --git
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
index c80c70d..16345bd 100644
---
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
+++
src/test/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
@@ -1122,7 +1122,8 @@ public class TestCombineFileInputFormat extends TestCase {
// returns true if the specified path matches the prefix stored
// in this TestFilter.
public boolean accept(Path path) {
- if (path.toString().indexOf(p.toString()) == 0) {
+ Path uriPath = new Path(path.toUri().getPath());
+ if (uriPath.toString().indexOf(p.toString()) == 0) {
return true;
}
return false;
{code}
> CombineFileInputFormat fix for paths not on default FS merge from branch-1 to
> branch-1-win
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5161
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1
> Affects Versions: 1-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: MAPREDUCE-5161-branch-1-win.1.patch
>
>
> MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
> paths that are not on the default file system. This same bug was fixed
> independently on branch-1-win. The code was slightly different, but
> equivalent to the branch-1 fix. This jira will apply the branch-1 fix to
> branch-1-win to keep the 2 code lines in agreement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
