[
https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-4885:
-------------------------------------
Attachment: MAPREDUCE-4885.1.patch
With this patch, all of the streaming tests pass consistently on Windows. Note
that to see the tests pass, you'll also need the patch for MAPREDUCE-5006,
which hasn't been committed yet.
The problems were:
# The now-infamous problem of attempting to use paths rooted on test.build.data
with HDFS, which rejects paths containing ':', such as the Windows drive spec.
The patch implements our standard work-around to allow overriding the test path
to /tmp/<test name>.
# There was an assumption of Unix-style commands available for use as streaming
mapper and reducer functions. To work around this, I introduced some cmd
scripts that roughly approximate Unix "cat" and "xargs cat".
# There was one actual bug in {{StreamJob}}. It was attempting to pass a
string file path into the {{URI}} constructor. On Windows, this would contain
drive spec, and {{URI}} would consider it invalid and throw an error. The only
reason we needed the {{URI}} was to pass it in to the constructor of {{Path}}.
Fortunately, we already have the logic in the {{Path}} constructor now to
handle this case correctly cross-platform, so the simple fix is just to call
the {{Path}} constructor with the string file path directly.
# I've increased a few test timeouts. The old timeout values were borderline
in my environment, sometimes causing the tests to fail sporadically on
timeouts. This was not a Windows-specific problem.
I've tested this patch on Mac and Windows.
> streaming tests have multiple failures on Windows
> -------------------------------------------------
>
> Key: MAPREDUCE-4885
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
> Affects Versions: 3.0.0, trunk-win
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: MAPREDUCE-4885.1.patch
>
>
> There are multiple test failures due to "Queue configuration missing child
> queue names for root".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
