[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12926074#action_12926074
]
Hudson commented on MAPREDUCE-1888:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See
[https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch, 1888.v3.patch,
1888.v4.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885104#action_12885104
]
Amareshwari Sriramadasu commented on MAPREDUCE-1888:
Patch looks good.
A couple of mumak tests failed with NoClassDefFoundError, they passed on my
local machine. TestSimulatorDeterministicReplay failed because of
MAPREDUCE-1834.
Will check this in.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch, 1888.v3.patch,
1888.v4.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884665#action_12884665
]
Hadoop QA commented on MAPREDUCE-1888:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448506/1888.v4.patch
against trunk revision 959867.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 35 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/591/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/591/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/591/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/591/console
This message is automatically generated.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch, 1888.v3.patch,
1888.v4.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883826#action_12883826
]
Amareshwari Sriramadasu commented on MAPREDUCE-1888:
While working on MAPREDUCE-1122, i realized that one more corner case got
missed here. For reducer=NONE, if the mapper is a command, then job output key
and value types should be set from IO identifier.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch, 1888.v3.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883833#action_12883833
]
Amareshwari Sriramadasu commented on MAPREDUCE-1888:
Ravi, Can you add testcases for all combination of the following?
1) Mapper is command or is a java mapper.
2) Reducer is command or is java mapper or is None.
You can rename the testcase TestStreamingOutputKeyValueTypes instead of
TestStreamingJavaTasks.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch, 1888.v3.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883506#action_12883506
]
Ravi Gummadi commented on MAPREDUCE-1888:
-
TrApp.java when used as mapper should not expect mapreduce_job_output_key_class
as Text and mapreduce_job_output_value_class as Text.
Fixing TrApp.java so that it expects mapreduce_map_output_key_class and
mapreduce_map_output_key_class.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883787#action_12883787
]
Amareshwari Sriramadasu commented on MAPREDUCE-1888:
+1 Latest patch looks fine.
Test failure is because of MAPREDUCE-1834.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch, 1888.v2.patch, 1888.v3.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883105#action_12883105
]
Amareshwari Sriramadasu commented on MAPREDUCE-1888:
Passing NONE for reducer command, still has problem with key, value types.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch, 1888.v1.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883032#action_12883032
]
Amareshwari Sriramadasu commented on MAPREDUCE-1888:
Some comments on the patch:
* Please don't change genArgs() in TestStreaming. You can override it in
TestStreamingJavaTasks as done in other tests.
* Is the Change in TestFileOutputFormat required?
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
