[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12926057#action_12926057 ] Hudson commented on MAPREDUCE-1893: --- Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/]) Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: SliveMultiR.patch, SliveMultiR.patch, SliveMultiR.patch Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884436#action_12884436 ] Ravi Phulari commented on MAPREDUCE-1893: - I have verified , it works. Sorry for the delay Konstantin. Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: SliveMultiR.patch, SliveMultiR.patch, SliveMultiR.patch Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[
https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12884130#action_12884130
]
Konstantin Shvachko commented on MAPREDUCE-1893:
Ravi, {{SliveTest.class.getSimpleName()}} returns string {{SliveTest}}.
Should work fine as is. Could you pls verify again.
Multiple reducers for Slive
---
Key: MAPREDUCE-1893
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: benchmarks, test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.22.0
Attachments: SliveMultiR.patch, SliveMultiR.patch, SliveMultiR.patch
Slive currently uses single reducer. It could use multiple ones.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883724#action_12883724 ] Ravi Phulari commented on MAPREDUCE-1893: - Thanks for adding Slive to Driver Konstantin. Patch looks good to me. Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: SliveMultiR.patch, SliveMultiR.patch, SliveMultiR.patch Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[
https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12883749#action_12883749
]
Ravi Phulari commented on MAPREDUCE-1893:
-
Sorry, I missed one thing earlier .
One correction is needed in MapredTestDriver.java
We need to make SliveTest visible through TestDriver which can be done by
adding SliveTest as first parameter to pgd.addClass()
{noformat}
pgd.addClass(SliveTest.class.getSimpleName(), SliveTest.class,
HDFS Stress Test and Live Data Verification.);
{noformat}
Should be like -
{noformat}
pgd.addClass(SliveTest,SliveTest.class.getSimpleName(), SliveTest.class,
HDFS Stress Test and Live Data Verification.);
{noformat}
Multiple reducers for Slive
---
Key: MAPREDUCE-1893
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: benchmarks, test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.22.0
Attachments: SliveMultiR.patch, SliveMultiR.patch, SliveMultiR.patch
Slive currently uses single reducer. It could use multiple ones.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882469#action_12882469 ] Hadoop QA commented on MAPREDUCE-1893: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448005/SliveMultiR.patch against trunk revision 957772. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/console This message is automatically generated. Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: SliveMultiR.patch Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882398#action_12882398 ] Konstantin Shvachko commented on MAPREDUCE-1893: Slive maps output different stats of the operations performed by the test. The stats are currently aggregated by a single reducer, which may run for a long time if the amount of generated data is large. I propose to add a parameter to Slive args, which specifies the number of reducers R. Then SlivePartitioner for each output stat calculates a hash value of the operation type, modular R. This defines the reducer number. Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
