[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117043#comment-13117043 ] Ravi Gummadi commented on MAPREDUCE-3118: - System tests patches are not part of this for now. They will be added by Vinay (may be later). Backporting patches (features and bugfixes) are from the following JIRAs: MAPREDUCE-2596: [Gridmix] Summarize Gridmix runs. MAPREDUCE-2107: [Gridmix] Total heap usage emulation in Gridmix. MAPREDUCE-2106: [Gridmix] Cumulative CPU usage emulation in Gridmix. MAPREDUCE-2543: [Gridmix] High-Ram feature emulation testcase. MAPREDUCE-2408: [Gridmix] Compression emulation in Gridmix. MAPREDUCE-2137: Provide mapping between jobs of trace file and the corresponding simulated cluster's jobs in Gridmix. MAPREDUCE-2407: Make GridMix emulate usage of distributed cache files in simulated jobs. MAPREDUCE-2416: Remove the restriction of specifying group names in users-list file for Gridmix in RoundRobinUserResolver mode. MAPREDUCE-1931: Gridmix documentation. MAPREDUCE-2417: Fix Gridmix in RoundRobinUserResolver mode to map testing/proxy users to unique users in a trace. MAPREDUCE-2095: Fix Gridmix to run from compressed traces. MAPREDUCE-1989: Fix error message in gridmix when user resolver is set and no user list is given. MAPREDUCE-1979: Fix 'Output directory already exists' error in gridmix when gridmix.output.directory is not defined. MAPREDUCE-1975: Fix unnecessary InterruptedException log in gridmix. \\ \\ \\ MAPREDUCE-2078: Fix TraceBuilder to generate traces when a globbed job history path is given. MAPREDUCE-2153: Rumen should bring in job configuration properties into the trace file. MAPREDUCE-2104: [Rumen] Add Cpu, Memory and Heap usages to TraceBuilder's output. MAPREDUCE-1978: Rumen TraceBuilder should provide recursive input folder scanning. MAPREDUCE-1918: Add documentation to Rumen. MAPREDUCE-2000: Rumen is not able to extract counters for Job history logs from Hadoop 0.20. MAPREDUCE-1982: Fix Rumen's TraceBuilder to extract job name from either of configuration properties mapreduce.job.name and mapred.job.name. MAPREDUCE-1925: Fix failing TestRumenJobTraces. \\ \\ This backporting also needs minor changes to mapreduce task progress reporting because some of these gridmix features rely on the task progress values. > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121416#comment-13121416 ] Matt Foley commented on MAPREDUCE-3118: --- Hi Ravi, it will be great to see these improvements in 0.20.206. Thanks. > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > Attachments: gridmix_rumen_backports.v2.4.patch, > gridmix_rumen_backports.v2.5.patch > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121694#comment-13121694 ] Amar Kamat commented on MAPREDUCE-3118: --- @Matt I have taken up the review of the latest patch. @All Vinay has tested this patch on a test cluster and seems like all the latest Gridmix features work fine. Vinay also backported the system tests he wrote for trunk to make sure that all the features are tested thoroughly. > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > Attachments: gridmix_rumen_backports.v2.4.patch, > gridmix_rumen_backports.v2.5.patch > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122577#comment-13122577 ] Amar Kamat commented on MAPREDUCE-3118: --- I see some MRJobConfig fields in DistributedCacheEmulator's javadoc. Can this be changed to their new locations maybe in JobContext or DistributedCache? Rest of the patch looks good to me. @Ravi: Can you upload test-patch and ant-test status along with the final patch? > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > Attachments: gridmix_rumen_backports.v2.4.patch, > gridmix_rumen_backports.v2.5.patch > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125580#comment-13125580 ] Amar Kamat commented on MAPREDUCE-3118: --- Folks, This patch adds a _getProgress()_ API to public classes and interfaces like: 1. Progress.java 2. TaskInputOutputContext.java 3. Reporter.java This API provides the task's current progress and is immensely useful of Gridmix. We tend to believe that this might get classified as a backward incompatible change. We want to be sure about the incompatibility side of the story and make a call accordingly. Kindly let us know your thoughts/comments regarding the same. > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > Attachments: gridmix_rumen_backports.v2.4.patch, > gridmix_rumen_backports.v2.5.patch, gridmix_rumen_backports.v2.6.patch > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129587#comment-13129587 ] Ravi Gummadi commented on MAPREDUCE-3118: - Please come back if there are any issues. We are planning to commit this soon. > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > Attachments: gridmix_rumen_backports.v2.4.patch, > gridmix_rumen_backports.v2.5.patch, gridmix_rumen_backports.v2.6.patch > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3118) Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129672#comment-13129672 ] Amar Kamat commented on MAPREDUCE-3118: --- Folks, We ran the wordcount and Gridmix3 from Hadoop 0.20.205 on a Hadoop cluster running a MR-3118 patched version of 0.20.205. All the jobs ran fine. We didn't see any compatibility issues. I will commit this patch. > Backport Gridmix and Rumen features from trunk to Hadoop 0.20 security branch > - > > Key: MAPREDUCE-3118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3118 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/gridmix, tools/rumen >Affects Versions: 0.20.206.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.20.206.0 > > Attachments: gridmix_rumen_backports.v2.4.patch, > gridmix_rumen_backports.v2.5.patch, gridmix_rumen_backports.v2.6.patch > > > Backporting all the features and bugfixes that went into gridmix and rumen of > trunk to hadoop 0.20 security branch. This will enable using all these > gridmix features and run gridmix/rumen on the history logs of 0.20 security > branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
