[jira] [Commented] (MAPREDUCE-4989) JSONify DataTables input data for Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577483#comment-13577483 ] Hudson commented on MAPREDUCE-4989: --- Integrated in Hadoop-Yarn-trunk #126 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/126/]) MAPREDUCE-4989. JSONify DataTables input data for Attempts page. Contributed by Ravi Prakash (Revision 1445448) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445448 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksPage.java > JSONify DataTables input data for Attempts page > --- > > Key: MAPREDUCE-4989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4989 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mr-am >Affects Versions: 0.23.6 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4989.branch-0.23.patch, MAPREDUCE-4989.patch, > MAPREDUCE-4989.patch > > > Use deferred rendering for the attempts page as was done in MAPREDUCE-4720. > I'm sorry I didn't realize earlier that this table could be huge too. Thanks > to [~jlowe] for pointing it out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577482#comment-13577482 ] Hudson commented on MAPREDUCE-4994: --- Integrated in Hadoop-Yarn-trunk #126 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/126/]) MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu) (Revision 1445330) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445330 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4992) AM hangs in RecoveryService when recovering tasks with speculative attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577481#comment-13577481 ] Hudson commented on MAPREDUCE-4992: --- Integrated in Hadoop-Yarn-trunk #126 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/126/]) MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with speculative attempts. Contributed by Robert Parker (Revision 1445456) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445456 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java > AM hangs in RecoveryService when recovering tasks with speculative attempts > --- > > Key: MAPREDUCE-4992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4992 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: trunk, 2.0.2-alpha, 0.23.6 >Reporter: Robert Parker >Assignee: Robert Parker >Priority: Critical > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4992v1.patch, MAPREDUCE-4992v2.patch > > > A job hung in the Recovery Service on an AM restart. There were four map > tasks events that were not processed and that prevented the complete task > count from reaching zero which exits the recovery service. All four tasks > were speculative -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4989) JSONify DataTables input data for Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577529#comment-13577529 ] Hudson commented on MAPREDUCE-4989: --- Integrated in Hadoop-Hdfs-0.23-Build #524 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/524/]) MAPREDUCE-4989. JSONify DataTables input data for Attempts page. Contributed by Ravi Prakash (Revision 1445451) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445451 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksPage.java > JSONify DataTables input data for Attempts page > --- > > Key: MAPREDUCE-4989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4989 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mr-am >Affects Versions: 0.23.6 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4989.branch-0.23.patch, MAPREDUCE-4989.patch, > MAPREDUCE-4989.patch > > > Use deferred rendering for the attempts page as was done in MAPREDUCE-4720. > I'm sorry I didn't realize earlier that this table could be huge too. Thanks > to [~jlowe] for pointing it out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4992) AM hangs in RecoveryService when recovering tasks with speculative attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577528#comment-13577528 ] Hudson commented on MAPREDUCE-4992: --- Integrated in Hadoop-Hdfs-0.23-Build #524 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/524/]) svn merge -c 1445456 FIXES: MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with speculative attempts. Contributed by Robert Parker (Revision 1445458) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445458 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java > AM hangs in RecoveryService when recovering tasks with speculative attempts > --- > > Key: MAPREDUCE-4992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4992 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: trunk, 2.0.2-alpha, 0.23.6 >Reporter: Robert Parker >Assignee: Robert Parker >Priority: Critical > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4992v1.patch, MAPREDUCE-4992v2.patch > > > A job hung in the Recovery Service on an AM restart. There were four map > tasks events that were not processed and that prevented the complete task > count from reaching zero which exits the recovery service. All four tasks > were speculative -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5004) Somebody working on Genetic Algorithm library on Map Reduce
Abhishek Bajpai created MAPREDUCE-5004: -- Summary: Somebody working on Genetic Algorithm library on Map Reduce Key: MAPREDUCE-5004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5004 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Abhishek Bajpai -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4992) AM hangs in RecoveryService when recovering tasks with speculative attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577534#comment-13577534 ] Hudson commented on MAPREDUCE-4992: --- Integrated in Hadoop-Hdfs-trunk #1315 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1315/]) MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with speculative attempts. Contributed by Robert Parker (Revision 1445456) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445456 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java > AM hangs in RecoveryService when recovering tasks with speculative attempts > --- > > Key: MAPREDUCE-4992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4992 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: trunk, 2.0.2-alpha, 0.23.6 >Reporter: Robert Parker >Assignee: Robert Parker >Priority: Critical > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4992v1.patch, MAPREDUCE-4992v2.patch > > > A job hung in the Recovery Service on an AM restart. There were four map > tasks events that were not processed and that prevented the complete task > count from reaching zero which exits the recovery service. All four tasks > were speculative -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577535#comment-13577535 ] Hudson commented on MAPREDUCE-4994: --- Integrated in Hadoop-Hdfs-trunk #1315 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1315/]) MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu) (Revision 1445330) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445330 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4989) JSONify DataTables input data for Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577536#comment-13577536 ] Hudson commented on MAPREDUCE-4989: --- Integrated in Hadoop-Hdfs-trunk #1315 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1315/]) MAPREDUCE-4989. JSONify DataTables input data for Attempts page. Contributed by Ravi Prakash (Revision 1445448) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445448 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksPage.java > JSONify DataTables input data for Attempts page > --- > > Key: MAPREDUCE-4989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4989 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mr-am >Affects Versions: 0.23.6 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4989.branch-0.23.patch, MAPREDUCE-4989.patch, > MAPREDUCE-4989.patch > > > Use deferred rendering for the attempts page as was done in MAPREDUCE-4720. > I'm sorry I didn't realize earlier that this table could be huge too. Thanks > to [~jlowe] for pointing it out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577552#comment-13577552 ] Gelesh commented on MAPREDUCE-4974: --- The Existing test case is enough , because its just a code optimization, Could any body, have a look and comment please .. ? > Optimising the LineRecordReader initialize() method > --- > > Key: MAPREDUCE-4974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, performance >Affects Versions: 2.0.2-alpha, 0.23.5 > Environment: Hadoop Linux >Reporter: Arun A K >Assignee: Gelesh > Labels: patch, performance > Attachments: MAPREDUCE-4974.1.patch, MAPREDUCE-4974.2.patch, > MAPREDUCE-4974.3.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > I found there is a a scope of optimizing the code, over initialize() if we > have compressionCodecs & codec instantiated only if its a compressed input. > Mean while Gelesh George Omathil, added if we could avoid the null check of > key & value. This would time save, since for every next key value generation, > null check is done. The intention being to instantiate only once and avoid > NPE as well. Hope both could be met if initialize key & value over > initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4992) AM hangs in RecoveryService when recovering tasks with speculative attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577576#comment-13577576 ] Hudson commented on MAPREDUCE-4992: --- Integrated in Hadoop-Mapreduce-trunk #1343 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1343/]) MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with speculative attempts. Contributed by Robert Parker (Revision 1445456) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445456 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java > AM hangs in RecoveryService when recovering tasks with speculative attempts > --- > > Key: MAPREDUCE-4992 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4992 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: trunk, 2.0.2-alpha, 0.23.6 >Reporter: Robert Parker >Assignee: Robert Parker >Priority: Critical > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4992v1.patch, MAPREDUCE-4992v2.patch > > > A job hung in the Recovery Service on an AM restart. There were four map > tasks events that were not processed and that prevented the complete task > count from reaching zero which exits the recovery service. All four tasks > were speculative -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577577#comment-13577577 ] Hudson commented on MAPREDUCE-4994: --- Integrated in Hadoop-Mapreduce-trunk #1343 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1343/]) MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu) (Revision 1445330) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445330 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4989) JSONify DataTables input data for Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577578#comment-13577578 ] Hudson commented on MAPREDUCE-4989: --- Integrated in Hadoop-Mapreduce-trunk #1343 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1343/]) MAPREDUCE-4989. JSONify DataTables input data for Attempts page. Contributed by Ravi Prakash (Revision 1445448) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445448 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksPage.java > JSONify DataTables input data for Attempts page > --- > > Key: MAPREDUCE-4989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4989 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mr-am >Affects Versions: 0.23.6 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-4989.branch-0.23.patch, MAPREDUCE-4989.patch, > MAPREDUCE-4989.patch > > > Use deferred rendering for the attempts page as was done in MAPREDUCE-4720. > I'm sorry I didn't realize earlier that this table could be huge too. Thanks > to [~jlowe] for pointing it out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5004) Somebody working on Genetic Algorithm library on Map Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5004. --- Resolution: Invalid Hi Abhishek, I'm not quite sure what you were trying to get at with this JIRA, but I recommend emailing either u...@hadoop.apache.org or common-...@hadoop.apache.org with your question. > Somebody working on Genetic Algorithm library on Map Reduce > --- > > Key: MAPREDUCE-5004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5004 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Abhishek Bajpai > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577767#comment-13577767 ] Jason Lowe commented on MAPREDUCE-4994: --- This change broke a number of unit tests in hadoop-mapreduce-client-jobclient, namely: * TestChainMapReduce * TestLineRecordReader * TestClientProtocolProviderImpls > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5000) TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating
[ https://issues.apache.org/jira/browse/MAPREDUCE-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577845#comment-13577845 ] Siddharth Seth commented on MAPREDUCE-5000: --- Thanks Jason. Committing... > TaskImpl.getCounters() can return the counters for the wrong task attempt > when task is speculating > -- > > Key: MAPREDUCE-5000 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5000 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 0.23.6 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5000-branch-0.23.patch, MAPREDUCE-5000.patch > > > When a task is speculating and one attempt completes then sometimes the > counters for the wrong attempt are aggregated into the total counters for the > job. The scenario looks like this: > # Two task attempts are racing, _0 and _1 > # _1 finishes first, causing the task to issue a TA_KILL to attempt _0 > # _0 receives TA_KILL, sets progress to 1.0f and waits for container cleanup > # if TaskImpl.getCounters() is called now, TaskImpl.selectBestAttempt() can > return _0 since it is not quite yet in the KILLED state yet progress is maxed > out and no other attempt has more progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577849#comment-13577849 ] Sandy Ryza commented on MAPREDUCE-4994: --- Sorry about that. It looks like TestChainMapReduce and TestLineRecordReader were broken by no longer setting all local jobs to have one mapper. TestClientProtocolProviderImpls was using the the mapreduce.jobtracker.address config that the patch took out. Uploading an addendum patch that fixes these. > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4994: -- Attachment: MAPREDUCE-4994-addendum.patch > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum.patch, > MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-4994: --- > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum.patch, > MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5000) TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating
[ https://issues.apache.org/jira/browse/MAPREDUCE-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577850#comment-13577850 ] Hudson commented on MAPREDUCE-5000: --- Integrated in Hadoop-trunk-Commit #3354 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3354/]) MAPREDUCE-5000. Fixes getCounters when speculating by fixing the selection of the best attempt for a task. Contributed by Jason Lowe. (Revision 1445871) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1445871 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java > TaskImpl.getCounters() can return the counters for the wrong task attempt > when task is speculating > -- > > Key: MAPREDUCE-5000 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5000 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 0.23.6 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-5000-branch-0.23.patch, MAPREDUCE-5000.patch > > > When a task is speculating and one attempt completes then sometimes the > counters for the wrong attempt are aggregated into the total counters for the > job. The scenario looks like this: > # Two task attempts are racing, _0 and _1 > # _1 finishes first, causing the task to issue a TA_KILL to attempt _0 > # _0 receives TA_KILL, sets progress to 1.0f and waits for container cleanup > # if TaskImpl.getCounters() is called now, TaskImpl.selectBestAttempt() can > return _0 since it is not quite yet in the KILLED state yet progress is maxed > out and no other attempt has more progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4994: -- Status: Patch Available (was: Reopened) > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum.patch, > MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5000) TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating
[ https://issues.apache.org/jira/browse/MAPREDUCE-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5000: -- Resolution: Fixed Fix Version/s: 2.0.4-beta 0.23.7 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > TaskImpl.getCounters() can return the counters for the wrong task attempt > when task is speculating > -- > > Key: MAPREDUCE-5000 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5000 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 0.23.6 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.7, 2.0.4-beta > > Attachments: MAPREDUCE-5000-branch-0.23.patch, MAPREDUCE-5000.patch > > > When a task is speculating and one attempt completes then sometimes the > counters for the wrong attempt are aggregated into the total counters for the > job. The scenario looks like this: > # Two task attempts are racing, _0 and _1 > # _1 finishes first, causing the task to issue a TA_KILL to attempt _0 > # _0 receives TA_KILL, sets progress to 1.0f and waits for container cleanup > # if TaskImpl.getCounters() is called now, TaskImpl.selectBestAttempt() can > return _0 since it is not quite yet in the KILLED state yet progress is maxed > out and no other attempt has more progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577853#comment-13577853 ] Alejandro Abdelnur commented on MAPREDUCE-4994: --- Mmhh, wouldn't be better to get 1 MAP as default if not set? > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum.patch, > MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[
https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577908#comment-13577908
]
Hadoop QA commented on MAPREDUCE-4994:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569249/MAPREDUCE-4994-addendum.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3331//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3331//console
This message is automatically generated.
> -jt generic command line option does not work
> -
>
> Key: MAPREDUCE-4994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.0.2-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Fix For: 2.0.4-beta
>
> Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum.patch,
> MAPREDUCE-4994.patch
>
>
> hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/
> should run a job using the local file system and the local job runner.
> Instead it tries to connect to a jobtracker.
> hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/
> does not use the given host/port
> This appears to be because Cluster#initialize, which loads the
> ClientProtocol, contains no special handling for mapred.job.tracker.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4994: -- Attachment: MAPREDUCE-4994-addendum-1.patch Attached updated addendum that uses 1 as default if not set. > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch, > MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577972#comment-13577972 ] Alejandro Abdelnur commented on MAPREDUCE-4994: --- +1 on the latest addendum, pending jenkins. > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch, > MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[
https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577993#comment-13577993
]
Hadoop QA commented on MAPREDUCE-4994:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569273/MAPREDUCE-4994-addendum-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3332//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3332//console
This message is automatically generated.
> -jt generic command line option does not work
> -
>
> Key: MAPREDUCE-4994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.0.2-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Fix For: 2.0.4-beta
>
> Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch,
> MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch
>
>
> hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/
> should run a job using the local file system and the local job runner.
> Instead it tries to connect to a jobtracker.
> hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/
> does not use the given host/port
> This appears to be because Cluster#initialize, which loads the
> ClientProtocol, contains no special handling for mapred.job.tracker.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5005) TestProcfsProcessTree#testProcessTree() doesn't wait long enough for the process to die
Karthik Kambatla created MAPREDUCE-5005:
---
Summary: TestProcfsProcessTree#testProcessTree() doesn't wait long
enough for the process to die
Key: MAPREDUCE-5005
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5005
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
TestProcfsProcessTree#testProcessTree fails occasionally with the following
stack trace
{noformat}
Stack Trace:
junit.framework.AssertionFailedError: expected: but was:
at
org.apache.hadoop.util.TestProcfsBasedProcessTree.testProcessTree(TestProcfsBasedProcessTree.java)
{noformat}
kill -9 is executed asynchronously, the signal is delivered when the process
comes out of the kernel (sys call). Checking if the process died immediately
after can fail at times.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5005) TestProcfsProcessTree#testProcessTree() doesn't wait long enough for the process to die
[
https://issues.apache.org/jira/browse/MAPREDUCE-5005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578000#comment-13578000
]
Karthik Kambatla commented on MAPREDUCE-5005:
-
Possible approaches:
# Throw in a sleep before checking, and hopefully the process would have
received the signal. Simple to do, but need not always work.
# Check if the process died OR (the signal has been delivered AND the process
is not stuck in kernel mode)
> TestProcfsProcessTree#testProcessTree() doesn't wait long enough for the
> process to die
> ---
>
> Key: MAPREDUCE-5005
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5005
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.2-alpha
>Reporter: Karthik Kambatla
>
> TestProcfsProcessTree#testProcessTree fails occasionally with the following
> stack trace
> {noformat}
> Stack Trace:
> junit.framework.AssertionFailedError: expected: but was:
> at
> org.apache.hadoop.util.TestProcfsBasedProcessTree.testProcessTree(TestProcfsBasedProcessTree.java)
> {noformat}
> kill -9 is executed asynchronously, the signal is delivered when the process
> comes out of the kernel (sys call). Checking if the process died immediately
> after can fail at times.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578062#comment-13578062 ] Mayank Bansal commented on MAPREDUCE-4305: -- Hi, Thanks Arun for your comments. I had a offline discussion with Arun. The reason behind the timeouts which I have added was due to save jobs from starving if we have priority however we need more work in that case. So I am refactoring my patch in to two patches. This patch is mostly Yarn-80 for Hadoop-1 with test framework for testing those scenarios like node local , rack local etc. I will file the JIRA with priority and timeouts and update the patch there. Thanks, Mayank > Implement delay scheduling in capacity scheduler for improving data locality > > > Key: MAPREDUCE-4305 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, > PATCH-MAPREDUCE-4305-MR1-1.patch, PATCH-MAPREDUCE-4305-MR1.patch > > > Capacity Scheduler data local tasks are about 40%-50% which is not good. > While my test with 70 node cluster i consistently get data locality around > 40-50% on a free cluster. > I think we need to implement something like delay scheduling in the capacity > scheduler for improving the data locality. > http://radlab.cs.berkeley.edu/publication/308 > After implementing the delay scheduling on Hadoop 22 I am getting 100 % data > locality in free cluster and around 90% data locality in busy cluster. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4305: - Attachment: PATCH-MAPREDUCE-4305-MR1-2.patch > Implement delay scheduling in capacity scheduler for improving data locality > > > Key: MAPREDUCE-4305 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, > PATCH-MAPREDUCE-4305-MR1-1.patch, PATCH-MAPREDUCE-4305-MR1-2.patch, > PATCH-MAPREDUCE-4305-MR1.patch > > > Capacity Scheduler data local tasks are about 40%-50% which is not good. > While my test with 70 node cluster i consistently get data locality around > 40-50% on a free cluster. > I think we need to implement something like delay scheduling in the capacity > scheduler for improving the data locality. > http://radlab.cs.berkeley.edu/publication/308 > After implementing the delay scheduling on Hadoop 22 I am getting 100 % data > locality in free cluster and around 90% data locality in busy cluster. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4988) ClusterWithCapacityScheduler and related testcases needs to be ported to JUnit4.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Sanjar updated MAPREDUCE-4988: --- Priority: Major (was: Trivial) > ClusterWithCapacityScheduler and related testcases needs to be ported to > JUnit4. > > > Key: MAPREDUCE-4988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.1.1, 1.1.2 > Environment: Linux on PPC and x86 >Reporter: Amir Sanjar > Fix For: 1.1.2 > > Attachments: MAPREDUCE-4988-1.1.1.patch > > > TestJobTrackerRestartWithCS, TestCapacitySchedulerServlet, and > TestCapacitySchedulerWithJobTracker testcases potentially could fail when > they are build with ant 1.8.4. > Solution: > port above testcases and ClusterWithCapacityScheduler class from Junit3 to > Junit4. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4988) ClusterWithCapacityScheduler and related testcases needs to be ported to JUnit4.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Sanjar updated MAPREDUCE-4988: --- Priority: Minor (was: Major) > ClusterWithCapacityScheduler and related testcases needs to be ported to > JUnit4. > > > Key: MAPREDUCE-4988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4988 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.1.1, 1.1.2 > Environment: Linux on PPC and x86 >Reporter: Amir Sanjar >Priority: Minor > Fix For: 1.1.2 > > Attachments: MAPREDUCE-4988-1.1.1.patch > > > TestJobTrackerRestartWithCS, TestCapacitySchedulerServlet, and > TestCapacitySchedulerWithJobTracker testcases potentially could fail when > they are build with ant 1.8.4. > Solution: > port above testcases and ClusterWithCapacityScheduler class from Junit3 to > Junit4. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4994: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for following up Sandy. Committed to trunk and branch-2. > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch, > MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578144#comment-13578144 ] Hudson commented on MAPREDUCE-4994: --- Integrated in Hadoop-trunk-Commit #3360 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3360/]) MAPREDUCE-4994. Addendum fixing testcases failures. (sandyr via tucu) (Revision 1446042) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1446042 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.4-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch, > MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5006) streaming tests failing
Alejandro Abdelnur created MAPREDUCE-5006: - Summary: streaming tests failing Key: MAPREDUCE-5006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5006 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 2.0.4-beta Reporter: Alejandro Abdelnur The following 2 tests are failing in trunk * org.apache.hadoop.streaming.TestStreamReduceNone * org.apache.hadoop.streaming.TestStreamXmlRecordReader -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
