[jira] Commented: (MAPREDUCE-2285) MiniMRCluster does not start after ant test-patch
[
https://issues.apache.org/jira/browse/MAPREDUCE-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990481#comment-12990481
]
Todd Lipcon commented on MAPREDUCE-2285:
This problem also causes failures in compiling mrunit's tests. eg this build:
https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/25/
(failed with weird errors about assert methods not existing in junit)
The patch I attached last week fixes it. Giri: was your "looks good" a +1? (ie
can I commit to branch and trunk?)
> MiniMRCluster does not start after ant test-patch
> -
>
> Key: MAPREDUCE-2285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Ramkumar Vadali
>Priority: Blocker
> Fix For: 0.22.0, 0.23.0
>
> Attachments: cp-bad, cp-good, fix-build.diff
>
>
> Any test using MiniMRCluster hangs in the MiniMRCluster constructor after
> running ant test-patch. Steps to reproduce:
> 1. ant -Dpatch.file= -Dforrest.home= forrest> -Dfindbugs.home= -Dscratch.dir=/tmp/testpatch
> -Djava5.home= test-patch
> 2. Run any test that creates MiniMRCluster, say ant test
> -Dtestcase=TestFileArgs (contrib/streaming)
> Expected result: Test should succeed
> Actual result: Test hangs in MiniMRCluster.. This does not happen if
> we run ant clean after ant test-patch
> Test output:
> {code}
> [junit] 11/01/27 12:11:43 INFO ipc.Server: IPC Server handler 3 on 58675:
> starting
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: TaskTracker up at:
> localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: Starting tracker
> tracker_host0.foo.com:localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:44 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 0 time(s).
> [junit] 11/01/27 12:11:45 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 1 time(s).
> [junit] 11/01/27 12:11:46 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 2 time(s).
> [junit] 11/01/27 12:11:47 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 3 time(s).
> [junit] 11/01/27 12:11:48 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 4 time(s).
> [junit] 11/01/27 12:11:49 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 5 time(s).
> [junit] 11/01/27 12:11:50 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 6 time(s).
> [junit] 11/01/27 12:11:51 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 7 time(s).
> [junit] 11/01/27 12:11:52 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 8 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 9 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.RPC: Server at localhost/127.0.0.1:0
> not available yet, Z...
> {code}
> Stack trace:
> {code}
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:611)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:429)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:504)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
> at org.apache.hadoop.ipc.Client.call(Client.java:1008)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> at org.apache.hadoop.mapred.$Proxy11.getProtocolVersion(Unknown
> Source)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:235)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:275)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:206)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:185)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:169)
> at org.apache.hadoop.mapred.TaskTracker$2.run(TaskTracker.java:699)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1142)
> at
> org.apache.hadoop.mapred.T
[jira] Commented: (MAPREDUCE-2188) The new API MultithreadedMapper doesn't call the initialize method of the RecordReader
[ https://issues.apache.org/jira/browse/MAPREDUCE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990447#comment-12990447 ] Hudson commented on MAPREDUCE-2188: --- Integrated in Hadoop-Mapreduce-trunk-Commit #601 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/601/]) MAPREDUCE-2188. The new API MultithreadedMapper doesn't initialize RecordReader. Contributed by Owen O'Malley. > The new API MultithreadedMapper doesn't call the initialize method of the > RecordReader > -- > > Key: MAPREDUCE-2188 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2188 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.22.0 > > Attachments: mr2188.patch > > > The wrapping RecordReader in the Multithreaded Mapper is never initialized. > With HADOOP-6685, this becomes a problem because the ReflectionUtils.copy > requires a non-null configuration. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (MAPREDUCE-2188) The new API MultithreadedMapper doesn't call the initialize method of the RecordReader
[ https://issues.apache.org/jira/browse/MAPREDUCE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved MAPREDUCE-2188. Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] I just committed this. Thank you Owen and Priyo for testing. > The new API MultithreadedMapper doesn't call the initialize method of the > RecordReader > -- > > Key: MAPREDUCE-2188 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2188 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.22.0 > > Attachments: mr2188.patch > > > The wrapping RecordReader in the Multithreaded Mapper is never initialized. > With HADOOP-6685, this becomes a problem because the ReflectionUtils.copy > requires a non-null configuration. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2077) Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin
[ https://issues.apache.org/jira/browse/MAPREDUCE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990441#comment-12990441 ] Hudson commented on MAPREDUCE-2077: --- Integrated in Hadoop-Mapreduce-trunk-Commit #600 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/600/]) MAPREDUCE-2077. Resolve name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin Contributed by Luke Lu. > Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin > -- > > Key: MAPREDUCE-2077 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2077 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Luke Lu >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-2077-trunk-v1.patch > > > Name clash compile error in the deprecated > org.apache.hadoop.util.MemoryCalculatorPlugin due to JLS3 8.4.8.3 (cf. > http://bugs.sun.com/view_bug.do?bug_id=6182950) > The bug doesn't manifest in jdk 1.6 up to 20, but shows up in NetBeans 6.9+ > due to its bundled (conforming) compiler. Fix is trivial: just remove the > offending method in the deprecated subclass as its equivalent erasure is > inherited from the parent class anyway. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2077) Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin
[ https://issues.apache.org/jira/browse/MAPREDUCE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-2077: --- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thank you Luke Lu. Thanks Po for testing this. > Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin > -- > > Key: MAPREDUCE-2077 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2077 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Luke Lu >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-2077-trunk-v1.patch > > > Name clash compile error in the deprecated > org.apache.hadoop.util.MemoryCalculatorPlugin due to JLS3 8.4.8.3 (cf. > http://bugs.sun.com/view_bug.do?bug_id=6182950) > The bug doesn't manifest in jdk 1.6 up to 20, but shows up in NetBeans 6.9+ > due to its bundled (conforming) compiler. Fix is trivial: just remove the > offending method in the deprecated subclass as its equivalent erasure is > inherited from the parent class anyway. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2188) The new API MultithreadedMapper doesn't call the initialize method of the RecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990437#comment-12990437
]
Konstantin Shvachko commented on MAPREDUCE-2188:
test-patch results
{code}
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no new tests are needed
for this patch.
[exec] Also please list what manual steps were
performed to verify this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
==
{code}
> The new API MultithreadedMapper doesn't call the initialize method of the
> RecordReader
> --
>
> Key: MAPREDUCE-2188
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2188
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: mr2188.patch
>
>
> The wrapping RecordReader in the Multithreaded Mapper is never initialized.
> With HADOOP-6685, this becomes a problem because the ReflectionUtils.copy
> requires a non-null configuration.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1915) IndexCache - getIndexInformation - check reduce index Out Of Bounds
[ https://issues.apache.org/jira/browse/MAPREDUCE-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990430#comment-12990430 ] Priyo Mustafi commented on MAPREDUCE-1915: -- [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 20 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] +1 system test framework. The patch passed system test framework compile. > IndexCache - getIndexInformation - check reduce index Out Of Bounds > --- > > Key: MAPREDUCE-1915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1915 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0 >Reporter: Rares Vernica >Assignee: Priyo Mustafi >Priority: Trivial > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1915.patch, MAPREDUCE-1915_1.patch > > > When checking if the "reduce" index is out of bounds the check should be > info.mapSpillRecord.size() <= reduce > instead of: > info.mapSpillRecord.size() < reduce > Not a big bug since an Out Of Bounds is thrown downstream anyway. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2267) Parallelize reading of blocks within a stripe
[ https://issues.apache.org/jira/browse/MAPREDUCE-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali updated MAPREDUCE-2267: --- Status: Open (was: Patch Available) Will upload another patch > Parallelize reading of blocks within a stripe > - > > Key: MAPREDUCE-2267 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2267 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Affects Versions: 0.22.0 >Reporter: Ramkumar Vadali >Assignee: Ramkumar Vadali > Attachments: MAPREDUCE-2267.1.patch, MAPREDUCE-2267.2.patch, > MAPREDUCE-2267.3.patch, MAPREDUCE-2267.4.patch, MAPREDUCE-2267.patch > > > RAID code has several instances where several blocks of data have to be read > to perform an operation. For example, computing a parity block requires > reading the blocks of the source file. Similarly, generating a fixed block > requires reading a parity block and the good blocks from the source file. > These read operations proceed sequentially currently. RAID code should use a > thread pool to increase the parallelism and thus reduce latency. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1706) Log RAID recoveries on HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990422#comment-12990422 ] Hudson commented on MAPREDUCE-1706: --- Integrated in Hadoop-Mapreduce-trunk-Commit #599 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/599/]) MAPREDUCE-1706. Log RAID recoveries on HDFS. (schen) > Log RAID recoveries on HDFS > --- > > Key: MAPREDUCE-1706 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1706 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Affects Versions: 0.22.0 >Reporter: Rodrigo Schmidt >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-1706.rebased.txt, MAPREDUCE-1706.txt, > test.result, test.result > > > It would be good to have a way to centralize all the recovery logs, since > recovery can be executed by any hdfs client. The best place to store this > information is HDFS itself. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1706) Log RAID recoveries on HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1706: -- Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I have committed this. Thanks Ram for the review. > Log RAID recoveries on HDFS > --- > > Key: MAPREDUCE-1706 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1706 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Affects Versions: 0.22.0 >Reporter: Rodrigo Schmidt >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-1706.rebased.txt, MAPREDUCE-1706.txt, > test.result, test.result > > > It would be good to have a way to centralize all the recovery logs, since > recovery can be executed by any hdfs client. The best place to store this > information is HDFS itself. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1410) Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is killed with a KillTaskAction
[ https://issues.apache.org/jira/browse/MAPREDUCE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990393#comment-12990393 ] Priyo Mustafi commented on MAPREDUCE-1410: -- Just ran it again after installing gcc and some other packages mentioned on the net and the system failures are gone now. [exec] +1 system test framework. The patch passed system test framework compile. > Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is > killed with a KillTaskAction > --- > > Key: MAPREDUCE-1410 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1410 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: MAPREDUCE-1410.patch, MAPREDUCE-1410.patch, > patch-1410.txt > > > If the main attempt is killed with a KillTaskAction and is added to > tasksReportedClosed, then the cleanup-attempt for the task (with same id) can > not be given a KillTaskAction, since tasksReportedClosed already contains the > attemptID. > The attemptID should be removed from tasksReportedClosed in > incompleteSubTask() method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1410) Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is killed with a KillTaskAction
[ https://issues.apache.org/jira/browse/MAPREDUCE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990381#comment-12990381 ] Konstantin Shvachko commented on MAPREDUCE-1410: > -1 system test framework. You probably have some problems with gcc there. I don't think it is blocking as there are no changes to native code, and this target succeeded on Doug's run before. > -1 release audit. Don't see any new files are introduced in the patch. We probably need to add some exclusions, which were omitted in previous commits. Other than that looks good to me. How are the tests doing with this patch? > Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is > killed with a KillTaskAction > --- > > Key: MAPREDUCE-1410 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1410 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: MAPREDUCE-1410.patch, MAPREDUCE-1410.patch, > patch-1410.txt > > > If the main attempt is killed with a KillTaskAction and is added to > tasksReportedClosed, then the cleanup-attempt for the task (with same id) can > not be given a KillTaskAction, since tasksReportedClosed already contains the > attemptID. > The attemptID should be removed from tasksReportedClosed in > incompleteSubTask() method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2077) Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin
[
https://issues.apache.org/jira/browse/MAPREDUCE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990380#comment-12990380
]
Po Cheung commented on MAPREDUCE-2077:
--
Results of running test-patch are below. This patch does not need new/modified
tests.
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no new tests are needed
for this patch.
[exec] Also please list what manual steps were
performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
{noformat}
test-patch.properties:
{noformat}
OK_RELEASEAUDIT_WARNINGS=2
OK_FINDBUGS_WARNINGS=13
OK_JAVADOC_WARNINGS=0
{noformat}
"ant run-commit-tests" was successful
> Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin
> --
>
> Key: MAPREDUCE-2077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Luke Lu
>Assignee: Luke Lu
> Attachments: mr-2077-trunk-v1.patch
>
>
> Name clash compile error in the deprecated
> org.apache.hadoop.util.MemoryCalculatorPlugin due to JLS3 8.4.8.3 (cf.
> http://bugs.sun.com/view_bug.do?bug_id=6182950)
> The bug doesn't manifest in jdk 1.6 up to 20, but shows up in NetBeans 6.9+
> due to its bundled (conforming) compiler. Fix is trivial: just remove the
> offending method in the deprecated subclass as its equivalent erasure is
> inherited from the parent class anyway.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2302) Add static factory methods in GaloisField
[
https://issues.apache.org/jira/browse/MAPREDUCE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990376#comment-12990376
]
Scott Chen commented on MAPREDUCE-2302:
---
{code}
[exec] BUILD SUCCESSFUL
[exec] Total time: 2 minutes 16 seconds
[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> Add static factory methods in GaloisField
> -
>
> Key: MAPREDUCE-2302
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2302
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
>Reporter: Scott Chen
>Assignee: Scott Chen
> Attachments: MAPREDUCE-2302.txt
>
>
> GaloisField is immutable and should be kept reuse after creation to avoid
> redundant calculation of the multiplication and division tables.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2118) optimize getJobSetupAndCleanupTasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990371#comment-12990371 ] Joydeep Sen Sarma commented on MAPREDUCE-2118: -- ok - i am running them and will upload the results when done. > optimize getJobSetupAndCleanupTasks > > > Key: MAPREDUCE-2118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2118 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Joydeep Sen Sarma >Assignee: Joydeep Sen Sarma > Attachments: mapreduce-2118.1.patch, mapreduce.2118.2.patch > > > in every heartbeat, while holding the JobTracker global lock, all jobs are > scanned for job setup/cleanup, task setup/cleanup. on a large system with > many trackers (and heartbeats) and many jobs - this becomes the bottleneck > for JT throughput. > One possible route may be to rework the code to not require the JT lock while > asking the JIP whether it has a setup/cleanup task. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990368#comment-12990368 ] Scott Chen commented on MAPREDUCE-1974: --- test-patch result [exec] [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2239) BlockPlacementPolicyRaid should call getBlockLocations only when necessary
[
https://issues.apache.org/jira/browse/MAPREDUCE-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990363#comment-12990363
]
Scott Chen commented on MAPREDUCE-2239:
---
{code}
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> BlockPlacementPolicyRaid should call getBlockLocations only when necessary
> --
>
> Key: MAPREDUCE-2239
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2239
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
>Affects Versions: 0.23.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2239-1.txt, MAPREDUCE-2239-2.txt,
> MAPREDUCE-2239.txt
>
>
> Currently BlockPlacementPolicyRaid calls getBlockLocations for every
> chooseTarget().
> This puts pressure on NameNode. We should avoid calling if this file is not
> raided or a parity file.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2206) The task-cleanup tasks should be optional
[
https://issues.apache.org/jira/browse/MAPREDUCE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990360#comment-12990360
]
Scott Chen commented on MAPREDUCE-2206:
---
test-patch
{code}
[exec] BUILD SUCCESSFUL
[exec] Total time: 2 minutes 2 seconds
[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> The task-cleanup tasks should be optional
> -
>
> Key: MAPREDUCE-2206
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2206
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.23.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2206.txt
>
>
> For job does not use OutputCommitter.abort(), this should be able to turn off.
> This improves the latency of the job because failed tasks are often the
> bottleneck of the jobs.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990359#comment-12990359 ] Scott Chen commented on MAPREDUCE-1974: --- I posted the result on the wrong patch. It should not have a +1 on test included. Will run this again. > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[
https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990356#comment-12990356
]
Scott Chen commented on MAPREDUCE-1974:
---
{code}
[exec]
[exec] BUILD SUCCESSFUL
[exec] Total time: 2 minutes 15 seconds
[exec]
[exec]
[exec]
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> FairScheduler can preempt the same task many times
> --
>
> Key: MAPREDUCE-1974
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
>Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Attachments: MAPREDUCE-1974.txt
>
>
> In FairScheduler.preemptTasks(), tasks are collected from
> JobInProgress.runningMapCache.
> But tasks repeat multiple times in JobInProgress.runningMapCache (on rack,
> node and cluster).
> This makes FairScheduler preempt the same task many times.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2118) optimize getJobSetupAndCleanupTasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990352#comment-12990352 ] Scott Chen commented on MAPREDUCE-2118: --- Joy: Looks good to me. Do you want to manually run test-patch and test? I think Hudson doesn't work these days. > optimize getJobSetupAndCleanupTasks > > > Key: MAPREDUCE-2118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2118 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Joydeep Sen Sarma >Assignee: Joydeep Sen Sarma > Attachments: mapreduce-2118.1.patch, mapreduce.2118.2.patch > > > in every heartbeat, while holding the JobTracker global lock, all jobs are > scanned for job setup/cleanup, task setup/cleanup. on a large system with > many trackers (and heartbeats) and many jobs - this becomes the bottleneck > for JT throughput. > One possible route may be to rework the code to not require the JT lock while > asking the JIP whether it has a setup/cleanup task. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1410) Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is killed with a KillTaskAction
[ https://issues.apache.org/jira/browse/MAPREDUCE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990347#comment-12990347 ] Priyo Mustafi commented on MAPREDUCE-1410: -- Ran test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 10 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings). [exec] [exec] -1 system test framework. The patch failed system test framework compile. [exec] The release audit complains about two files, smoke-test and robots.txt. These don't have AL and are probably correct. > Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is > killed with a KillTaskAction > --- > > Key: MAPREDUCE-1410 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1410 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: MAPREDUCE-1410.patch, MAPREDUCE-1410.patch, > patch-1410.txt > > > If the main attempt is killed with a KillTaskAction and is added to > tasksReportedClosed, then the cleanup-attempt for the task (with same id) can > not be given a KillTaskAction, since tasksReportedClosed already contains the > attemptID. > The attemptID should be removed from tasksReportedClosed in > incompleteSubTask() method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1410) Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is killed with a KillTaskAction
[ https://issues.apache.org/jira/browse/MAPREDUCE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyo Mustafi updated MAPREDUCE-1410: - Attachment: MAPREDUCE-1410.patch Existing patch wasn't valid anymore because 1) JobTracker.jobs don't hold completed job in memory any longer. 2) Code was trying to contact namenode. Added MiniDFSCluster. The changes are only to Junit's and the original code fix remains as is in the patch > Task-Cleanup attempt cannot be given KillTaskAction if the main attempt is > killed with a KillTaskAction > --- > > Key: MAPREDUCE-1410 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1410 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: MAPREDUCE-1410.patch, MAPREDUCE-1410.patch, > patch-1410.txt > > > If the main attempt is killed with a KillTaskAction and is added to > tasksReportedClosed, then the cleanup-attempt for the task (with same id) can > not be given a KillTaskAction, since tasksReportedClosed already contains the > attemptID. > The attemptID should be removed from tasksReportedClosed in > incompleteSubTask() method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2301) TestDataDrivenDBInputFormat fails if port 9001 is occupied
[ https://issues.apache.org/jira/browse/MAPREDUCE-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2301: --- Status: Patch Available (was: Open) > TestDataDrivenDBInputFormat fails if port 9001 is occupied > -- > > Key: MAPREDUCE-2301 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2301 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: mapreduce-2301.txt > > > If you have any process listening on port 9001 (the default hsqldb port) then > TestDataDrivenDBInputFormat will fail with: > Connection is broken: java.io.EOFException > java.sql.SQLException: Connection is broken: java.io.EOFException > at org.hsqldb.jdbc.Util.sqlException(Unknown Source) > in the logs you see: > [Server@42704baa]: [Thread[HSQLDB Server @42704baa,5,main]]: > run()/openServerSocket(): > java.net.BindException: Address already in use -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2301) TestDataDrivenDBInputFormat fails if port 9001 is occupied
[ https://issues.apache.org/jira/browse/MAPREDUCE-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2301: --- Attachment: mapreduce-2301.txt This patch has HSQLDB pick a random port and try again until it gets one that's open (up to 10 retries) > TestDataDrivenDBInputFormat fails if port 9001 is occupied > -- > > Key: MAPREDUCE-2301 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2301 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: mapreduce-2301.txt > > > If you have any process listening on port 9001 (the default hsqldb port) then > TestDataDrivenDBInputFormat will fail with: > Connection is broken: java.io.EOFException > java.sql.SQLException: Connection is broken: java.io.EOFException > at org.hsqldb.jdbc.Util.sqlException(Unknown Source) > in the logs you see: > [Server@42704baa]: [Thread[HSQLDB Server @42704baa,5,main]]: > run()/openServerSocket(): > java.net.BindException: Address already in use -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1825) jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps and finishedReduces when job is not initialized
[
https://issues.apache.org/jira/browse/MAPREDUCE-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990340#comment-12990340
]
Scott Chen commented on MAPREDUCE-1825:
---
test-patch result:
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no new tests are needed
for this patch.
[exec] Also please list what manual steps were
performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps
> and finishedReduces when job is not initialized
> ---
>
> Key: MAPREDUCE-1825
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1825
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.20.1
>Reporter: Amareshwari Sriramadasu
>Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1825.txt, MAPREDUCE-1825_1.txt,
> MAPREDUCE-1825_2.txt, MAPREDUCE-1825_3.txt, jstacks.zip
>
>
> JobInProgress.finishedMaps() and finishedReduces() are synchronized. They are
> called from jobqueue_details.jsp and FairSchedulerServelet which iterates
> through all jobs. If any job is in initialization, these pages don't come up
> until the initialization finishes.
> See
> [comment|https://issues.apache.org/jira/browse/MAPREDUCE-1354?focusedCommentId=12834139&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12834139]
> for more details
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2118) optimize getJobSetupAndCleanupTasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joydeep Sen Sarma updated MAPREDUCE-2118: - Attachment: mapreduce.2118.2.patch Refreshed patch against latest trunk. I didn't remove the comment because it's actually in a block of comment containing notes on why the call was removed. i think it's better to leave these comments in because the change is subtle but very important. > optimize getJobSetupAndCleanupTasks > > > Key: MAPREDUCE-2118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2118 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Joydeep Sen Sarma >Assignee: Joydeep Sen Sarma > Attachments: mapreduce-2118.1.patch, mapreduce.2118.2.patch > > > in every heartbeat, while holding the JobTracker global lock, all jobs are > scanned for job setup/cleanup, task setup/cleanup. on a large system with > many trackers (and heartbeats) and many jobs - this becomes the bottleneck > for JT throughput. > One possible route may be to rework the code to not require the JT lock while > asking the JIP whether it has a setup/cleanup task. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2302) Add static factory methods in GaloisField
[ https://issues.apache.org/jira/browse/MAPREDUCE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-2302: -- Status: Patch Available (was: In Progress) > Add static factory methods in GaloisField > - > > Key: MAPREDUCE-2302 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2302 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-2302.txt > > > GaloisField is immutable and should be kept reuse after creation to avoid > redundant calculation of the multiplication and division tables. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2302) Add static factory methods in GaloisField
[ https://issues.apache.org/jira/browse/MAPREDUCE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-2302: -- Attachment: MAPREDUCE-2302.txt > Add static factory methods in GaloisField > - > > Key: MAPREDUCE-2302 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2302 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-2302.txt > > > GaloisField is immutable and should be kept reuse after creation to avoid > redundant calculation of the multiplication and division tables. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Work started: (MAPREDUCE-2302) Add static factory methods in GaloisField
[ https://issues.apache.org/jira/browse/MAPREDUCE-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-2302 started by Scott Chen. > Add static factory methods in GaloisField > - > > Key: MAPREDUCE-2302 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2302 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-2302.txt > > > GaloisField is immutable and should be kept reuse after creation to avoid > redundant calculation of the multiplication and division tables. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2296) Fix references to misspelled message name getProtocolSigature
[ https://issues.apache.org/jira/browse/MAPREDUCE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990335#comment-12990335 ] Hudson commented on MAPREDUCE-2296: --- Integrated in Hadoop-Mapreduce-trunk-Commit #598 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/598/]) MAPREDUCE-2296. Fix references to misspelled method name getProtocolSigature. Contributed by Todd Lipcon > Fix references to misspelled message name getProtocolSigature > - > > Key: MAPREDUCE-2296 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2296 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: mr-2296.txt > > > HADOOP-7129 fixed the typo, need to update usages in MR. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2302) Add static factory methods in GaloisField
Add static factory methods in GaloisField - Key: MAPREDUCE-2302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2302 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/raid Reporter: Scott Chen Assignee: Scott Chen GaloisField is immutable and should be kept reuse after creation to avoid redundant calculation of the multiplication and division tables. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2200) TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to be conditional
[ https://issues.apache.org/jira/browse/MAPREDUCE-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990334#comment-12990334 ] Hudson commented on MAPREDUCE-2200: --- Integrated in Hadoop-Mapreduce-trunk-Commit #598 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/598/]) Move MAPREDUCE-2200 to 0.22 section of CHANGES.txt since it was applied to that branch. > TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to > be conditional > > > Key: MAPREDUCE-2200 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2200 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2200.patch > > > TestUmbilicalProtocolWithJobToken requires Krb environment to be set. For > testing some 'pseudo' environment is needed (similar to HDFS-1284). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2301) TestDataDrivenDBInputFormat fails if port 9001 is occupied
TestDataDrivenDBInputFormat fails if port 9001 is occupied -- Key: MAPREDUCE-2301 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2301 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor If you have any process listening on port 9001 (the default hsqldb port) then TestDataDrivenDBInputFormat will fail with: Connection is broken: java.io.EOFException java.sql.SQLException: Connection is broken: java.io.EOFException at org.hsqldb.jdbc.Util.sqlException(Unknown Source) in the logs you see: [Server@42704baa]: [Thread[HSQLDB Server @42704baa,5,main]]: run()/openServerSocket(): java.net.BindException: Address already in use -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1974) FairScheduler can preempt the same task many times
[ https://issues.apache.org/jira/browse/MAPREDUCE-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990329#comment-12990329 ] Joydeep Sen Sarma commented on MAPREDUCE-1974: -- +1. this seems like a no-brainer! > FairScheduler can preempt the same task many times > -- > > Key: MAPREDUCE-1974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1974 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/fair-share >Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Attachments: MAPREDUCE-1974.txt > > > In FairScheduler.preemptTasks(), tasks are collected from > JobInProgress.runningMapCache. > But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, > node and cluster). > This makes FairScheduler preempt the same task many times. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2206) The task-cleanup tasks should be optional
[ https://issues.apache.org/jira/browse/MAPREDUCE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990328#comment-12990328 ] Joydeep Sen Sarma commented on MAPREDUCE-2206: -- +1 from my side. one thing is that very very few people will be aware that this can be turned off. in particular - i think the default outputformats don't need a task cleanup. i am wondering how this can be turned on automatically for more use cases. - we can make the setting a default one in hive-default.xml - i will file a jira for that. - how about hadoop streaming? can we turn task cleanup off if hadoop streaming is used with the (default) fileoutputformat? > The task-cleanup tasks should be optional > - > > Key: MAPREDUCE-2206 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2206 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2206.txt > > > For job does not use OutputCommitter.abort(), this should be able to turn off. > This improves the latency of the job because failed tasks are often the > bottleneck of the jobs. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990325#comment-12990325 ] Hudson commented on MAPREDUCE-2300: --- Integrated in Hadoop-Mapreduce-trunk-Commit #597 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/597/]) MAPREDUCE-2300. Fix TestUmbilicalProtocolWithJobToken on trunk after HADOOP-6904. Contributed by Todd Lipcon > TestUmbilicalProtocolWithJobToken failing > - > > Key: MAPREDUCE-2300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: mapreduce-2300.txt > > > Testcase: testJobTokenRpc took 0.678 sec > Caused an ERROR > null > java.lang.NullPointerException > at > org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) > at > org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (MAPREDUCE-2296) Fix references to misspelled message name getProtocolSigature
[ https://issues.apache.org/jira/browse/MAPREDUCE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-2296. Resolution: Fixed Hadoop Flags: [Reviewed] Verified this compiles against the current Common snapshot. Committed to trunk > Fix references to misspelled message name getProtocolSigature > - > > Key: MAPREDUCE-2296 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2296 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: mr-2296.txt > > > HADOOP-7129 fixed the typo, need to update usages in MR. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2200) TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to be conditional
[ https://issues.apache.org/jira/browse/MAPREDUCE-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2200: --- Affects Version/s: (was: 0.23.0) 0.22.0 Fix Version/s: (was: 0.23.0) 0.22.0 Added this to branch-0.22 and moved it in CHANGES.txt on trunk. > TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to > be conditional > > > Key: MAPREDUCE-2200 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2200 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2200.patch > > > TestUmbilicalProtocolWithJobToken requires Krb environment to be set. For > testing some 'pseudo' environment is needed (similar to HDFS-1284). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2200) TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to be conditional
[ https://issues.apache.org/jira/browse/MAPREDUCE-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990312#comment-12990312 ] Todd Lipcon commented on MAPREDUCE-2200: I think this needs to go on branch-0.22 as well - the 22 build is failing with the same error. > TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to > be conditional > > > Key: MAPREDUCE-2200 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2200 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2200.patch > > > TestUmbilicalProtocolWithJobToken requires Krb environment to be set. For > testing some 'pseudo' environment is needed (similar to HDFS-1284). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2300: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk > TestUmbilicalProtocolWithJobToken failing > - > > Key: MAPREDUCE-2300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: mapreduce-2300.txt > > > Testcase: testJobTokenRpc took 0.678 sec > Caused an ERROR > null > java.lang.NullPointerException > at > org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) > at > org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2026) JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()
[ https://issues.apache.org/jira/browse/MAPREDUCE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990288#comment-12990288 ] Hudson commented on MAPREDUCE-2026: --- Integrated in Hadoop-Mapreduce-trunk-Commit #596 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/596/]) MAPREDUCE-2026. Make JobTracker.getJobCounters() and JobInProgress.getCounters() aquire locks in a shorter time period. (Joydeep Sen Sarma via schen) > JobTracker.getJobCounters() should not hold JobTracker lock while calling > JobInProgress.getCounters() > - > > Key: MAPREDUCE-2026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2026 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Joydeep Sen Sarma > Fix For: 0.22.0, 0.23.0 > > Attachments: 2026.1.patch, MAPREDUCE-2026.txt > > > JobTracker.getJobCounter() will lock JobTracker and call > JobInProgress.getCounters(). > JobInProgress.getCounters() can be very expensive because it aggregates all > the task counters. > We found that from the JobTracker jstacks that this method is one of the > bottleneck of the JobTracker performance. > JobInProgress.getCounters() should be able to be called out side the > JobTracker lock because it already has JobInProgress lock. > For example, it is used by jobdetails.jsp without a JobTracker lock. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990283#comment-12990283 ] Konstantin Boudnik commented on MAPREDUCE-2300: --- +1 patch looks good > TestUmbilicalProtocolWithJobToken failing > - > > Key: MAPREDUCE-2300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: mapreduce-2300.txt > > > Testcase: testJobTokenRpc took 0.678 sec > Caused an ERROR > null > java.lang.NullPointerException > at > org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) > at > org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2292) Provide a shell interface for querying the status of FairScheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990270#comment-12990270 ] Scott Chen commented on MAPREDUCE-2292: --- +1 to Allen's idea. > Provide a shell interface for querying the status of FairScheduler > -- > > Key: MAPREDUCE-2292 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2292 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Scott Chen >Priority: Minor > > It will be useful if we have some shell interface to obtain some status in > FairScheduler. > Just like we can use > bin/hadoop job [-list|-list-trackers|-kill-task|...] > to obtain JobTracker information. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2026) JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()
[ https://issues.apache.org/jira/browse/MAPREDUCE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-2026: -- Resolution: Fixed Fix Version/s: 0.23.0 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Joydeep. > JobTracker.getJobCounters() should not hold JobTracker lock while calling > JobInProgress.getCounters() > - > > Key: MAPREDUCE-2026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2026 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Joydeep Sen Sarma > Fix For: 0.22.0, 0.23.0 > > Attachments: 2026.1.patch, MAPREDUCE-2026.txt > > > JobTracker.getJobCounter() will lock JobTracker and call > JobInProgress.getCounters(). > JobInProgress.getCounters() can be very expensive because it aggregates all > the task counters. > We found that from the JobTracker jstacks that this method is one of the > bottleneck of the JobTracker performance. > JobInProgress.getCounters() should be able to be called out side the > JobTracker lock because it already has JobInProgress lock. > For example, it is used by jobdetails.jsp without a JobTracker lock. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (MAPREDUCE-2298) FairScheduler status page allows users to set pool and priority
[ https://issues.apache.org/jira/browse/MAPREDUCE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Angeles resolved MAPREDUCE-2298. Resolution: Duplicate HADOOP-5485 > FairScheduler status page allows users to set pool and priority > --- > > Key: MAPREDUCE-2298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2298 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Patrick Angeles > > The FairScheduler status page should allow users to view status without > enabling admin functions like set pool and set priority. > One proposal is to disable all admin commands from the FairScheduler status > page to make it read only. We would have to provide an alternate > administration interface, for example via shell (MAPREDUCE-2292). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2298) FairScheduler status page allows users to set pool and priority
[ https://issues.apache.org/jira/browse/MAPREDUCE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990253#comment-12990253 ] Patrick Angeles commented on MAPREDUCE-2298: Turning it off is half of it. The other half is allowing authorized users to still set pool/priority, which I suppose can be addressed by MAPREDUCE-2292. > FairScheduler status page allows users to set pool and priority > --- > > Key: MAPREDUCE-2298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2298 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Patrick Angeles > > The FairScheduler status page should allow users to view status without > enabling admin functions like set pool and set priority. > One proposal is to disable all admin commands from the FairScheduler status > page to make it read only. We would have to provide an alternate > administration interface, for example via shell (MAPREDUCE-2292). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2026) JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()
[ https://issues.apache.org/jira/browse/MAPREDUCE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990252#comment-12990252 ] Scott Chen commented on MAPREDUCE-2026: --- +1 Patch looks good. And we have been using this in production for a while. I will commit this. > JobTracker.getJobCounters() should not hold JobTracker lock while calling > JobInProgress.getCounters() > - > > Key: MAPREDUCE-2026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2026 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Joydeep Sen Sarma > Attachments: 2026.1.patch, MAPREDUCE-2026.txt > > > JobTracker.getJobCounter() will lock JobTracker and call > JobInProgress.getCounters(). > JobInProgress.getCounters() can be very expensive because it aggregates all > the task counters. > We found that from the JobTracker jstacks that this method is one of the > bottleneck of the JobTracker performance. > JobInProgress.getCounters() should be able to be called out side the > JobTracker lock because it already has JobInProgress lock. > For example, it is used by jobdetails.jsp without a JobTracker lock. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (MAPREDUCE-2281) Fix javac, javadoc, findbugs warnings
[ https://issues.apache.org/jira/browse/MAPREDUCE-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reassigned MAPREDUCE-2281: -- Assignee: Po Cheung (was: Chad Metcalf) > Fix javac, javadoc, findbugs warnings > - > > Key: MAPREDUCE-2281 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2281 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Po Cheung >Assignee: Po Cheung > Fix For: 0.22.0 > > Attachments: mapreduce-2281-trunk.patch > > > Split from HADOOP-6642 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2026) JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()
[ https://issues.apache.org/jira/browse/MAPREDUCE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990248#comment-12990248 ] Joydeep Sen Sarma commented on MAPREDUCE-2026: -- yeah - i think we have this patch in production for many months now - let me regenerate the patch since it's been a long while. > JobTracker.getJobCounters() should not hold JobTracker lock while calling > JobInProgress.getCounters() > - > > Key: MAPREDUCE-2026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2026 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Joydeep Sen Sarma > Attachments: 2026.1.patch, MAPREDUCE-2026.txt > > > JobTracker.getJobCounter() will lock JobTracker and call > JobInProgress.getCounters(). > JobInProgress.getCounters() can be very expensive because it aggregates all > the task counters. > We found that from the JobTracker jstacks that this method is one of the > bottleneck of the JobTracker performance. > JobInProgress.getCounters() should be able to be called out side the > JobTracker lock because it already has JobInProgress lock. > For example, it is used by jobdetails.jsp without a JobTracker lock. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2298) FairScheduler status page allows users to set pool and priority
[ https://issues.apache.org/jira/browse/MAPREDUCE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990244#comment-12990244 ] Todd Lipcon commented on MAPREDUCE-2298: In trunk we already check JSPUtil.privateActionsAllowed() (webinterface.private.actions) before allowing these. Is that not enough? Clearly using the authentication system and an ACL would be preferable, but I don't think we should wholesale disable this feature since it's already easy to turn off. > FairScheduler status page allows users to set pool and priority > --- > > Key: MAPREDUCE-2298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2298 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Patrick Angeles > > The FairScheduler status page should allow users to view status without > enabling admin functions like set pool and set priority. > One proposal is to disable all admin commands from the FairScheduler status > page to make it read only. We would have to provide an alternate > administration interface, for example via shell (MAPREDUCE-2292). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2300: --- Status: Patch Available (was: Open) > TestUmbilicalProtocolWithJobToken failing > - > > Key: MAPREDUCE-2300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: mapreduce-2300.txt > > > Testcase: testJobTokenRpc took 0.678 sec > Caused an ERROR > null > java.lang.NullPointerException > at > org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) > at > org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2300: --- Attachment: mapreduce-2300.txt Patch fixes the broken test. > TestUmbilicalProtocolWithJobToken failing > - > > Key: MAPREDUCE-2300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: mapreduce-2300.txt > > > Testcase: testJobTokenRpc took 0.678 sec > Caused an ERROR > null > java.lang.NullPointerException > at > org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) > at > org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned MAPREDUCE-2300: -- Assignee: Todd Lipcon > TestUmbilicalProtocolWithJobToken failing > - > > Key: MAPREDUCE-2300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: mapreduce-2300.txt > > > Testcase: testJobTokenRpc took 0.678 sec > Caused an ERROR > null > java.lang.NullPointerException > at > org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) > at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) > at > org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2300) TestUmbilicalProtocolWithJobToken failing
TestUmbilicalProtocolWithJobToken failing - Key: MAPREDUCE-2300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2300 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Todd Lipcon Priority: Blocker Testcase: testJobTokenRpc took 0.678 sec Caused an ERROR null java.lang.NullPointerException at org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:241) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) at org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken$1.run(TestUmbilicalProtocolWithJobToken.java:102) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2299) In the Job Tracker UI -> task details page, machine and logs links are navigating to page not found error.
In the Job Tracker UI -> task details page, machine and logs links are navigating to page not found error. -- Key: MAPREDUCE-2299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2299 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Devaraj K 1. In the page showing All task attempts-On clicking of machine link is navigating to page not found error. 2. In the page showing All task attempts-On clicking of Task logs Link is navigating to page not found error. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2298) FairScheduler status page allows users to set pool and priority
FairScheduler status page allows users to set pool and priority --- Key: MAPREDUCE-2298 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2298 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Patrick Angeles The FairScheduler status page should allow users to view status without enabling admin functions like set pool and set priority. One proposal is to disable all admin commands from the FairScheduler status page to make it read only. We would have to provide an alternate administration interface, for example via shell (MAPREDUCE-2292). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2298) FairScheduler status page allows users to set pool and priority
[ https://issues.apache.org/jira/browse/MAPREDUCE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Angeles updated MAPREDUCE-2298: --- Component/s: contrib/fair-share > FairScheduler status page allows users to set pool and priority > --- > > Key: MAPREDUCE-2298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2298 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Patrick Angeles > > The FairScheduler status page should allow users to view status without > enabling admin functions like set pool and set priority. > One proposal is to disable all admin commands from the FairScheduler status > page to make it read only. We would have to provide an alternate > administration interface, for example via shell (MAPREDUCE-2292). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2297) All map reduce tasks are failing if we give invalid path jar file for Job
All map reduce tasks are failing if we give invalid path jar file for Job
-
Key: MAPREDUCE-2297
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2297
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.20.2
Reporter: Devaraj K
Priority: Minor
This can be reproduced by giving the invalid jar file for the Job or it can be
reproduced from hive.
In hive-default.xml
hive.aux.jars.path
Provided for adding auxillaryjarsPath
If we configure an invalid path for jar file, It is making all map reduce tasks
to fail even those jobs are not depending on this jar file and it is giving the
below exception.
{code:xml}
hive> select * from a join b on(a.b=b.c);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
java.io.FileNotFoundException: File does not exist: /user/root/grade.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:495)
at
org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:651)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:783)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:698)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
{code}
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
