[jira] [Commented] (MAPREDUCE-5360) TestMRJobClient fails on Windows due to path format
[
https://issues.apache.org/jira/browse/MAPREDUCE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701314#comment-13701314
]
Hudson commented on MAPREDUCE-5360:
---
Integrated in Hadoop-Yarn-trunk #262 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/262/])
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
Contributed by Chuan Liu. (Revision 1500194)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500194
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
TestMRJobClient fails on Windows due to path format
---
Key: MAPREDUCE-5360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5360
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5360-trunk.2.patch, MAPREDUCE-5360-trunk.patch
This unit test has following two failed test cases on Windows due to path
name format.
* testJobHistory
* testSubmit
When passing local path to the command line, the test cases use file:/// +
a string derived from Java API {{File.getAbsolutePath()}}. The mixed use of
forward and back slashes in the path leads to the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5360) TestMRJobClient fails on Windows due to path format
[
https://issues.apache.org/jira/browse/MAPREDUCE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701335#comment-13701335
]
Hudson commented on MAPREDUCE-5360:
---
Integrated in Hadoop-Hdfs-trunk #1452 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1452/])
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
Contributed by Chuan Liu. (Revision 1500194)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500194
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
TestMRJobClient fails on Windows due to path format
---
Key: MAPREDUCE-5360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5360
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5360-trunk.2.patch, MAPREDUCE-5360-trunk.patch
This unit test has following two failed test cases on Windows due to path
name format.
* testJobHistory
* testSubmit
When passing local path to the command line, the test cases use file:/// +
a string derived from Java API {{File.getAbsolutePath()}}. The mixed use of
forward and back slashes in the path leads to the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5364) Deadlock between RenewalTimerTask methods cancel() and run()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701351#comment-13701351
]
Karthik Kambatla commented on MAPREDUCE-5364:
-
Thanks Sid.
bq. The addendum patch can cause deadlocks on the call to
{{setTimerForTokenRenewal}}
Looking at the code, I don't see a deadlock possibility. While a call to
{{setTimerForTokenRenewal}} requires a lock on DelegationTokenRenewer.class, I
don't see any method holding a lock on DelegationTokenRenewer.class requiring a
lock on delegationTokens or cancelled flag. Am I missing something here?
bq. A cancelled flag could be used on the DelegationTokenToRenew structure
itself. Set intent to cancel before attempting to cancel the timer task, and
check this during renewal and before queuing another renewal.
I think I like this approach better - {{setTimerForTokenRenewal}} can be called
conditionally based on the success of {{DelegationTokenToRenew#renew()}}. Let
me take a stab.
Deadlock between RenewalTimerTask methods cancel() and run()
Key: MAPREDUCE-5364
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5364
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 1.2.1
Attachments: mr-5364-1.patch, mr-5364-addendum-1.patch
MAPREDUCE-4860 introduced a local variable {{cancelled}} in
{{RenewalTimerTask}} to fix the race where {{DelegationTokenRenewal}}
attempts to renew a token even after the job is removed. However, the patch
also makes {{run()}} and {{cancel()}} synchronized methods leading to a
potential deadlock against {{run()}}'s catch-block (error-path).
The deadlock stacks below:
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.cancel()
@bci=0, line=240 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeDelegationTokenRenewalForJob(org.apache.hadoop.mapreduce.JobID)
@bci=109, line=319 (Interpreted frame)
{noformat}
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeFailedDelegationToken(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=62, line=297 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.access$300(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=1, line=47 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run()
@bci=148, line=234 (Interpreted frame)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5360) TestMRJobClient fails on Windows due to path format
[
https://issues.apache.org/jira/browse/MAPREDUCE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701353#comment-13701353
]
Hudson commented on MAPREDUCE-5360:
---
Integrated in Hadoop-Mapreduce-trunk #1479 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1479/])
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
Contributed by Chuan Liu. (Revision 1500194)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500194
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
TestMRJobClient fails on Windows due to path format
---
Key: MAPREDUCE-5360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5360
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5360-trunk.2.patch, MAPREDUCE-5360-trunk.patch
This unit test has following two failed test cases on Windows due to path
name format.
* testJobHistory
* testSubmit
When passing local path to the command line, the test cases use file:/// +
a string derived from Java API {{File.getAbsolutePath()}}. The mixed use of
forward and back slashes in the path leads to the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5256) CombineInputFormat isn't thread safe affecting HiveServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-5256: -- Fix Version/s: 1.2.1 CombineInputFormat isn't thread safe affecting HiveServer - Key: MAPREDUCE-5256 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5256 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 1.2.1 Attachments: MAPREDUCE-5256-20130517.txt This was originally fixed as part of MAPREDUCE-5038, but that got reverted now. Which uncovers this issue, breaking HiveServer. Originally reported by [~thejas]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5256) CombineInputFormat isn't thread safe affecting HiveServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved MAPREDUCE-5256. --- Resolution: Fixed +1. Committed to branch-1.2 and branch-1. Thanks, Vinod! CombineInputFormat isn't thread safe affecting HiveServer - Key: MAPREDUCE-5256 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5256 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 1.2.1 Attachments: MAPREDUCE-5256-20130517.txt This was originally fixed as part of MAPREDUCE-5038, but that got reverted now. Which uncovers this issue, breaking HiveServer. Originally reported by [~thejas]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MAPREDUCE-5256) CombineInputFormat isn't thread safe affecting HiveServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701449#comment-13701449 ] Matt Foley edited comment on MAPREDUCE-5256 at 7/6/13 11:37 PM: +1. Committed to branch-1.2. Seems to be superfluous in branch-1. Thanks, Vinod! was (Author: mattf): +1. Committed to branch-1.2 and branch-1. Thanks, Vinod! CombineInputFormat isn't thread safe affecting HiveServer - Key: MAPREDUCE-5256 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5256 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 1.2.1 Attachments: MAPREDUCE-5256-20130517.txt This was originally fixed as part of MAPREDUCE-5038, but that got reverted now. Which uncovers this issue, breaking HiveServer. Originally reported by [~thejas]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-5351: -- Resolution: Fixed Status: Resolved (was: Patch Available) Both patch and addendum patch have been applied by Arun, to both branch-1.2 and branch-1. Resolving as Fixed. Thanks, Sandy! JobTracker memory leak caused by CleanupQueue reopening FileSystem -- Key: MAPREDUCE-5351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Critical Fix For: 1.2.1 Attachments: MAPREDUCE-5351-1.patch, MAPREDUCE-5351-2.patch, MAPREDUCE-5351-addendum-1.patch, MAPREDUCE-5351-addendum.patch, MAPREDUCE-5351.patch When a job is completed, closeAllForUGI is called to close all the cached FileSystems in the FileSystem cache. However, the CleanupQueue may run after this occurs and call FileSystem.get() to delete the staging directory, adding a FileSystem to the cache that will never be closed. People on the user-list have reported this causing their JobTrackers to OOME every two weeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules
[ https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701510#comment-13701510 ] Matt Foley commented on MAPREDUCE-5227: --- Needs review by MapReduce subject area experts. Pushing to release 1.3.0, pending review. JobTrackerMetricsSource and QueueMetrics should standardize naming rules Key: MAPREDUCE-5227 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.1.3, 1.2.1 Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: MAPREDUCE-5227-1.1-branch.1.patch, MAPREDUCE-5227.1.patch, MAPREDUCE-5227-branch-1.1.patch JobTrackerMetricsSource and QueueMetrics provides users with some metrics, but its naming rules( jobs_running, running_maps, running_reduces) sometimes confuses users. It should be standardized. One concern is backward compatibility, so one idea is to share MetricMutableGaugeInt object from old and new property name. e.g. to share runningMaps from running_maps and maps_running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules
[ https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-5227: -- Target Version/s: 1.3.0 (was: 1.1.3, 1.2.1) JobTrackerMetricsSource and QueueMetrics should standardize naming rules Key: MAPREDUCE-5227 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.1.3, 1.2.1 Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Attachments: MAPREDUCE-5227-1.1-branch.1.patch, MAPREDUCE-5227.1.patch, MAPREDUCE-5227-branch-1.1.patch JobTrackerMetricsSource and QueueMetrics provides users with some metrics, but its naming rules( jobs_running, running_maps, running_reduces) sometimes confuses users. It should be standardized. One concern is backward compatibility, so one idea is to share MetricMutableGaugeInt object from old and new property name. e.g. to share runningMaps from running_maps and maps_running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
