[jira] [Commented] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules
[ https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701510#comment-13701510 ] Matt Foley commented on MAPREDUCE-5227: --- Needs review by MapReduce subject area experts. Pushing to release 1.3.0, pending review. > JobTrackerMetricsSource and QueueMetrics should standardize naming rules > > > Key: MAPREDUCE-5227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 1.1.3, 1.2.1 >Reporter: Tsuyoshi OZAWA >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: MAPREDUCE-5227-1.1-branch.1.patch, > MAPREDUCE-5227.1.patch, MAPREDUCE-5227-branch-1.1.patch > > > JobTrackerMetricsSource and QueueMetrics provides users with some metrics, > but its naming rules( "jobs_running", "running_maps", "running_reduces") > sometimes confuses users. It should be standardized. > One concern is backward compatibility, so one idea is to share > MetricMutableGaugeInt object from old and new property name. > e.g. to share runningMaps from "running_maps" and "maps_running". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules
[ https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-5227: -- Target Version/s: 1.3.0 (was: 1.1.3, 1.2.1) > JobTrackerMetricsSource and QueueMetrics should standardize naming rules > > > Key: MAPREDUCE-5227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 1.1.3, 1.2.1 >Reporter: Tsuyoshi OZAWA >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: MAPREDUCE-5227-1.1-branch.1.patch, > MAPREDUCE-5227.1.patch, MAPREDUCE-5227-branch-1.1.patch > > > JobTrackerMetricsSource and QueueMetrics provides users with some metrics, > but its naming rules( "jobs_running", "running_maps", "running_reduces") > sometimes confuses users. It should be standardized. > One concern is backward compatibility, so one idea is to share > MetricMutableGaugeInt object from old and new property name. > e.g. to share runningMaps from "running_maps" and "maps_running". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-5351: -- Resolution: Fixed Status: Resolved (was: Patch Available) Both patch and addendum patch have been applied by Arun, to both branch-1.2 and branch-1. Resolving as Fixed. Thanks, Sandy! > JobTracker memory leak caused by CleanupQueue reopening FileSystem > -- > > Key: MAPREDUCE-5351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Assignee: Sandy Ryza >Priority: Critical > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5351-1.patch, MAPREDUCE-5351-2.patch, > MAPREDUCE-5351-addendum-1.patch, MAPREDUCE-5351-addendum.patch, > MAPREDUCE-5351.patch > > > When a job is completed, closeAllForUGI is called to close all the cached > FileSystems in the FileSystem cache. However, the CleanupQueue may run after > this occurs and call FileSystem.get() to delete the staging directory, adding > a FileSystem to the cache that will never be closed. > People on the user-list have reported this causing their JobTrackers to OOME > every two weeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MAPREDUCE-5256) CombineInputFormat isn't thread safe affecting HiveServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701449#comment-13701449 ] Matt Foley edited comment on MAPREDUCE-5256 at 7/6/13 11:37 PM: +1. Committed to branch-1.2. Seems to be superfluous in branch-1. Thanks, Vinod! was (Author: mattf): +1. Committed to branch-1.2 and branch-1. Thanks, Vinod! > CombineInputFormat isn't thread safe affecting HiveServer > - > > Key: MAPREDUCE-5256 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5256 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5256-20130517.txt > > > This was originally fixed as part of MAPREDUCE-5038, but that got reverted > now. Which uncovers this issue, breaking HiveServer. Originally reported by > [~thejas]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5256) CombineInputFormat isn't thread safe affecting HiveServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved MAPREDUCE-5256. --- Resolution: Fixed +1. Committed to branch-1.2 and branch-1. Thanks, Vinod! > CombineInputFormat isn't thread safe affecting HiveServer > - > > Key: MAPREDUCE-5256 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5256 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5256-20130517.txt > > > This was originally fixed as part of MAPREDUCE-5038, but that got reverted > now. Which uncovers this issue, breaking HiveServer. Originally reported by > [~thejas]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5256) CombineInputFormat isn't thread safe affecting HiveServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-5256: -- Fix Version/s: 1.2.1 > CombineInputFormat isn't thread safe affecting HiveServer > - > > Key: MAPREDUCE-5256 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5256 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 1.2.1 > > Attachments: MAPREDUCE-5256-20130517.txt > > > This was originally fixed as part of MAPREDUCE-5038, but that got reverted > now. Which uncovers this issue, breaking HiveServer. Originally reported by > [~thejas]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5360) TestMRJobClient fails on Windows due to path format
[
https://issues.apache.org/jira/browse/MAPREDUCE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701353#comment-13701353
]
Hudson commented on MAPREDUCE-5360:
---
Integrated in Hadoop-Mapreduce-trunk #1479 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1479/])
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
Contributed by Chuan Liu. (Revision 1500194)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500194
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
> TestMRJobClient fails on Windows due to path format
> ---
>
> Key: MAPREDUCE-5360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Minor
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: MAPREDUCE-5360-trunk.2.patch, MAPREDUCE-5360-trunk.patch
>
>
> This unit test has following two failed test cases on Windows due to path
> name format.
> * testJobHistory
> * testSubmit
> When passing local path to the command line, the test cases use "file:///" +
> a string derived from Java API {{File.getAbsolutePath()}}. The mixed use of
> forward and back slashes in the path leads to the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5364) Deadlock between RenewalTimerTask methods cancel() and run()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701351#comment-13701351
]
Karthik Kambatla commented on MAPREDUCE-5364:
-
Thanks Sid.
bq. The addendum patch can cause deadlocks on the call to
{{setTimerForTokenRenewal}}
Looking at the code, I don't see a deadlock possibility. While a call to
{{setTimerForTokenRenewal}} requires a lock on DelegationTokenRenewer.class, I
don't see any method holding a lock on DelegationTokenRenewer.class requiring a
lock on delegationTokens or cancelled flag. Am I missing something here?
bq. A cancelled flag could be used on the DelegationTokenToRenew structure
itself. Set intent to cancel before attempting to cancel the timer task, and
check this during renewal and before queuing another renewal.
I think I like this approach better - {{setTimerForTokenRenewal}} can be called
conditionally based on the success of {{DelegationTokenToRenew#renew()}}. Let
me take a stab.
> Deadlock between RenewalTimerTask methods cancel() and run()
>
>
> Key: MAPREDUCE-5364
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5364
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Fix For: 1.2.1
>
> Attachments: mr-5364-1.patch, mr-5364-addendum-1.patch
>
>
> MAPREDUCE-4860 introduced a local variable {{cancelled}} in
> {{RenewalTimerTask}} to fix the race where {{DelegationTokenRenewal}}
> attempts to renew a token even after the job is removed. However, the patch
> also makes {{run()}} and {{cancel()}} synchronized methods leading to a
> potential deadlock against {{run()}}'s catch-block (error-path).
> The deadlock stacks below:
> {noformat}
> -
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.cancel()
> @bci=0, line=240 (Interpreted frame)
> -
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeDelegationTokenRenewalForJob(org.apache.hadoop.mapreduce.JobID)
> @bci=109, line=319 (Interpreted frame)
> {noformat}
> {noformat}
> -
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeFailedDelegationToken(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
> @bci=62, line=297 (Interpreted frame)
> -
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.access$300(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
> @bci=1, line=47 (Interpreted frame)
> -
> org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run()
> @bci=148, line=234 (Interpreted frame)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5360) TestMRJobClient fails on Windows due to path format
[
https://issues.apache.org/jira/browse/MAPREDUCE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701335#comment-13701335
]
Hudson commented on MAPREDUCE-5360:
---
Integrated in Hadoop-Hdfs-trunk #1452 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1452/])
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
Contributed by Chuan Liu. (Revision 1500194)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500194
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
> TestMRJobClient fails on Windows due to path format
> ---
>
> Key: MAPREDUCE-5360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Minor
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: MAPREDUCE-5360-trunk.2.patch, MAPREDUCE-5360-trunk.patch
>
>
> This unit test has following two failed test cases on Windows due to path
> name format.
> * testJobHistory
> * testSubmit
> When passing local path to the command line, the test cases use "file:///" +
> a string derived from Java API {{File.getAbsolutePath()}}. The mixed use of
> forward and back slashes in the path leads to the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5360) TestMRJobClient fails on Windows due to path format
[
https://issues.apache.org/jira/browse/MAPREDUCE-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701314#comment-13701314
]
Hudson commented on MAPREDUCE-5360:
---
Integrated in Hadoop-Yarn-trunk #262 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/262/])
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
Contributed by Chuan Liu. (Revision 1500194)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500194
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
> TestMRJobClient fails on Windows due to path format
> ---
>
> Key: MAPREDUCE-5360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Minor
> Fix For: 3.0.0, 2.1.0-beta
>
> Attachments: MAPREDUCE-5360-trunk.2.patch, MAPREDUCE-5360-trunk.patch
>
>
> This unit test has following two failed test cases on Windows due to path
> name format.
> * testJobHistory
> * testSubmit
> When passing local path to the command line, the test cases use "file:///" +
> a string derived from Java API {{File.getAbsolutePath()}}. The mixed use of
> forward and back slashes in the path leads to the failure.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
