[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070295#comment-13070295 ] Mahadev konar commented on MAPREDUCE-2589: -- Sherry, Sorry I looked at it again and I think its good if we make the DEFAULT_USER_LOG_RETAIN_MAX_HOURS configurable. Also, -1 should disable the feature. I think its important to be able to switch off misbehaving configuration. Also, why do we need a call to get jobs that are still running? I thought the call was made only on restart/reinit? We should be able to clean old user logs without calling jc.jobstocomplete? I think we should avoid adding a dependency on calling jobtracker client methods in the tasktracker itself. what do you think? TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Fix For: 0.20.205.0 Attachments: MAPREDUCE-2589.patch, MAPREDUCE-2589_1.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2621) TestCapacityScheduler fails with Queue q1 does not exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-2621:
-
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Sherry!
TestCapacityScheduler fails with Queue q1 does not exist
Key: MAPREDUCE-2621
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2621
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.205.0
Environment: 0.20.1xx-Secondary
Reporter: Sherry Chen
Assignee: Sherry Chen
Priority: Minor
Fix For: 0.20.205.0
Attachments: MAPREDUCE-2621.patch, MAPREDUCE-2621_1.patch
{quote}
Error Message
Queue q1 does not exist
Stacktrace
java.io.IOException: Queue q1 does not exist
at org.apache.hadoop.mapred.JobInProgress.init(JobInProgress.java:354)
at
org.apache.hadoop.mapred.TestCapacityScheduler$FakeJobInProgress.init(TestCapacityScheduler.java:172)
at
org.apache.hadoop.mapred.TestCapacityScheduler.submitJob(TestCapacityScheduler.java:794)
at
org.apache.hadoop.mapred.TestCapacityScheduler.submitJob(TestCapacityScheduler.java:818)
at
org.apache.hadoop.mapred.TestCapacityScheduler.submitJobAndInit(TestCapacityScheduler.java:825)
at
org.apache.hadoop.mapred.TestCapacityScheduler.testMultiTaskAssignmentInMultipleQueues(TestCapacityScheduler.java:1109)
{quote}
When queue name is invalid, an exception is thrown now.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2650) back-port MAPREDUCE-2238 to 0.20-security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070305#comment-13070305 ] Mahadev konar commented on MAPREDUCE-2650: -- Sherry, I think Todd's comment was on localFS.setPermission() calls. Those calls would throw an exception in this patch and would just return false with a log warning in the original code. Is that correct? Infact all the calls to setpermission have to been changed. The mkdirs change is ok, but the other changes am not sure should be part of this patch? back-port MAPREDUCE-2238 to 0.20-security - Key: MAPREDUCE-2650 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2650 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.20.205.0 Reporter: Sherry Chen Assignee: Sherry Chen Fix For: 0.20.205.0 Attachments: MAPREDUCE-2650.patch Dev had seen the attempt directory permission getting set to 000 or 111 in the CI builds and tests run on dev desktops with 0.20-security. MAPREDUCE-2238 reported and fixed the issue for 0.22.0, back-port to 0.20-security is needed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2650) back-port MAPREDUCE-2238 to 0.20-security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070308#comment-13070308 ] Mahadev konar commented on MAPREDUCE-2650: -- Actually just read through the comments in MAPREDUCE-2238. The patch looks good. can you please post the results of ant test? back-port MAPREDUCE-2238 to 0.20-security - Key: MAPREDUCE-2650 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2650 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.20.205.0 Reporter: Sherry Chen Assignee: Sherry Chen Fix For: 0.20.205.0 Attachments: MAPREDUCE-2650.patch Dev had seen the attempt directory permission getting set to 000 or 111 in the CI builds and tests run on dev desktops with 0.20-security. MAPREDUCE-2238 reported and fixed the issue for 0.22.0, back-port to 0.20-security is needed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2575) TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2575: - Fix Version/s: 0.23.0 TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test -- Key: MAPREDUCE-2575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2575 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.0 Attachments: MAPREDUCE-2575.patch TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2575) TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2575: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1. I just pushed this. Thanks Thomas! TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test -- Key: MAPREDUCE-2575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2575 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.0 Attachments: MAPREDUCE-2575.patch TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2667) MR279: mapred job -kill leaves application in RUNNING state
[ https://issues.apache.org/jira/browse/MAPREDUCE-2667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2667: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just pushed this. Thanks Thomas! MR279: mapred job -kill leaves application in RUNNING state --- Key: MAPREDUCE-2667 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2667 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.0 Attachments: MAPREDUCE-2587-279-v2.patch, MAPREDUCE-2667-mr279-v2.patch, MAPREDUCE-2667-mr279.patch the mapred job -kill command doesn't seem to fully clean up the application. If you kill a job and run mapred job -list again it still shows up as running: mapred job -kill job_1310072430717_0003 Killed job job_1310072430717_0003 mapred job -list Total jobs:1 JobId State StartTime UserNameQueue Priority SchedulingInfo job_1310072430717_0003 RUNNING 0 tgraves default NORMAL 98.139.92.22:19888/yarn/job/job_1310072430717_3_3 Running kill again will error out. It also still shows up in the RM Applications UI as running with a note of: Kill Job received from client job_1310072430717_0003 Job received Kill while in RUNNING state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2575) TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070314#comment-13070314 ] Hudson commented on MAPREDUCE-2575: --- Integrated in Hadoop-Mapreduce-trunk-Commit #754 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/754/]) MAPREDUCE-2575. TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1150533 Files : * /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/MRCaching.java * /hadoop/common/trunk/mapreduce/CHANGES.txt TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test -- Key: MAPREDUCE-2575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2575 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.0 Attachments: MAPREDUCE-2575.patch TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2701) MR-279: app/Job.java needs UGI for the user that launched it
[
https://issues.apache.org/jira/browse/MAPREDUCE-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070315#comment-13070315
]
Mahadev konar commented on MAPREDUCE-2701:
--
Robert,
Wont it be simpler to just store the username in Job? Rather than the whole
UGI? What do you think? Storing the whole UGI is probably not required and also
doesnt sound like a good API.
Just something like:
{code}
String getUserName();
{code}
MR-279: app/Job.java needs UGI for the user that launched it
Key: MAPREDUCE-2701
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2701
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Fix For: 0.23.0
Attachments: MR-2701-v1.patch, MR-2701-v2.patch, MR-2701-v2.patch
./mr-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
is missing some data that is needed by the Job History GUI. It needs the
UGI for the user that launched it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070318#comment-13070318 ] Mahadev konar commented on MAPREDUCE-2489: -- Jeffrey, Sorry, I am a little unclear on what the patch is doing. Can you please specify what you are trying to achieve with the patch? The patch seems to create a URI with hostname and checking if its a valid URI or not? How is that verifying if a hostname is valid or not? Jobsplits with random hostnames can make the queue unusable --- Key: MAPREDUCE-2489 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2489-0.20s-v2.patch, MAPREDUCE-2489-0.20s-v3.patch, MAPREDUCE-2489-0.20s.patch, MAPREDUCE-2489-mapred-v2.patch, MAPREDUCE-2489-mapred-v3.patch, MAPREDUCE-2489-mapred-v4.patch, MAPREDUCE-2489-mapred.patch We saw an issue where a custom InputSplit was returning invalid hostnames for the splits that were then causing the JobTracker to attempt to excessively resolve host names. This caused a major slowdown for the JobTracker. We should prevent invalid InputSplit hostnames from affecting everyone else. I propose we implement some verification for the hostnames to try to ensure that we only do DNS lookups on valid hostnames (and fail otherwise). We could also fail the job after a certain number of failures in the resolve. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
