date:20090810


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-838:
--

Attachment: patch-838.txt

Patch with the fix for trunk

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-838:
--

Status: Patch Available  (was: Open)

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-757) JobConf will not be deleted from the logs folder if job retires from finalizeJob()


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741223#action_12741223
 ] 

Amar Kamat commented on MAPREDUCE-757:
--

Attached a short term fix for now.

 JobConf will not be deleted from the logs folder if job retires from 
 finalizeJob()
 --

 Key: MAPREDUCE-757
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-757
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Amar Kamat
Assignee: Amar Kamat
 Attachments: MAPREDUCE-757-v1.0.patch, 
 MAPREDUCE-757-v2.0-branch-0.20.patch, MAPREDUCE-757-v2.0.patch


 MAPREDUCE-130 fixed the case where the job is retired from the retire jobs 
 thread. But jobs can also retire when the num-job-per-user limit is exceeded. 
 In such cases the conf file will not be deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException

2009-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741243#action_12741243
 ] 

Hadoop QA commented on MAPREDUCE-838:
-

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416024/patch-838.txt
  against trunk revision 802645.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed core unit tests.

-1 contrib tests.  The patch failed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/460/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/460/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/460/artifact/trunk/build/test/checkstyle-errors.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/460/console

This message is automatically generated.

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the

[jira] Commented: (MAPREDUCE-745) TestRecoveryManager fails sometimes


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741247#action_12741247
 ] 

Amareshwari Sriramadasu commented on MAPREDUCE-745:
---

Observed a test timeout in one of the hudson builds. @ 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/460/testReport/org.apache.hadoop.mapred/TestRecoveryManager/testJobTracker

 TestRecoveryManager fails sometimes
 ---

 Key: MAPREDUCE-745
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-745
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Amareshwari Sriramadasu
 Fix For: 0.21.0

 Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, 
 MAPREDUCE-745-v1.3.patch, 
 TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, 
 TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-720) Task logs should have right access-control


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V resolved MAPREDUCE-720.
-

Resolution: Duplicate

The latest approach slated at HADOOP-4491 invalidates this JIRA issue. 
Task-logs are set secure permissions of 2770 user:mapred as part of HADOOP-4491 
itself. Closing this issue as a duplicate.

 Task logs should have right access-control
 --

 Key: MAPREDUCE-720
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-720
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Reporter: Vinod K V
Assignee: Sreekanth Ramakrishnan
 Attachments: mapreduce-720-1.patch, mapreduce-720.patch




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-777:
--

Attachment: patch-777.txt

Attaching an early patch for review.

Patch does the following:
1. Deprecates RunningJob api. Job should be used instead. Made sure that job 
has all methods that RunningJob has.
2. Ports the existing JobClient functionality to new api.
This made org.apache.hadoop.mapred.HistoryViewer, 
org.apache.hadoop.mapred.LocalJobRunner, QueueAclsInfo and 
JobSubmissionProtocol classes public.
Modified JobSubmissionProtocol to use new api.

Owen, can you please look at the patch once and let me know if you want to 
add/remove/change any api?




 A method for finding and tracking jobs from the new API
 ---

 Key: MAPREDUCE-777
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-777
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: client
Reporter: Owen O'Malley
Assignee: Amareshwari Sriramadasu
 Fix For: 0.21.0

 Attachments: patch-777.txt


 We need to create a replacement interface for the JobClient API in the new 
 interface. In particular, the user needs to be able to query and track jobs 
 that were launched by other processes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-777) A method for finding and tracking jobs from the new API


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-777:
--

Fix Version/s: 0.21.0
   Status: Patch Available  (was: Open)

 A method for finding and tracking jobs from the new API
 ---

 Key: MAPREDUCE-777
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-777
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: client
Reporter: Owen O'Malley
Assignee: Amareshwari Sriramadasu
 Fix For: 0.21.0

 Attachments: patch-777.txt


 We need to create a replacement interface for the JobClient API in the new 
 interface. In particular, the user needs to be able to query and track jobs 
 that were launched by other processes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-838:
--

Status: Open  (was: Patch Available)

Uploaded patch was missing retry logic for umbilical

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-838:
--

Attachment: patch-838-1-0.20.txt

Patch for 0.20 with proper fix

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838-1-0.20.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Moved: (MAPREDUCE-842) Per-job local data on the TaskTracker node should have right access-control


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V moved HADOOP-4491 to MAPREDUCE-842:
-

Component/s: (was: security)
 tasktracker
Key: MAPREDUCE-842  (was: HADOOP-4491)
Project: Hadoop Map/Reduce  (was: Hadoop Common)

 Per-job local data on the TaskTracker node should have right access-control
 ---

 Key: MAPREDUCE-842
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-842
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Reporter: Arun C Murthy
Assignee: Vinod K V
 Attachments: HADOOP-4491-20090623-common.1.txt, 
 HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, 
 HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, 
 HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, 
 HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, 
 HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, 
 HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, 
 HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-838:
--

Status: Patch Available  (was: Open)

TestTaskCommit passes on both branch 0.20 and trunk. It Fails without the patch

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838-1-0.20.txt, 
 patch-838-1.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-842) Per-job local data on the TaskTracker node should have right access-control


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V updated MAPREDUCE-842:


Issue Type: Sub-task  (was: Bug)
Parent: MAPREDUCE-563

 Per-job local data on the TaskTracker node should have right access-control
 ---

 Key: MAPREDUCE-842
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-842
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: tasktracker
Reporter: Arun C Murthy
Assignee: Vinod K V
 Attachments: HADOOP-4491-20090623-common.1.txt, 
 HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, 
 HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, 
 HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, 
 HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, 
 HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, 
 HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, 
 HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-842) Per-job local data on the TaskTracker node should have right access-control


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V updated MAPREDUCE-842:


Status: Patch Available  (was: Open)

 Per-job local data on the TaskTracker node should have right access-control
 ---

 Key: MAPREDUCE-842
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-842
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: tasktracker
Reporter: Arun C Murthy
Assignee: Vinod K V
 Attachments: HADOOP-4491-20090623-common.1.txt, 
 HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, 
 HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, 
 HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, 
 HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, 
 HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, 
 HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, 
 HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741284#action_12741284
 ] 

Amareshwari Sriramadasu commented on MAPREDUCE-838:
---

test-patch result:
{noformat}
 [exec]
 [exec]
 [exec] +1 overall.
 [exec]
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec]
 [exec] +1 tests included.  The patch appears to include 3 new or 
modified tests.
 [exec]
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec]
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec]
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec]
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.
 [exec]
{noformat}

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838-1-0.20.txt, 
 patch-838-1.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-805) Deadlock in Jobtracker


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat updated MAPREDUCE-805:
-

Attachment: MAPREDUCE-805-v1.11.patch

Attaching a new patch with some bug fixes. Result of test-patch
 [exec] +1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 18 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.

All tests except TestReduceFetch and TestJobTrackerRestartWithLostTracker 
passed on my box. Rerun of TestJobTrackerRestartWithLostTracker passed. 
TestReduceFetch is a known issue.

 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-843) Cleanup JobInProgress

Cleanup JobInProgress 
--

 Key: MAPREDUCE-843
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-843
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobtracker
Reporter: Amar Kamat


Rework/Redesign JobInProgress contracts and code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-844) TestJobTrackerRestartWithLostTracker fails sometimes

TestJobTrackerRestartWithLostTracker fails sometimes


 Key: MAPREDUCE-844
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-844
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Amar Kamat


After restart the tasks fail with the following error 

2009-08-10 16:35:56,673 WARN  mapred.TaskTracker 
(TaskTracker.java:startNewTask(1717)) - Error initializing 
attempt_200908101626_0001_m_30_1003:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find 
taskTracker/jobcache/job_200908101626_0001/work in any of the configured local 
directories
at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:394)
at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:142)
at 
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:1865)
at 
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:1978)
at 
org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:860)
at 
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:854)
at 
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1713)
at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:99)
at 
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1678)

leading to job failure. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-805) Deadlock in Jobtracker


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat updated MAPREDUCE-805:
-

Attachment: MAPREDUCE-805-v1.11-branch-0.20.patch

patch for branch 0.20

 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, 
 MAPREDUCE-805-v1.11-branch-0.20.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-832) Too many WARN messages about deprecated memorty config variables in JobTacker log

2009-08-10 Thread rahul k singh (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rahul k singh updated MAPREDUCE-832:


Attachment: mapreduce-832-20.patch

Added the new patch.
All the warning messages are moved to constructor

 Too many WARN messages about deprecated memorty config variables in JobTacker 
 log
 -

 Key: MAPREDUCE-832
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-832
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.1
Reporter: Karam Singh
 Attachments: mapreduce-832-20.patch


 When user submit a mapred job using old memory config vairiable 
 (mapred.task.maxmem) followinig message too many times in JobTracker logs -:
 [
 WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no 
 longer used instead use  mapred.job.map.memory.mb and 
 mapred.job.reduce.memory.mb
 ]

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-813) Streaming Doc and M/R-Tutorial Doc - updates

[
https://issues.apache.org/jira/browse/MAPREDUCE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Devaraj Das updated MAPREDUCE-813:
--

Attachment: MAPREDUCE-813-3.patch

For some reason, the 813-2.patch doesn't apply cleanly on trunk. Attaching a
patch that applies cleanly on trunk

Streaming Doc and M/R-Tutorial Doc - updates
-

Key: MAPREDUCE-813
URL: https://issues.apache.org/jira/browse/MAPREDUCE-813
Project: Hadoop Map/Reduce
Issue Type: Task
Components: documentation
Affects Versions: 0.21.0
Reporter: Corinne Chandel
Priority: Minor
Fix For: 0.21.0

Attachments: mapred_tutorial.pdf, MAPREDUCE-813-2.patch,
MAPREDUCE-813-3.patch, MAPREDUCE-813.patch, streaming.pdf

This JIRA addresses issues in the Streaming doc that also require a
cross-link to and update in the M/R Tutorial doc. All changes approved by the
reviewing engineer.
Streaming issues:
1. During the execution of a streaming job, the names of the mapred
parameters are transformed. The dots ( . ) become underscores ( _ ).
Docs affected: streaming and m/r tutorial (new sub-sections added under Task
Execution Environment section)
2. For -files and -archives options, Hadoop now creates symlink with same
name as file (user-defined symlinks, #mysymlink, currently not supported)
Docs affected:streaming
3. Streaming supports streaming command options and generic command options.
Generic options must be placed before streaming options, otherwise command
fails.
Docs affected: streaming (reorganized the streaming doc to make distinctions
between 2 sets of command options more clear)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-813) Streaming Doc and M/R-Tutorial Doc - updates

[
https://issues.apache.org/jira/browse/MAPREDUCE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Devaraj Das updated MAPREDUCE-813:
--

Status: Patch Available (was: Open)

Streaming Doc and M/R-Tutorial Doc - updates
-

Attachments: mapred_tutorial.pdf, MAPREDUCE-813-2.patch,
MAPREDUCE-813-3.patch, MAPREDUCE-813.patch, streaming.pdf

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-813) Streaming Doc and M/R-Tutorial Doc - updates

[
https://issues.apache.org/jira/browse/MAPREDUCE-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Devaraj Das updated MAPREDUCE-813:
--

Status: Open (was: Patch Available)

Streaming Doc and M/R-Tutorial Doc - updates
-

Attachments: mapred_tutorial.pdf, MAPREDUCE-813-2.patch,
MAPREDUCE-813-3.patch, MAPREDUCE-813.patch, streaming.pdf

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741313#action_12741313
 ] 

Amar Kamat commented on MAPREDUCE-805:
--

ant tests on branch-0.20 failed on 
 hdfs.TestDistributedFileSystem, namenode.TestStartup and TestReduceFetch. 
Contrib tests failed on TestStreamingExitStatus.


 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, 
 MAPREDUCE-805-v1.11-branch-0.20.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741323#action_12741323
 ] 

Amareshwari Sriramadasu commented on MAPREDUCE-838:
---

All unit tests passed except TestRecoveryManager (due to MAPREDUCE-745)

 Task succeeds even when committer.commitTask fails with IOException
 ---

 Key: MAPREDUCE-838
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-838
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.20.1
Reporter: Koji Noguchi
Assignee: Amareshwari Sriramadasu
Priority: Blocker
 Fix For: 0.20.1

 Attachments: patch-838-0.20.txt, patch-838-1-0.20.txt, 
 patch-838-1.txt, patch-838.txt


 In MAPREDUCE-837, job succeeded with empty output even though all the tasks 
 were throwing IOException at commiter.commitTask.
 {noformat}
 2009-08-07 17:51:47,458 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,466 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 committing: java.io.IOException: Can not get the relative path: \
 base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,468 WARN org.apache.hadoop.mapred.TaskRunner: Failure 
 asking whether task can commit: java.io.IOException: \
 Can not get the relative path: base = 
 hdfs://mynamenode:8020/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0
  \
 child = 
 hdfs://mynamenode/user/knoguchi/test2.har/_temporary/_attempt_200907301448_8771_r_00_0/_index
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:150)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:106)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:126)
   at 
 org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:86)
   at 
 org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:171)
   at org.apache.hadoop.mapred.Task.commit(Task.java:768)
   at org.apache.hadoop.mapred.Task.done(Task.java:692)
   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)
 2009-08-07 17:51:47,469 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 attempt_200907301448_8771_r_00_0 is allowed to commit now
 2009-08-07 17:51:47,472 INFO org.apache.hadoop.mapred.TaskRunner: Task 
 'attempt_200907301448_8771_r_00_0' done.
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-745:
--

Attachment: TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt

Saw different assertion failure for TestRecoveryManager

 TestRecoveryManager fails sometimes
 ---

 Key: MAPREDUCE-745
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-745
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Reporter: Amareshwari Sriramadasu
 Fix For: 0.21.0

 Attachments: MAPREDUCE-745-v1.0.patch, MAPREDUCE-745-v1.2.patch, 
 MAPREDUCE-745-v1.3.patch, 
 TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, 
 TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt, 
 TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API

2009-08-10 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741326#action_12741326
]

Hadoop QA commented on MAPREDUCE-777:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12416042/patch-777.txt
against trunk revision 802645.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 9 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 2272 javac compiler warnings (more
than the trunk's current 2232 warnings).

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/461/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/461/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/461/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/461/console

This message is automatically generated.

A method for finding and tracking jobs from the new API
---

Key: MAPREDUCE-777
URL: https://issues.apache.org/jira/browse/MAPREDUCE-777
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: client
Reporter: Owen O'Malley
Assignee: Amareshwari Sriramadasu
Fix For: 0.21.0

Attachments: patch-777.txt

We need to create a replacement interface for the JobClient API in the new
interface. In particular, the user needs to be able to query and track jobs
that were launched by other processes.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-814) Move completed Job history files to HDFS

2009-08-10 Thread Sharad Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharad Agarwal updated MAPREDUCE-814:
-

Release Note: Provides an ability to move completed job history files to a 
HDFS location via  configuring mapred.job.tracker.history.completed.location.

 Move completed Job history files to HDFS
 

 Key: MAPREDUCE-814
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-814
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: jobtracker
Reporter: Sharad Agarwal
Assignee: Sharad Agarwal
 Fix For: 0.21.0

 Attachments: 814_v1.patch, 814_v2.patch, 814_v3.patch, 814_v4.patch, 
 814_v5.patch, 814_ydist.patch


 Currently completed job history files remain on the jobtracker node. Having 
 the files available on HDFS will enable clients to access these files more 
 easily.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-808) Buffer objects incorrectly serialized to typed bytes


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj Das updated MAPREDUCE-808:
--

   Resolution: Fixed
Fix Version/s: 0.21.0
   Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Klaas!

 Buffer objects incorrectly serialized to typed bytes
 

 Key: MAPREDUCE-808
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-808
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Klaas Bosteels
Assignee: Klaas Bosteels
 Fix For: 0.21.0

 Attachments: MAPREDUCE-808.patch


 {{TypedBytesOutput.write()}} should do something like
 {code}
 Buffer buf = (Buffer) obj;
 writeBytes(buf.get(), 0, bug.getCount());
 {code}
 instead of
 {code}
 writeBytes(((Buffer) obj).get());
 {code}
 since the bytes returned by {{Buffer.get()}} are only valid between 0 and 
 getCount() - 1.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-750) Extensible ConnManager factory API

2009-08-10 Thread Aaron Kimball (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741482#action_12741482
 ] 

Aaron Kimball commented on MAPREDUCE-750:
-

Failures are in streaming. This adds a release audit warning, as I mentioned it 
would.

 Extensible ConnManager factory API
 --

 Key: MAPREDUCE-750
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-750
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Attachments: MAPREDUCE-750.2.patch, MAPREDUCE-750.3.patch, 
 MAPREDUCE-750.patch


 Sqoop uses the ConnFactory class to instantiate a ConnManager implementation 
 based on the connect string and other arguments supplied by the user. This 
 allows per-database logic to be encapsulated in different ConnManager 
 instances, and dynamically chosen based on which database the user is 
 actually importing from. But adding new ConnManager implementations requires 
 modifying the source of a common ConnFactory class. An indirection layer 
 should be used to delegate instantiation to a number of factory 
 implementations which can be specified in the static configuration or at 
 runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-845) build.xml


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lee Tucker updated MAPREDUCE-845:
-

Attachment: MAPRED-845.patch

Parameterize the heap size for the findbugs command, setting the default to 
512M as it's currently set.

 build.xml
 -

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-845) build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lee Tucker updated MAPREDUCE-845:
-

Summary: build.xml hard codes findbugs heap size, in some configurations 
512M is insufficient to successfully build  (was: build.xml)

 build.xml hard codes findbugs heap size, in some configurations 512M is 
 insufficient to successfully build
 --

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Assignee: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-845) build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build

2009-08-10 Thread Nigel Daley (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Daley updated MAPREDUCE-845:
--

Hadoop Flags: [Reviewed]

+1 code review.

 build.xml hard codes findbugs heap size, in some configurations 512M is 
 insufficient to successfully build
 --

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Assignee: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-845) build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741602#action_12741602
 ] 

Lee Tucker commented on MAPREDUCE-845:
--

Results of a local run of test-patch:

 [exec] -1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] -1 tests included.  The patch doesn't appear to include any new 
or modified tests.
 [exec] Please justify why no new tests are needed 
for this patch.
 [exec] Also please list what manual steps were 
performed to verify this patch.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.
 [exec] 


 build.xml hard codes findbugs heap size, in some configurations 512M is 
 insufficient to successfully build
 --

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Assignee: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-845) build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build

2009-08-10 Thread Tsz Wo (Nicholas), SZE (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741607#action_12741607
 ] 

Tsz Wo (Nicholas), SZE commented on MAPREDUCE-845:
--

Just a question: after the patch, are we supposed to pass 
-Dfindbugs.heap.size=... when we run ant findbugs?

 build.xml hard codes findbugs heap size, in some configurations 512M is 
 insufficient to successfully build
 --

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Assignee: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-845) build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741611#action_12741611
 ] 

Lee Tucker commented on MAPREDUCE-845:
--

No need, I preserved the original value of 512M as the default.   However, if 
needed you can override it on the command line.

 build.xml hard codes findbugs heap size, in some configurations 512M is 
 insufficient to successfully build
 --

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Assignee: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-845) build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build

2009-08-10 Thread Tsz Wo (Nicholas), SZE (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated MAPREDUCE-845:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Since this is a build.xml change, no new test is needed.

I have committed this.  Thanks, Lee!

 build.xml hard codes findbugs heap size, in some configurations 512M is 
 insufficient to successfully build
 --

 Key: MAPREDUCE-845
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-845
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.21.0
 Environment: building on RHEL5 with both javadoc and findbugs in the 
 same line
Reporter: Lee Tucker
Assignee: Lee Tucker
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-845.patch

   Original Estimate: 0.03h
  Remaining Estimate: 0.03h

 When attempting the build with the hardcoded value of 512M for findbugs heap 
 size, the build fails with:
  [findbugs] Java Result: -1
  [xslt] Processing 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
  to 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.html
  [xslt] Loading stylesheet 
 /homes/hadoopqa/tools/findbugs/latest/src/xsl/default.xsl
  [xslt] : Error! Premature end of file.
  [xslt] : Error! 
 com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end 
 of file.
  [xslt] Failed to process 
 /grid/0/gs/gridre/SpringMapRedLevel2/build/test/findbugs/hadoop-findbugs-report.xml
 BUILD FAILED

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.

2009-08-10 Thread Philip Zeyliger (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741630#action_12741630
 ] 

Philip Zeyliger commented on MAPREDUCE-157:
---

Would this be a good place to try the Avro serialization format in Hadoop 
proper?  If text-formatting is desired, AVRO-50 has a text format for Avro, 
which is JSON already.  So you'd basically be implementing the same thing, but 
with the extra context of an Avro schema.

-- Philip

 Job History log file format is not friendly for external tools.
 ---

 Key: MAPREDUCE-157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-157
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Jothi Padmanabhan

 Currently, parsing the job history logs with external tools is very difficult 
 because of the format. The most critical problem is that newlines aren't 
 escaped in the strings. That makes using tools like grep, sed, and awk very 
 tricky.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.

2009-08-10 Thread Philip Zeyliger (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741634#action_12741634
 ] 

Philip Zeyliger commented on MAPREDUCE-157:
---

done

On Mon, Aug 10, 2009 at 1:08 AM, Jothi Padmanabhan (JIRA)



 Job History log file format is not friendly for external tools.
 ---

 Key: MAPREDUCE-157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-157
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Jothi Padmanabhan

 Currently, parsing the job history logs with external tools is very difficult 
 because of the format. The most critical problem is that newlines aren't 
 escaped in the strings. That makes using tools like grep, sed, and awk very 
 tricky.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-157) Job History log file format is not friendly for external tools.

2009-08-10 Thread Philip Zeyliger (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741641#action_12741641
 ] 

Philip Zeyliger commented on MAPREDUCE-157:
---

Ack, ignore that done.  Was in the wrong browser tab.

 Job History log file format is not friendly for external tools.
 ---

 Key: MAPREDUCE-157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-157
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Jothi Padmanabhan

 Currently, parsing the job history logs with external tools is very difficult 
 because of the format. The most critical problem is that newlines aren't 
 escaped in the strings. That makes using tools like grep, sed, and awk very 
 tricky.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-842) Per-job local data on the TaskTracker node should have right access-control

2009-08-10 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741664#action_12741664
]

Hadoop QA commented on MAPREDUCE-842:
-

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12416038/HADOOP-4491-20090810.3.txt
against trunk revision 802954.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 47 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 5 new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/463/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/463/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/463/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/463/console

This message is automatically generated.

Per-job local data on the TaskTracker node should have right access-control
---

Key: MAPREDUCE-842
URL: https://issues.apache.org/jira/browse/MAPREDUCE-842
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: tasktracker
Reporter: Arun C Murthy
Assignee: Vinod K V
Attachments: HADOOP-4491-20090623-common.1.txt,
HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt,
HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt,
HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt,
HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt,
HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt,
HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt,
HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-768) Configuration information should generate dump in a standard format.

2009-08-10 Thread Sreekanth Ramakrishnan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741670#action_12741670
 ] 

Sreekanth Ramakrishnan commented on MAPREDUCE-768:
--

Resource field of the configuration property is the last resource from which 
the properties value has been loaded. The motivation of this field would be for 
administrators to know, for if they have accidentally overridden any property 
they didn't mean to.

 Configuration information should generate dump in a standard format.
 

 Key: MAPREDUCE-768
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-768
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: rahul k singh

  We need to generate the configuration dump in a standard format .

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741680#action_12741680
 ] 

Amar Kamat commented on MAPREDUCE-805:
--

We cannot test the deadlock code but tested the code that has changed i.e 
job-init, job-kill, empty-job, job-with-no-setup-cleanup and job with 
0-maps/reduces. 
||job-empty?||setup-cleanup-required?||killed in init?||result||
|yes|yes|yes|pass (job killed)|
|yes|yes|no|pass (setup-cleanup launched and job succeeded)|
|yes|no|yes|pass (job killed)|
|yes|no|no|pass (job marked succeeded in JobTracker.initJob())|
|no|yes|yes|pass (job killed after init)|
|no|yes|no|pass (job runs to completion)| 
|no|no|yes|pass (job killed after init)|
|no|no|no|pass (job runs to completion)|

Did I miss anything?

 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, 
 MAPREDUCE-805-v1.11-branch-0.20.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741681#action_12741681
 ] 

Amar Kamat commented on MAPREDUCE-805:
--

Note that I purposefully added sleeps in JobTracker.initJob() and 
JobInProgress.initTasks to take care of race conditions. I didnt see any side 
effect. With this patch init will always keep the job in PREP state but based 
on whether 
- setup is required or not 
- tasks are needed to run 
- job-kill was issued during init 
- job-init failed

the job can move to RUNNING or SUCCCEEDED or KILLED or FAILED state or remain 
in PREP state. Here is how the state transition happens (note that after 
job.initTasks() the job will be in PREP state)
||setup needed?||maps=0 and reduces=0?||job killed during init?||init 
failed?||new state||comment||
|*|*|*|yes|FAILED|irrespective of what the config is, if the job fails in init, 
its marked as FAILED|
|*|*|yes|no|KILLED|irrespective of what the config is, if the job is killed 
during init and init passed normally then the job is marked as KILLED|
|yes|*|no|no|PREP|if job is configured to run setup then the job will remain in 
PREP state|
|no|yes|no|no|SUCCEEDED|if the job has no setup configured and if there are no 
maps and reduces then the job is marked SUCCEEDED|
|no|no|no|no|RUNNING|if the job has no setup configured and if there are maps 
and reduces then the job is marked RUNNING|


 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, 
 MAPREDUCE-805-v1.11-branch-0.20.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-842) Per-job local data on the TaskTracker node should have right access-control


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V updated MAPREDUCE-842:


Status: Open  (was: Patch Available)

 Per-job local data on the TaskTracker node should have right access-control
 ---

 Key: MAPREDUCE-842
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-842
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: tasktracker
Reporter: Arun C Murthy
Assignee: Vinod K V
 Attachments: HADOOP-4491-20090623-common.1.txt, 
 HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, 
 HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, 
 HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, 
 HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, 
 HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, 
 HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, 
 HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741683#action_12741683
 ] 

Amar Kamat commented on MAPREDUCE-805:
--

bq. Note that I purposefully added sleeps in JobTracker.initJob() and 
JobInProgress.initTasks to take care of race conditions.
I meant during testing, my bad.

 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, 
 MAPREDUCE-805-v1.11-branch-0.20.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-842) Per-job local data on the TaskTracker node should have right access-control


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V updated MAPREDUCE-842:


Attachment: HADOOP-4491-20090811.txt

Updated patch fixing the testcase problem and findbugs warnings.

 Per-job local data on the TaskTracker node should have right access-control
 ---

 Key: MAPREDUCE-842
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-842
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: tasktracker
Reporter: Arun C Murthy
Assignee: Vinod K V
 Attachments: HADOOP-4491-20090623-common.1.txt, 
 HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, 
 HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, 
 HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, 
 HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, 
 HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, 
 HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, 
 HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt, 
 HADOOP-4491-20090811.txt




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-355) Change org.apache.hadoop.mapred.join to use new api


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-355:
--

Status: Open  (was: Patch Available)

patch doesnt apply to trunk

 Change org.apache.hadoop.mapred.join to use new api
 ---

 Key: MAPREDUCE-355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-355
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.21.0

 Attachments: patch-355-1.txt, patch-355-2.txt, patch-355.txt


 To change org.apache.hadoop.examples.Join to use new api, we need to change 
 org.apache.hadoop.mapred.join to use new api. So,
 Deprecate the code in org.apache.hadoop.mapred.join. 
 Copy the code to org.apache.hadoop.mapreduce.lib.join and Change it to use 
 new api. 
 Thoughts ?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-355) Change org.apache.hadoop.mapred.join to use new api


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated MAPREDUCE-355:
--

Status: Patch Available  (was: Open)

 Change org.apache.hadoop.mapred.join to use new api
 ---

 Key: MAPREDUCE-355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-355
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
 Fix For: 0.21.0

 Attachments: patch-355-1.txt, patch-355-2.txt, patch-355-3.txt, 
 patch-355.txt


 To change org.apache.hadoop.examples.Join to use new api, we need to change 
 org.apache.hadoop.mapred.join to use new api. So,
 Deprecate the code in org.apache.hadoop.mapred.join. 
 Copy the code to org.apache.hadoop.mapreduce.lib.join and Change it to use 
 new api. 
 Thoughts ?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-67) Few tasks failed while creating the work directory for a job, when job tracker was restarted


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu resolved MAPREDUCE-67.
--

Resolution: Invalid

That was a cluster configuration issue.

 Few tasks failed while creating the work directory for a job, when job 
 tracker was restarted
 

 Key: MAPREDUCE-67
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-67
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Hemanth Yamijala
Assignee: Amareshwari Sriramadasu

 A randomwriter job was running when the job tracker restarted. After the 
 jobtracker restarted, some tasktrackers were sent a reinit action. After 
 this, some new tasks of the random writer were scheduled to be run on the 
 same task trackers. These failed in the job localization while creating the 
 work directory. However, the next attempts of the same job ran successfully 
 and the job succeeded. This happened in about 1% of the total number of tasks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-805) Deadlock in Jobtracker


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12741697#action_12741697
 ] 

Vinod K V commented on MAPREDUCE-805:
-

This issue clashes with the changes in MAPREDUCE-802. The two issues should be 
coordinated.

 Deadlock in Jobtracker
 --

 Key: MAPREDUCE-805
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-805
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Michael Tamm
 Attachments: MAPREDUCE-805-v1.1.patch, 
 MAPREDUCE-805-v1.11-branch-0.20.patch, MAPREDUCE-805-v1.11.patch, 
 MAPREDUCE-805-v1.2.patch, MAPREDUCE-805-v1.3.patch, MAPREDUCE-805-v1.6.patch, 
 MAPREDUCE-805-v1.7.patch


 We are running a hadoop cluster (version 0.20.0) and have detected the 
 following deadlock on our jobtracker:
 {code}
 IPC Server handler 51 on 9001:
   at 
 org.apache.hadoop.mapred.JobInProgress.getCounters(JobInProgress.java:943)
   - waiting to lock 0x7f2b6fb46130 (a 
 org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobTracker.getJobCounters(JobTracker.java:3102)
   - locked 0x7f2b5f026000 (a org.apache.hadoop.mapred.JobTracker)
   at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
  pool-1-thread-2:
   at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2017)
   - waiting to lock 0x7f2b5f026000 (a 
 org.apache.hadoop.mapred.JobTracker)
   at 
 org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2483)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2152)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2169)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at org.apache.hadoop.mapred.JobInProgress.fail(JobInProgress.java:2245)
   - locked 0x7f2b6fb46130 (a org.apache.hadoop.mapred.JobInProgress)
   at 
 org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:86)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
 {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (MAPREDUCE-838) Task succeeds even when committer.commitTask fails with IOException