[jira] Updated: (MAPREDUCE-732) node health check script should not log UNHEALTHY status for every heartbeat in INFO mode

2009-07-09 Thread Hemanth Yamijala (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hemanth Yamijala updated MAPREDUCE-732:
---

Attachment: MAPREDUCE-732-2.patch

The attached patch file did not apply to trunk. I created a new one with the 
same fix.

 node health check script should not log UNHEALTHY status for every 
 heartbeat in INFO mode
 ---

 Key: MAPREDUCE-732
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-732
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Ramya R
Assignee: Sreekanth Ramakrishnan
Priority: Minor
 Fix For: 0.21.0

 Attachments: mapreduce-732-1.patch, MAPREDUCE-732-2.patch, 
 mapreduce-732.patch


 Currently, when a TT is blacklisted by the node health check script, for 
 every heartbeat a message such as the following is being logged.
 {noformat}
 date time INFO org.apache.hadoop.mapred.JobTracker: Adding blacklisted 
 reason for tracker : blacklisted TT Reason for blacklisting is : 
 NODE_UNHEALTHY
 {noformat}
 Due to this, the the JT logs fill up rapidly clogging the logdirs. Hence this 
 message should be logged in DEBUG mode instead of INFO mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-732) node health check script should not log UNHEALTHY status for every heartbeat in INFO mode

2009-07-09 Thread Hemanth Yamijala (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729136#action_12729136
 ] 

Hemanth Yamijala commented on MAPREDUCE-732:


Since this is a trivial fix and a change only to LOG messages, I am going to 
commit this. I ran TestTaskTrackerBlacklisting just to be sure.

 node health check script should not log UNHEALTHY status for every 
 heartbeat in INFO mode
 ---

 Key: MAPREDUCE-732
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-732
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Ramya R
Assignee: Sreekanth Ramakrishnan
Priority: Minor
 Fix For: 0.21.0

 Attachments: mapreduce-732-1.patch, MAPREDUCE-732-2.patch, 
 mapreduce-732.patch


 Currently, when a TT is blacklisted by the node health check script, for 
 every heartbeat a message such as the following is being logged.
 {noformat}
 date time INFO org.apache.hadoop.mapred.JobTracker: Adding blacklisted 
 reason for tracker : blacklisted TT Reason for blacklisting is : 
 NODE_UNHEALTHY
 {noformat}
 Due to this, the the JT logs fill up rapidly clogging the logdirs. Hence this 
 message should be logged in DEBUG mode instead of INFO mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-733) When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker heartbeat exception occurs.

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreekanth Ramakrishnan updated MAPREDUCE-733:
-

Attachment: MAPREDUCE-733-1.patch

Attaching patch adding new test case:

The patch will apply after the patch for MAPREDUCE-734 has been applied.

 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat exception occurs. 
 --

 Key: MAPREDUCE-733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 0.21.0
Reporter: Iyappan Srinivasan
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-733-1.patch, MAPREDUCE-733_0_20090708.patch, 
 MAPREDUCE-733_0_20090708_yhadoop20.patch


 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat. 
 It seems when a  task tracker is killed , it throws exception. Instead it 
 should catch it and process it and allow the rest of the flow to go through.
 2009-07-08 11:58:26,116 INFO  ipc.Server (Server.java:run(973)) - IPC Server 
 handler 7 on 40193, call 
 heartbeat(org.apache.hadoop.mapred.tasktrackersta...@13ec758, false, false, 
 true, 6) from 127.0.0.1:40200: error: java.io.IOException: 
 java.lang.RuntimeException: tracker_host1.rack.com:localhost/127.0.0.1:40197 
 already has slots reserved for null; being asked to un-reserve for 
 job_200907081158_0001
 java.io.IOException: java.lang.RuntimeException: 
 tracker_host1.rack.com:localhost/127.0.0.1:40197 already has slots reserved 
 for null; being asked to un-reserve for job_200907081158_0001
 at 
 org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker.unreserveSlots(TaskTracker.java:162)
 at 
 org.apache.hadoop.mapred.JobInProgress.addTrackerTaskFailure(JobInProgress.java:1580)
 at 
 org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2908)
 at 
 org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1025)
 at 
 org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3869)
 at 
 org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3081)
 at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2819)
 at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
 2009-07-08 11:58:26,162 INFO  mapred.TaskTracker 
 (TaskTracker.java:transmitHeartBeat(1196)) - Resending 'status' to 
 'localhost' with reponseId '6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-733) When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker heartbeat exception occurs.

2009-07-09 Thread Iyappan Srinivasan (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729147#action_12729147
 ] 

Iyappan Srinivasan commented on MAPREDUCE-733:
--

I ran TestTrackerBlacklistAcrossJobs and it passed and logs do not have 
java.io.IOException: java.lang.RuntimeException.  I also brought up a  
cluster, submitted jobs randomly and killed some task attempts and did not come 
across this string in jobtracker log. 

 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat exception occurs. 
 --

 Key: MAPREDUCE-733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 0.21.0
Reporter: Iyappan Srinivasan
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-733-1.patch, MAPREDUCE-733_0_20090708.patch, 
 MAPREDUCE-733_0_20090708_yhadoop20.patch


 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat. 
 It seems when a  task tracker is killed , it throws exception. Instead it 
 should catch it and process it and allow the rest of the flow to go through.
 2009-07-08 11:58:26,116 INFO  ipc.Server (Server.java:run(973)) - IPC Server 
 handler 7 on 40193, call 
 heartbeat(org.apache.hadoop.mapred.tasktrackersta...@13ec758, false, false, 
 true, 6) from 127.0.0.1:40200: error: java.io.IOException: 
 java.lang.RuntimeException: tracker_host1.rack.com:localhost/127.0.0.1:40197 
 already has slots reserved for null; being asked to un-reserve for 
 job_200907081158_0001
 java.io.IOException: java.lang.RuntimeException: 
 tracker_host1.rack.com:localhost/127.0.0.1:40197 already has slots reserved 
 for null; being asked to un-reserve for job_200907081158_0001
 at 
 org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker.unreserveSlots(TaskTracker.java:162)
 at 
 org.apache.hadoop.mapred.JobInProgress.addTrackerTaskFailure(JobInProgress.java:1580)
 at 
 org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2908)
 at 
 org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1025)
 at 
 org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3869)
 at 
 org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3081)
 at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2819)
 at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
 2009-07-08 11:58:26,162 INFO  mapred.TaskTracker 
 (TaskTracker.java:transmitHeartBeat(1196)) - Resending 'status' to 
 'localhost' with reponseId '6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-626) Modify TestLostTracker to improve execution time

2009-07-09 Thread Jothi Padmanabhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jothi Padmanabhan updated MAPREDUCE-626:


Status: Open  (was: Patch Available)

 Modify TestLostTracker to improve execution time
 

 Key: MAPREDUCE-626
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-626
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
Priority: Minor
 Attachments: mapred-626.patch, mapred-626.patch


 This test can be made faster with a few modifications

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-733) When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker heartbeat exception occurs.

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreekanth Ramakrishnan updated MAPREDUCE-733:
-

Attachment: MAPREDUCE-733-2.patch

Added some comments to the test case.

 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat exception occurs. 
 --

 Key: MAPREDUCE-733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 0.21.0
Reporter: Iyappan Srinivasan
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-733-1.patch, MAPREDUCE-733-2.patch, 
 MAPREDUCE-733_0_20090708.patch, MAPREDUCE-733_0_20090708_yhadoop20.patch


 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat. 
 It seems when a  task tracker is killed , it throws exception. Instead it 
 should catch it and process it and allow the rest of the flow to go through.
 2009-07-08 11:58:26,116 INFO  ipc.Server (Server.java:run(973)) - IPC Server 
 handler 7 on 40193, call 
 heartbeat(org.apache.hadoop.mapred.tasktrackersta...@13ec758, false, false, 
 true, 6) from 127.0.0.1:40200: error: java.io.IOException: 
 java.lang.RuntimeException: tracker_host1.rack.com:localhost/127.0.0.1:40197 
 already has slots reserved for null; being asked to un-reserve for 
 job_200907081158_0001
 java.io.IOException: java.lang.RuntimeException: 
 tracker_host1.rack.com:localhost/127.0.0.1:40197 already has slots reserved 
 for null; being asked to un-reserve for job_200907081158_0001
 at 
 org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker.unreserveSlots(TaskTracker.java:162)
 at 
 org.apache.hadoop.mapred.JobInProgress.addTrackerTaskFailure(JobInProgress.java:1580)
 at 
 org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2908)
 at 
 org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1025)
 at 
 org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3869)
 at 
 org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3081)
 at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2819)
 at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
 2009-07-08 11:58:26,162 INFO  mapred.TaskTracker 
 (TaskTracker.java:transmitHeartBeat(1196)) - Resending 'status' to 
 'localhost' with reponseId '6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881

2009-07-09 Thread rahul k singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rahul k singh updated MAPREDUCE-40:
---

Attachment: hadoop-5919-10.patch

 Memory management variables need a backwards compatibility option after 
 HADOOP-5881
 ---

 Key: MAPREDUCE-40
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-40
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Hemanth Yamijala
Assignee: rahul k singh
Priority: Blocker
 Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, 
 hadoop-5919-2.patch, hadoop-5919-3.patch, hadoop-5919-4.patch, 
 hadoop-5919-5.patch, hadoop-5919-6.patch, hadoop-5919-7.patch, 
 hadoop-5919-8.patch, hadoop-5919-9.patch


 HADOOP-5881 modified variables related to memory management without looking 
 at the backwards compatibility angle. This JIRA is to adress the gap. Marking 
 it a blocker for 0.20.1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881

2009-07-09 Thread rahul k singh (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729187#action_12729187
 ] 

rahul k singh commented on MAPREDUCE-40:


The new patch gives higher preference to old config settings. It takes 
precedence over the new settings.

 Memory management variables need a backwards compatibility option after 
 HADOOP-5881
 ---

 Key: MAPREDUCE-40
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-40
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Hemanth Yamijala
Assignee: rahul k singh
Priority: Blocker
 Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, 
 hadoop-5919-2.patch, hadoop-5919-3.patch, hadoop-5919-4.patch, 
 hadoop-5919-5.patch, hadoop-5919-6.patch, hadoop-5919-7.patch, 
 hadoop-5919-8.patch, hadoop-5919-9.patch


 HADOOP-5881 modified variables related to memory management without looking 
 at the backwards compatibility angle. This JIRA is to adress the gap. Marking 
 it a blocker for 0.20.1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-716) org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle

2009-07-09 Thread Enis Soztutar (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729203#action_12729203
 ] 

Enis Soztutar commented on MAPREDUCE-716:
-

Ok, after reading the mysql forum, setting fetch size to int min seems the only 
way until server supports fetch sizes. Yet, I am as surprised as the guy who 
asked the original question. 

bq. So are you saying that DBRR should be a top-level class? I don't have 
strong opinions about this. I can pull it up to top level easily enough. I will 
only do this on the trunk branch, not the 0.20 branch patch.
Yes, extending an inner static class from a top-level-class seems inelegant to 
me.  

+1 overall. 

 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
 -

 Key: MAPREDUCE-716
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-716
 Project: Hadoop Map/Reduce
  Issue Type: Bug
 Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
Reporter: evanand
Assignee: Aaron Kimball
 Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, 
 HADOOP-5482.trunk.patch, MAPREDUCE-716.2.branch20.patch, 
 MAPREDUCE-716.2.trunk.patch, MAPREDUCE-716.3.trunk.patch, 
 MAPREDUCE-716.4.branch20.patch, MAPREDUCE-716.4.trunk.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
 The out of the box implementation of the Hadoop is working properly with 
 mysql/hsqldb, but NOT with oracle.
 Reason is DBInputformat is implemented with mysql/hsqldb specific query 
 constructs like LIMIT, OFFSET.
 FIX:
 building a database provider specific logic based on the database 
 providername (which we can get using connection).
 I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-732) node health check script should not log UNHEALTHY status for every heartbeat in INFO mode

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreekanth Ramakrishnan updated MAPREDUCE-732:
-

Attachment: MAPRED-732-ydist.patch

Yahoo distribution patch for the issue.

 node health check script should not log UNHEALTHY status for every 
 heartbeat in INFO mode
 ---

 Key: MAPREDUCE-732
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-732
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Ramya R
Assignee: Sreekanth Ramakrishnan
Priority: Minor
 Fix For: 0.21.0

 Attachments: MAPRED-732-ydist.patch, mapreduce-732-1.patch, 
 MAPREDUCE-732-2.patch, mapreduce-732.patch


 Currently, when a TT is blacklisted by the node health check script, for 
 every heartbeat a message such as the following is being logged.
 {noformat}
 date time INFO org.apache.hadoop.mapred.JobTracker: Adding blacklisted 
 reason for tracker : blacklisted TT Reason for blacklisting is : 
 NODE_UNHEALTHY
 {noformat}
 Due to this, the the JT logs fill up rapidly clogging the logdirs. Hence this 
 message should be logged in DEBUG mode instead of INFO mode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-699) Several streaming test cases seem to be failing

2009-07-09 Thread Enis Soztutar (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729205#action_12729205
 ] 

Enis Soztutar commented on MAPREDUCE-699:
-

All of the recent patch builds fail on hudson. 7 tests for streaming fail with 
NPE. Examples :
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/366/
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/365/

 Several streaming test cases seem to be failing
 ---

 Key: MAPREDUCE-699
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-699
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Jothi Padmanabhan

 ant test is failing several streaming tests with the following error
 Error Message
 java.lang.NullPointerException  at 
 org.apache.commons.cli.GnuParser.flatten(GnuParser.java:110)  at 
 org.apache.commons.cli.Parser.parse(Parser.java:143)  at 
 org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:374)
   at 
 org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:153)
   at 
 org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:138)
   at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1314)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:414) 
  at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:278)  at 
 org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:119)  at 
 org.apache.hadoop.streaming.TestMultipleCachefiles.testMultipleCachefiles(TestMultipleCachefiles.java:68)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
  at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)  at 
 junit.framework.TestCase.runTest(TestCase.java:168)  at 
 junit.framework.TestCase.runBare(TestCase.java:134)  at 
 junit.framework.TestResult$1.protect(TestResult.java:110)  at 
 junit.framework.TestResult.runProtected(TestResult.java:128)  at 
 junit.framework.TestResult.run(TestResult.java:113)  at 
 junit.framework.TestCase.run(TestCase.java:124)  at 
 junit.framework.TestSuite.runTest(TestSuite.java:232)  at 
 junit.framework.TestSuite.run(TestSuite.java:227)  at 
 org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79) 
  at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)  at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
  
 Stacktrace
 junit.framework.AssertionFailedError: java.lang.NullPointerException
   at org.apache.commons.cli.GnuParser.flatten(GnuParser.java:110)
   at org.apache.commons.cli.Parser.parse(Parser.java:143)
   at 
 org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:374)
   at 
 org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:153)
   at 
 org.apache.hadoop.util.GenericOptionsParser.init(GenericOptionsParser.java:138)
   at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1314)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:414)
   at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:278)
   at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:119)
   at 
 org.apache.hadoop.streaming.TestMultipleCachefiles.testMultipleCachefiles(TestMultipleCachefiles.java:68)
   at 
 org.apache.hadoop.streaming.TestMultipleCachefiles.failTrace(TestMultipleCachefiles.java:141)
   at 
 org.apache.hadoop.streaming.TestMultipleCachefiles.testMultipleCachefiles(TestMultipleCachefiles.java:133)
 The following are links to two such failures
 http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/337/testReport/
 http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/336/testReport/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-677) TestNodeRefresh timesout

2009-07-09 Thread Amar Kamat (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729206#action_12729206
 ] 

Amar Kamat commented on MAPREDUCE-677:
--

Result of test-patch
[exec] +1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 3 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.


 TestNodeRefresh timesout
 

 Key: MAPREDUCE-677
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-677
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Amar Kamat
Assignee: Amar Kamat
 Attachments: MAPREDUCE-677-v1.0.patch, MAPREDUCE-677-v1.1.patch




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Assigned: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce

2009-07-09 Thread Vinod K V (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V reassigned MAPREDUCE-711:
---

Assignee: Vinod K V

 Move Distributed Cache from Common to Map/Reduce
 

 Key: MAPREDUCE-711
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-711
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Vinod K V

 Distributed Cache logically belongs as part of map/reduce and not Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-734) java.util.ConcurrentModificationException observed in unreserving slots for HiRam Jobs

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729209#action_12729209
 ] 

Sreekanth Ramakrishnan commented on MAPREDUCE-734:
--

All tests passed locally. Output from ant test-patch

{noformat}
 [exec]
 [exec] +1 overall.
 [exec]
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec]
 [exec] +1 tests included.  The patch appears to include 3 new or 
modified tests.
 [exec]
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec]
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec]
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec]
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.
 [exec]
 [exec]
 [exec]
{noformat}

 java.util.ConcurrentModificationException observed in unreserving slots for 
 HiRam Jobs
 --

 Key: MAPREDUCE-734
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-734
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched
Affects Versions: 0.21.0
Reporter: Karam Singh
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-734-1.patch, MAPREDUCE-734-2.patch, 
 MAPREDUCE-734-ydist.patch, MAPREDUCE-734_0_20090708.patch, 
 MAPREDUCE-734_0_20090708_yhadoop20.patch


 Ran jobs out which 3 were HiRAM, the job were not removed from scheduler 
 queue even after they successfully completed
 hadoop queue -info queue -showJobs displays somwthing like -:
 job_200907080724_0031   2   1247059146868   username  NORMAL  0 running 
 map tasks using 0 map slots. 0 additional slots reserved. 0 running reduce 
 tasks using 0 reduce slots. 60 additional slots reserved.
 job_200907080724_0030   2   1247059146972   username  NORMAL  0 running 
 map tasks using 0 map slots. 0 additional slots reserved. 0 running reduce 
 tasks using 0 reduce slots. 60 additional slots reserved.
 But it does not block anything, but seems like zombie process of system
 Jobtracker log show java.util.ConcurrentModificationException

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-626) Modify TestLostTracker to improve execution time

2009-07-09 Thread Jothi Padmanabhan (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729211#action_12729211
 ] 

Jothi Padmanabhan commented on MAPREDUCE-626:
-

Test patch results:

 [exec] +1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 9 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.


 Modify TestLostTracker to improve execution time
 

 Key: MAPREDUCE-626
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-626
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
Priority: Minor
 Attachments: mapred-626-v1.patch, mapred-626.patch, mapred-626.patch


 This test can be made faster with a few modifications

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-626) Modify TestLostTracker to improve execution time

2009-07-09 Thread Jothi Padmanabhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jothi Padmanabhan updated MAPREDUCE-626:


Status: Patch Available  (was: Open)

 Modify TestLostTracker to improve execution time
 

 Key: MAPREDUCE-626
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-626
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
Priority: Minor
 Attachments: mapred-626-v1.patch, mapred-626.patch, mapred-626.patch


 This test can be made faster with a few modifications

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce

2009-07-09 Thread Vinod K V (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod K V updated MAPREDUCE-711:


Attachment: MAPREDUCE-711-20090709-mapreduce.txt
MAPREDUCE-711-20090709-common.txt

I am taking this up; this is need for other DistributedCache related issues - 
HADOOP-4493 and MAPREDUCE-476.

Attached are two patches, one for common project and one for mapreduce project. 
The patches are generated by simple refactoring in eclipse across projects. 
I've created new package org.apache.hadoop.mapred.filecache in both src/java as 
well as src/test/mapred.

I have run the tests that are directly affected by these patches, and they 
pass. Will run all the common and mapreduce tests in the background.

Few questions:
 - Do I need a separate jira issue for common part, given It is just moving of 
files across projects? Can open a new issue if felt otherwise.
 - What happens to the repository history of these files? Is there any possible 
way we can move repo history too?

 Move Distributed Cache from Common to Map/Reduce
 

 Key: MAPREDUCE-711
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-711
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Vinod K V
 Attachments: MAPREDUCE-711-20090709-common.txt, 
 MAPREDUCE-711-20090709-mapreduce.txt


 Distributed Cache logically belongs as part of map/reduce and not Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-685) Sqoop will fail with OutOfMemory on large tables using mysql

2009-07-09 Thread Tom White (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated MAPREDUCE-685:


   Resolution: Fixed
Fix Version/s: 0.21.0
 Hadoop Flags: [Reviewed]
   Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Aaron!

 Sqoop will fail with OutOfMemory on large tables using mysql
 

 Key: MAPREDUCE-685
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-685
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Fix For: 0.21.0

 Attachments: MAPREDUCE-685.3.patch, MAPREDUCE-685.patch, 
 MAPREDUCE-685.patch.2


 The default MySQL JDBC client behavior is to buffer the entire ResultSet in 
 the client before allowing the user to use the ResultSet object. On large 
 SELECTs, this can cause OutOfMemory exceptions, even when the client intends 
 to close the ResultSet after reading only a few rows. The MySQL ConnManager 
 should configure its connection to use row-at-a-time delivery of results to 
 the client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Resolved: (MAPREDUCE-734) java.util.ConcurrentModificationException observed in unreserving slots for HiRam Jobs

2009-07-09 Thread Hemanth Yamijala (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hemanth Yamijala resolved MAPREDUCE-734.


  Resolution: Fixed
Hadoop Flags: [Reviewed]

I just committed this. Thanks, Arun and Sreekanth !

 java.util.ConcurrentModificationException observed in unreserving slots for 
 HiRam Jobs
 --

 Key: MAPREDUCE-734
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-734
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/capacity-sched
Affects Versions: 0.21.0
Reporter: Karam Singh
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-734-1.patch, MAPREDUCE-734-2.patch, 
 MAPREDUCE-734-ydist.patch, MAPREDUCE-734_0_20090708.patch, 
 MAPREDUCE-734_0_20090708_yhadoop20.patch


 Ran jobs out which 3 were HiRAM, the job were not removed from scheduler 
 queue even after they successfully completed
 hadoop queue -info queue -showJobs displays somwthing like -:
 job_200907080724_0031   2   1247059146868   username  NORMAL  0 running 
 map tasks using 0 map slots. 0 additional slots reserved. 0 running reduce 
 tasks using 0 reduce slots. 60 additional slots reserved.
 job_200907080724_0030   2   1247059146972   username  NORMAL  0 running 
 map tasks using 0 map slots. 0 additional slots reserved. 0 running reduce 
 tasks using 0 reduce slots. 60 additional slots reserved.
 But it does not block anything, but seems like zombie process of system
 Jobtracker log show java.util.ConcurrentModificationException

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce

2009-07-09 Thread Tom White (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729236#action_12729236
 ] 

Tom White commented on MAPREDUCE-711:
-

We shouldn't just repackage DistributedCache without deprecating it first, 
since it is a public interface. For this Jira, it might be better to move it to 
the MapReduce project while keeping it in the same package 
(org.apache.hadoop.filecache), since there are other Jiras to evolve its 
interface (MAPREDUCE-476, MAPREDUCE-303). Moving it to a new package 
(org.apache.hadoop.mapreduce.distcache?) could happen in one of those.

 Move Distributed Cache from Common to Map/Reduce
 

 Key: MAPREDUCE-711
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-711
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Vinod K V
 Attachments: MAPREDUCE-711-20090709-common.txt, 
 MAPREDUCE-711-20090709-mapreduce.txt


 Distributed Cache logically belongs as part of map/reduce and not Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-733) When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker heartbeat exception occurs.

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreekanth Ramakrishnan updated MAPREDUCE-733:
-

Attachment: MAPREDUCE-733-3.patch

Fixing a bug in previous patch.

* Added new test case to test issue.

 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat exception occurs. 
 --

 Key: MAPREDUCE-733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 0.21.0
Reporter: Iyappan Srinivasan
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-733-1.patch, MAPREDUCE-733-2.patch, 
 MAPREDUCE-733-3.patch, MAPREDUCE-733_0_20090708.patch, 
 MAPREDUCE-733_0_20090708_yhadoop20.patch


 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat. 
 It seems when a  task tracker is killed , it throws exception. Instead it 
 should catch it and process it and allow the rest of the flow to go through.
 2009-07-08 11:58:26,116 INFO  ipc.Server (Server.java:run(973)) - IPC Server 
 handler 7 on 40193, call 
 heartbeat(org.apache.hadoop.mapred.tasktrackersta...@13ec758, false, false, 
 true, 6) from 127.0.0.1:40200: error: java.io.IOException: 
 java.lang.RuntimeException: tracker_host1.rack.com:localhost/127.0.0.1:40197 
 already has slots reserved for null; being asked to un-reserve for 
 job_200907081158_0001
 java.io.IOException: java.lang.RuntimeException: 
 tracker_host1.rack.com:localhost/127.0.0.1:40197 already has slots reserved 
 for null; being asked to un-reserve for job_200907081158_0001
 at 
 org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker.unreserveSlots(TaskTracker.java:162)
 at 
 org.apache.hadoop.mapred.JobInProgress.addTrackerTaskFailure(JobInProgress.java:1580)
 at 
 org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2908)
 at 
 org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1025)
 at 
 org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3869)
 at 
 org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3081)
 at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2819)
 at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
 2009-07-08 11:58:26,162 INFO  mapred.TaskTracker 
 (TaskTracker.java:transmitHeartBeat(1196)) - Resending 'status' to 
 'localhost' with reponseId '6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881

2009-07-09 Thread rahul k singh (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rahul k singh updated MAPREDUCE-40:
---

Attachment: hadoop-5919-11.patch

Handled some corner cases.

 Memory management variables need a backwards compatibility option after 
 HADOOP-5881
 ---

 Key: MAPREDUCE-40
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-40
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Hemanth Yamijala
Assignee: rahul k singh
Priority: Blocker
 Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, 
 hadoop-5919-11.patch, hadoop-5919-2.patch, hadoop-5919-3.patch, 
 hadoop-5919-4.patch, hadoop-5919-5.patch, hadoop-5919-6.patch, 
 hadoop-5919-7.patch, hadoop-5919-8.patch, hadoop-5919-9.patch


 HADOOP-5881 modified variables related to memory management without looking 
 at the backwards compatibility angle. This JIRA is to adress the gap. Marking 
 it a blocker for 0.20.1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-733) When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker heartbeat exception occurs.

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreekanth Ramakrishnan updated MAPREDUCE-733:
-

Attachment: MAPREDUCE-733-5.patch

Latest patch with removing log statements.

 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat exception occurs. 
 --

 Key: MAPREDUCE-733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 0.21.0
Reporter: Iyappan Srinivasan
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-733-1.patch, MAPREDUCE-733-2.patch, 
 MAPREDUCE-733-3.patch, MAPREDUCE-733-4.patch, MAPREDUCE-733-5.patch, 
 MAPREDUCE-733_0_20090708.patch, MAPREDUCE-733_0_20090708_yhadoop20.patch


 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat. 
 It seems when a  task tracker is killed , it throws exception. Instead it 
 should catch it and process it and allow the rest of the flow to go through.
 2009-07-08 11:58:26,116 INFO  ipc.Server (Server.java:run(973)) - IPC Server 
 handler 7 on 40193, call 
 heartbeat(org.apache.hadoop.mapred.tasktrackersta...@13ec758, false, false, 
 true, 6) from 127.0.0.1:40200: error: java.io.IOException: 
 java.lang.RuntimeException: tracker_host1.rack.com:localhost/127.0.0.1:40197 
 already has slots reserved for null; being asked to un-reserve for 
 job_200907081158_0001
 java.io.IOException: java.lang.RuntimeException: 
 tracker_host1.rack.com:localhost/127.0.0.1:40197 already has slots reserved 
 for null; being asked to un-reserve for job_200907081158_0001
 at 
 org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker.unreserveSlots(TaskTracker.java:162)
 at 
 org.apache.hadoop.mapred.JobInProgress.addTrackerTaskFailure(JobInProgress.java:1580)
 at 
 org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2908)
 at 
 org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1025)
 at 
 org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3869)
 at 
 org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3081)
 at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2819)
 at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
 2009-07-08 11:58:26,162 INFO  mapred.TaskTracker 
 (TaskTracker.java:transmitHeartBeat(1196)) - Resending 'status' to 
 'localhost' with reponseId '6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-733) When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker heartbeat exception occurs.

2009-07-09 Thread Sreekanth Ramakrishnan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreekanth Ramakrishnan updated MAPREDUCE-733:
-

Attachment: MAPREDUCE-733-ydist.patch

Internal Y! distribution patch

 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat exception occurs. 
 --

 Key: MAPREDUCE-733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 0.21.0
Reporter: Iyappan Srinivasan
Assignee: Arun C Murthy
 Fix For: 0.21.0

 Attachments: MAPREDUCE-733-1.patch, MAPREDUCE-733-2.patch, 
 MAPREDUCE-733-3.patch, MAPREDUCE-733-4.patch, MAPREDUCE-733-5.patch, 
 MAPREDUCE-733-ydist.patch, MAPREDUCE-733_0_20090708.patch, 
 MAPREDUCE-733_0_20090708_yhadoop20.patch


 When running ant test TestTrackerBlacklistAcrossJobs, losing task tracker 
 heartbeat. 
 It seems when a  task tracker is killed , it throws exception. Instead it 
 should catch it and process it and allow the rest of the flow to go through.
 2009-07-08 11:58:26,116 INFO  ipc.Server (Server.java:run(973)) - IPC Server 
 handler 7 on 40193, call 
 heartbeat(org.apache.hadoop.mapred.tasktrackersta...@13ec758, false, false, 
 true, 6) from 127.0.0.1:40200: error: java.io.IOException: 
 java.lang.RuntimeException: tracker_host1.rack.com:localhost/127.0.0.1:40197 
 already has slots reserved for null; being asked to un-reserve for 
 job_200907081158_0001
 java.io.IOException: java.lang.RuntimeException: 
 tracker_host1.rack.com:localhost/127.0.0.1:40197 already has slots reserved 
 for null; being asked to un-reserve for job_200907081158_0001
 at 
 org.apache.hadoop.mapreduce.server.jobtracker.TaskTracker.unreserveSlots(TaskTracker.java:162)
 at 
 org.apache.hadoop.mapred.JobInProgress.addTrackerTaskFailure(JobInProgress.java:1580)
 at 
 org.apache.hadoop.mapred.JobInProgress.failedTask(JobInProgress.java:2908)
 at 
 org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1025)
 at 
 org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3869)
 at 
 org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3081)
 at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2819)
 at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
 2009-07-08 11:58:26,162 INFO  mapred.TaskTracker 
 (TaskTracker.java:transmitHeartBeat(1196)) - Resending 'status' to 
 'localhost' with reponseId '6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-11) Cleanup JobHistory file naming to do with job recovery

2009-07-09 Thread Amar Kamat (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-11?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat updated MAPREDUCE-11:


Attachment: MAPREDUCE-11-v1.8.patch

Attaching a patch that simplifies the job history filename and recovery. 
Changes are as follows :
# job history filename is of the format _hostname_jobid_username_jobname_
# conf filenames are of the format _hostname_jobid_conf.xml_
# upon every restart all the new updates will be directed to 
_history-file.recover_
# once the job finishes the _history-file.recover_ file will be renamed to 
_history-file_
# note that the master file ( _hostname_jobid_username_jobname_) will exist 
throughout the lifecycle of the job
# if the jobtracker restart again, new updates will be lost
# there is no searching involved in any case
# for now the old jobhistory files are supported via web-ui

Tested the patch locally and so far no issues. Result of test-patch 
[exec] +1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 9 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
 [exec] 
 [exec] +1 release audit.  The applied patch does not increase the 
total number of release audit warnings.


Running ant tests now and testing in progress.

Things I tested
# submitted a job allowed it to completed. New job files move to done folder. 
# submitted a job and killed the jobtracker  while job files was empty, 
restarted the jobtracker and the files upon completion move to done folder
# submitted a job and killed the jobtracker  while job files was written, 
restarted the jobtracker and the files upon completion move to done folder. job 
was also recovered
# checked webui
 ## history shows old and new files (there is no difference between the layout)
 ## history pages for old and new jobs have functional links (check random 
links and conf links)
 ## search facility in history works across files 

 Cleanup JobHistory file naming to do with job recovery
 --

 Key: MAPREDUCE-11
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-11
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Devaraj Das
 Attachments: MAPREDUCE-11-v1.8.patch


 The JobTracker uses the job history files for doing job recovery upon 
 startup. To handle cases where JobTracker goes down again while the recovered 
 job is running, there is some logic that plays with files and it ends up 
 having two history files for some window of time during the life of the job - 
 actual history file, .recover file. The idea being that upon the next restart 
 we should be able to the maximal number of events for the job. It led to 
 performance problems in the job submission / recovery (part of which got 
 addressed in HADOOP-4372). It also looks pretty unlikely that a running job 
 will traverse across multiple JT restarts. Even if it did, without the 
 .recover file, it'd only mean that we lose some tasks that got completed in a 
 subsequent restart. I propose that we remove the .recover file logic and base 
 the recovery on only the original job history file. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner

2009-07-09 Thread Aaron Kimball (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Kimball updated MAPREDUCE-710:


Status: Patch Available  (was: Open)

 Sqoop should read and transmit passwords in a more secure manner
 

 Key: MAPREDUCE-710
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-710
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.patch


 Sqoop's current support for passwords involves reading passwords from the 
 command line --password foo, which makes the password visible to other 
 users via 'ps'. An invisible-console approach should be taken.
 Related, Sqoop transmits passwords to mysqldump in the same fashion, which is 
 also insecure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner

2009-07-09 Thread Aaron Kimball (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Kimball updated MAPREDUCE-710:


Attachment: MAPREDUCE-710.2.patch

Attaching rebased patch after MAPREDUCE-685, 713.

 Sqoop should read and transmit passwords in a more secure manner
 

 Key: MAPREDUCE-710
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-710
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.patch


 Sqoop's current support for passwords involves reading passwords from the 
 command line --password foo, which makes the password visible to other 
 users via 'ps'. An invisible-console approach should be taken.
 Related, Sqoop transmits passwords to mysqldump in the same fashion, which is 
 also insecure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-742) Improv e the java comments for the π examples

2009-07-09 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated MAPREDUCE-742:
-

Affects Version/s: 0.21.0
Fix Version/s: 0.21.0
  Summary: Improve the java comments for the π examples  (was: 
Improve the java comments for the pi; examples)

 Improve the java comments for the π examples
 

 Key: MAPREDUCE-742
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-742
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation, examples
Affects Versions: 0.21.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
 Fix For: 0.21.0

 Attachments: m742_20090708.patch


 There are 3 examples, pi, bbp and distbbp for pi; computation.  We should 
 tell the difference between them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-705) User-configurable quote and delimiter characters for Sqoop records and record reparsing

2009-07-09 Thread Aaron Kimball (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Kimball updated MAPREDUCE-705:


Attachment: MAPREDUCE-705.2.patch

Attaching rebased patch after MAPREDUCE-685 

 User-configurable quote and delimiter characters for Sqoop records and record 
 reparsing
 ---

 Key: MAPREDUCE-705
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-705
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Attachments: MAPREDUCE-705.2.patch, MAPREDUCE-705.patch


 Sqoop needs a mechanism for users to govern how fields are quoted and what 
 delimiter characters separate fields and records. With delimiters providing 
 an unambiguous format, a parse method can reconstitute the generated record 
 data object from a text-based representation of the same record.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-421) mapred pipes might return exit code 0 even when failing

2009-07-09 Thread Christian Kunz (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kunz updated MAPREDUCE-421:
-

Attachment: MAPREDUCE-421.patch

 mapred pipes might return exit code 0 even when failing
 ---

 Key: MAPREDUCE-421
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-421
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Reporter: Christian Kunz
Assignee: Christian Kunz
 Fix For: 0.20.1

 Attachments: MAPREDUCE-421.patch


 up to  hadoop 0.18.3 org.apache.hadoop.mapred.JobShell ensured that 'hadoop 
 jar' returns non-zero exit code when the job fails.
 This is no longer true after moving this to org.apache.hadoop.util.RunJar.
 Pipes jobs submitted through cli never returned proper exit code.
 The main methods in org.apache.hadoop.util.RunJar. and 
 org.apache.hadoop.mapred.pipes.Submitter should be modified to return an exit 
 code similar to how org.apache.hadoop.mapred.JobShell did it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-421) mapred pipes might return exit code 0 even when failing

2009-07-09 Thread Christian Kunz (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kunz updated MAPREDUCE-421:
-

Fix Version/s: 0.20.1
   Status: Patch Available  (was: Open)

 mapred pipes might return exit code 0 even when failing
 ---

 Key: MAPREDUCE-421
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-421
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Reporter: Christian Kunz
Assignee: Christian Kunz
 Fix For: 0.20.1

 Attachments: MAPREDUCE-421.patch


 up to  hadoop 0.18.3 org.apache.hadoop.mapred.JobShell ensured that 'hadoop 
 jar' returns non-zero exit code when the job fails.
 This is no longer true after moving this to org.apache.hadoop.util.RunJar.
 Pipes jobs submitted through cli never returned proper exit code.
 The main methods in org.apache.hadoop.util.RunJar. and 
 org.apache.hadoop.mapred.pipes.Submitter should be modified to return an exit 
 code similar to how org.apache.hadoop.mapred.JobShell did it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-476) extend DistributedCache to work locally (LocalJobRunner)

2009-07-09 Thread Philip Zeyliger (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Zeyliger updated MAPREDUCE-476:
--

Attachment: MAPREDUCE-476-v2.patch

In light of MAPREDUCE-711, I generated a new patch.  I applied 
MAPREDUCE-711-20090709-mapreduce.1.txt first, so this shouldn't be submitted to 
Hudson until after that gets checked in.  I generated the patch by applying

bq. cat HADOOP-2914-v3.patch | sed -e 's%src/core/%src/java/%g' | sed -e 
's%src/mapred/%src/java/%g' | sed -e 's%src/test/core%src/test/mapred%g' | 
patch -p0

I had to clean up DistributedCache.java a tiny bit (there were 2 rejects) 
because some Javadoc links were removed in the project move; I've reinstated 
them.  (I think they were removed because they pointed to MR from Common, but 
that's no longer an issue with MAPREDUCE-711.)


 extend DistributedCache to work locally (LocalJobRunner)
 

 Key: MAPREDUCE-476
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-476
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: sam rash
Assignee: Philip Zeyliger
Priority: Minor
 Attachments: HADOOP-2914-v1-full.patch, 
 HADOOP-2914-v1-since-4041.patch, HADOOP-2914-v2.patch, HADOOP-2914-v3.patch, 
 MAPREDUCE-476-v2.patch, MAPREDUCE-476.patch


 The DistributedCache does not work locally when using the outlined recipe at 
 http://hadoop.apache.org/core/docs/r0.16.0/api/org/apache/hadoop/filecache/DistributedCache.html
  
 Ideally, LocalJobRunner would take care of populating the JobConf and copying 
 remote files to the local file sytem (http, assume hdfs = default fs = local 
 fs when doing local development.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-740) Provide summary information per job once a job is finished.

2009-07-09 Thread Hong Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729400#action_12729400
 ] 

Hong Tang commented on MAPREDUCE-740:
-

@vinod

I do have a specific usage case where we want to keep track of the amount of 
resources being used by each job, each user, or each queue (for capacity 
scheduler). Granted, all these information is readily available in job history 
log. However, there are a few drawbacks by depending on job history logs: (1) 
we are interested in keeping a history of finished and possibly do group-by for 
user and queue. so scrapping individual history log is messy; (2) the added 
dependency to keep up with possible future changes to the history log format.

For starter, I think the summary should include the following information: 
- job queuing/waiting time
- job start time
- job finish time
- total maps/reduces
- user id
- job id (job-tracker ID + job sequence number)
- map/reduce slot hours (need to apply multiplier for high ram tasks 
that take multiple slots per map/reduce task)
- queue name
- job status (success or failure)
- cluster map/reduce slot capacity

The only thing that job history log does not provide currently is the slot 
hours for all maps and reduces belonging to the same job.

 Provide summary information per job once a job is finished.
 ---

 Key: MAPREDUCE-740
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-740
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Hong Tang
Priority: Minor

 It would be nice if JobTracker can output a one line summary information per 
 job once a job is finished. Otherwise, users or system administrators would 
 end up scraping individual job history logs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-717) Fix some corner case issues in speculative execution (post hadoop-2141)

2009-07-09 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729413#action_12729413
 ] 

Hadoop QA commented on MAPREDUCE-717:
-

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412839/717.patch
  against trunk revision 792613.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

-1 contrib tests.  The patch failed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/368/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/368/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/368/artifact/trunk/build/test/checkstyle-errors.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/368/console

This message is automatically generated.

 Fix some corner case issues in speculative execution (post hadoop-2141)
 ---

 Key: MAPREDUCE-717
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-717
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.21.0
Reporter: Devaraj Das
Assignee: Devaraj Das
 Fix For: 0.21.0

 Attachments: 717.patch


 Some corner case issues can be fixed:
 1) Setup task should not add anything to the job statistics (since they are 
 really fast and might affect the statistics of a job with few tasks)
 2) The statistics computations should be guarded for cases where things like 
 sumOfSquares could become less than zero (due to rounding errors mostly).
 3) The method TaskInProgress.getCurrentProgressRate() should take into 
 account the COMMIT_PENDING state
 4) The testcase TestSpeculativeExecution.testTaskLATEScheduling could be made 
 more robust

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-742) Improv e the java comments for the π examples

2009-07-09 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated MAPREDUCE-742:
-

Attachment: m742_20090709.patch

m742_20090709.patch: slightly changed the comments.

 Improve the java comments for the π examples
 

 Key: MAPREDUCE-742
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-742
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation, examples
Affects Versions: 0.21.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
 Fix For: 0.21.0

 Attachments: m742_20090708.patch, m742_20090709.patch


 There are 3 examples, pi, bbp and distbbp for pi; computation.  We should 
 tell the difference between them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-740) Provide summary information per job once a job is finished.

2009-07-09 Thread Hong Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729455#action_12729455
 ] 

Hong Tang commented on MAPREDUCE-740:
-

Additionally:
- We should summarize the information in one line in an easy-to-parse format, 
eg comma separated key=value list.
- We should also specify the number of map slots and reduce slots taken by each 
map task and reduce task.
- We may want to use a distinctive appender so that the administrator may 
choose to redirect the output of the summary info.
- The cluster wide capacity of map slots and reduce slots change over time. For 
now, let's simplify the definition as the map/reduce slot capacity by the time 
the job finishes.

 Provide summary information per job once a job is finished.
 ---

 Key: MAPREDUCE-740
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-740
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Hong Tang
Priority: Minor

 It would be nice if JobTracker can output a one line summary information per 
 job once a job is finished. Otherwise, users or system administrators would 
 end up scraping individual job history logs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-680) Reuse of Writable objects is improperly handled by MRUnit

2009-07-09 Thread Aaron Kimball (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729465#action_12729465
 ] 

Aaron Kimball commented on MAPREDUCE-680:
-

The contrib failures are in streaming, not MRUnit.

 Reuse of Writable objects is improperly handled by MRUnit
 -

 Key: MAPREDUCE-680
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-680
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Attachments: MAPREDUCE-680.patch


 As written, MRUnit's MockOutputCollector simply stores references to the 
 objects passed in to its collect() method. Thus if the same Text (or other 
 Writable) object is reused as an output containiner multiple times with 
 different values, these separate values will not all be collected. 
 MockOutputCollector needs to properly use io.serializations to deep copy the 
 objects sent in.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-740) Provide summary information per job once a job is finished.

2009-07-09 Thread Hong Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729484#action_12729484
 ] 

Hong Tang commented on MAPREDUCE-740:
-

bq. can we report queued/submit time instead?
Submit time should be enough. Waiting time is just launch time - submit time.

 Provide summary information per job once a job is finished.
 ---

 Key: MAPREDUCE-740
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-740
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Hong Tang
Priority: Minor

 It would be nice if JobTracker can output a one line summary information per 
 job once a job is finished. Otherwise, users or system administrators would 
 end up scraping individual job history logs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-18) Under load the shuffle sometimes gets incorrect data

2009-07-09 Thread Ravi Gummadi (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-18:
--

Attachment: MR-18.patch

Attaching patch for trunk.
Please review and provide your comments.

 Under load the shuffle sometimes gets incorrect data
 

 Key: MAPREDUCE-18
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-18
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Ravi Gummadi
Priority: Blocker
 Attachments: MR-18.patch


 While testing HADOOP-5223 under load, we found reduces receiving completely 
 incorrect data. It was often random, but sometimes was the output of the 
 wrong map for the wrong map. It appears to either be a Jetty or JVM bug, but 
 it is clearly happening on the server side. In the HADOOP-5223 code, I added 
 information about the map and reduce that were included and we should add 
 similar protection to 0.20 and trunk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (MAPREDUCE-741) New Hadoop MapReduce Site

2009-07-09 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12729512#action_12729512
 ] 

Hadoop QA commented on MAPREDUCE-741:
-

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412939/mapreduce-741.patch
  against trunk revision 792704.

+1 @author.  The patch does not contain any @author tags.

+0 tests included.  The patch appears to be a documentation patch that 
doesn't require tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/371/console

This message is automatically generated.

 New Hadoop MapReduce Site
 -

 Key: MAPREDUCE-741
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-741
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: documentation
Reporter: Corinne Chandel
 Attachments: mapreduce-741.patch


 New Hadoop MapReduce Site
 Set up site (initial pass).
 May need to add more content.
 May need to update some links.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Updated: (MAPREDUCE-630) TestKillCompletedJob can be modified to improve execution times

2009-07-09 Thread Jothi Padmanabhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jothi Padmanabhan updated MAPREDUCE-630:


Status: Open  (was: Patch Available)

 TestKillCompletedJob can be modified to improve execution times
 ---

 Key: MAPREDUCE-630
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-630
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
Priority: Minor
 Attachments: hadoop-6068.patch, mapred-630.patch


 This test can be easily made into a unit test

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Resolved: (MAPREDUCE-677) TestNodeRefresh timesout

2009-07-09 Thread Sharad Agarwal (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharad Agarwal resolved MAPREDUCE-677.
--

   Resolution: Fixed
Fix Version/s: 0.21.0
 Hadoop Flags: [Reviewed]

I committed this. Thanks Amar!

 TestNodeRefresh timesout
 

 Key: MAPREDUCE-677
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-677
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Amar Kamat
Assignee: Amar Kamat
 Fix For: 0.21.0

 Attachments: MAPREDUCE-677-v1.0.patch, MAPREDUCE-677-v1.1.patch




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.