[jira] Updated: (MAPREDUCE-735) ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
[ https://issues.apache.org/jira/browse/MAPREDUCE-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-735: - Attachment: MAPREDUCE-735-v1.2.patch test-patch results [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant tests now. > ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner > > > Key: MAPREDUCE-735 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-735 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Suman Sehgal >Assignee: Amar Kamat > Attachments: HADOOP-6130-v1.0.patch, MAPREDUCE-735-v1.2.patch > > > KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of > the specified key is missing. > Scenario : > === > when value of num.key.fields.for.partition is greater than the separators > provided in the input. > Command: > > hadoop jar streaming.jar -Dmapred.reduce.tasks=3 > -Dnum.key.fields.for.partition=5 -input -output > -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer > org.apache.hadoop.mapred.lib.IdentityReducer -inputformat > org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner > org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881
[ https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729559#action_12729559 ] Vinod K V commented on MAPREDUCE-40: The latest changes look good to me. +1 for the patch. > Memory management variables need a backwards compatibility option after > HADOOP-5881 > --- > > Key: MAPREDUCE-40 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-40 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Hemanth Yamijala >Assignee: rahul k singh >Priority: Blocker > Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, > hadoop-5919-11.patch, hadoop-5919-12.patch, hadoop-5919-2.patch, > hadoop-5919-3.patch, hadoop-5919-4.patch, hadoop-5919-5.patch, > hadoop-5919-6.patch, hadoop-5919-7.patch, hadoop-5919-8.patch, > hadoop-5919-9.patch > > > HADOOP-5881 modified variables related to memory management without looking > at the backwards compatibility angle. This JIRA is to adress the gap. Marking > it a blocker for 0.20.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-735) ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
[ https://issues.apache.org/jira/browse/MAPREDUCE-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729561#action_12729561 ] Amar Kamat commented on MAPREDUCE-735: -- Looks like the file I intended to patch got moved to mapreduce. I forgot to add the testcase changes to mapreduce tests. Will upload a new patch for the same. > ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner > > > Key: MAPREDUCE-735 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-735 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Suman Sehgal >Assignee: Amar Kamat > Attachments: HADOOP-6130-v1.0.patch, MAPREDUCE-735-v1.2.patch > > > KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of > the specified key is missing. > Scenario : > === > when value of num.key.fields.for.partition is greater than the separators > provided in the input. > Command: > > hadoop jar streaming.jar -Dmapred.reduce.tasks=3 > -Dnum.key.fields.for.partition=5 -input -output > -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer > org.apache.hadoop.mapred.lib.IdentityReducer -inputformat > org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner > org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-745) TestRecoveryManager fails sometimes
TestRecoveryManager fails sometimes --- Key: MAPREDUCE-745 URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Amareshwari Sriramadasu Fix For: 0.21.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-745: -- Attachment: TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt attaching test failure log. > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-623) Resolve javac warnings in mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-623: - Issue Type: Improvement (was: Bug) > Resolve javac warnings in mapred > > > Key: MAPREDUCE-623 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-623 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Attachments: mapreduce-623.patch > > > Towards a solution for HADOOP-5628, we need to resolve all javac warnings. > This jira will try to resolve javac warnings where ever possible and suppress > them where resolution is not possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-623) Resolve javac warnings in mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-623: - Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this. Thanks Jothi! > Resolve javac warnings in mapred > > > Key: MAPREDUCE-623 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-623 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Fix For: 0.21.0 > > Attachments: mapreduce-623.patch > > > Towards a solution for HADOOP-5628, we need to resolve all javac warnings. > This jira will try to resolve javac warnings where ever possible and suppress > them where resolution is not possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-746) When a task tracker is killed, there is a Null Pointer exception thrown.
When a task tracker is killed, there is a Null Pointer exception thrown. - Key: MAPREDUCE-746 URL: https://issues.apache.org/jira/browse/MAPREDUCE-746 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Iyappan Srinivasan Priority: Minor When a task tracker is killed, the job completes. But tehre is a null pointer exception thrown: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.removeHostCapacity(JobTracker.java:759) at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.blackListTracker(JobTracker.java:624) at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.incrementFaults(JobTracker.java:601) at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2337) at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2998) at org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2584) at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2473) at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1047) at org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3867) at org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3079) at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2817) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958) Steps to reproduce the issue: 1) Bring up a 5 node cluster. 2) set mapred.max.tracker.failures to 1 3) Run a sleep command with 5 maps and 5 redcues. 4) Kill a task tracker, when map is 80% complete. 5) Kill the task tracker, by using "kill -9 process_id". 6) At the time of killing, it was running 2 maps and 2 reduces. 7) Allow 12 minutes to elapse for that task tracker to go out of job nodes list 8) Then job completes successfully by giving the task attempts of the lost trackers to other nodes. 9) The job tracker logs has this exception. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-746) When a task tracker is killed, there is a Null Pointer exception thrown.
[ https://issues.apache.org/jira/browse/MAPREDUCE-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729570#action_12729570 ] Iyappan Srinivasan commented on MAPREDUCE-746: -- The job tracker logs just before this null pointer exception. I have replaced teh anme of killed task tracker server name with this string "" "2009-07-10 07:51:43,626 INFO org.apache.hadoop.mapred.JobTracker: Adding to the blacklist across all jobs 2009-07-10 07:51:43,628 INFO org.apache.hadoop.mapred.JobTracker: Blacklisting tracker : Reason for blacklisting is : EXCEEDING_FAILURES 2009-07-10 07:51:43,631 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9020, call heartbeat(org.apache.hadoop.mapred.tasktrackersta...@1fb050c, false, false, true, 269) from 98.136.99.54:55815: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException" > When a task tracker is killed, there is a Null Pointer exception thrown. > - > > Key: MAPREDUCE-746 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-746 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Iyappan Srinivasan >Priority: Minor > > When a task tracker is killed, the job completes. But tehre is a null pointer > exception thrown: > java.io.IOException: java.lang.NullPointerException > at > org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.removeHostCapacity(JobTracker.java:759) > at > org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.blackListTracker(JobTracker.java:624) > at > org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.incrementFaults(JobTracker.java:601) > at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2337) > at > org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2998) > at > org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2584) > at > org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2473) > at > org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1047) > at > org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3867) > at > org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3079) > at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2817) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958) > Steps to reproduce the issue: > 1) Bring up a 5 node cluster. > 2) set mapred.max.tracker.failures to 1 > 3) Run a sleep command with 5 maps and 5 redcues. > 4) Kill a task tracker, when map is 80% complete. > 5) Kill the task tracker, by using "kill -9 process_id". > 6) At the time of killing, it was running 2 maps and 2 reduces. > 7) Allow 12 minutes to elapse for that task tracker to go out of job nodes > list > 8) Then job completes successfully by giving the task attempts of the lost > trackers to other nodes. > 9) The job tracker logs has this exception. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-655) Change KeyValueLineRecordReader and KeyValueTextInputFormat to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-655: -- Attachment: patch-655-2.txt Patch changing the dummy reporter create a counters and do findCounter. This removes the change in LineRecordReader. Ran both TestMRKeyValueTextInputFormat and TestKeyValueTextInputFormat > Change KeyValueLineRecordReader and KeyValueTextInputFormat to use new api. > --- > > Key: MAPREDUCE-655 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-655 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-655-1.txt, patch-655-2.txt, patch-655.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-655) Change KeyValueLineRecordReader and KeyValueTextInputFormat to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-655: - Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thanks Amareshwari! > Change KeyValueLineRecordReader and KeyValueTextInputFormat to use new api. > --- > > Key: MAPREDUCE-655 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-655 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-655-1.txt, patch-655-2.txt, patch-655.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-355) Change org.apache.hadoop.mapred.join to use new api
[ https://issues.apache.org/jira/browse/MAPREDUCE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-355: -- Status: Open (was: Patch Available) > Change org.apache.hadoop.mapred.join to use new api > --- > > Key: MAPREDUCE-355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-355 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-355-1.txt, patch-355.txt > > > To change org.apache.hadoop.examples.Join to use new api, we need to change > org.apache.hadoop.mapred.join to use new api. So, > Deprecate the code in org.apache.hadoop.mapred.join. > Copy the code to org.apache.hadoop.mapreduce.lib.join and Change it to use > new api. > Thoughts ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-355) Change org.apache.hadoop.mapred.join to use new api
[ https://issues.apache.org/jira/browse/MAPREDUCE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-355: -- Attachment: patch-355-2.txt bq. If the key is modified in the map, the outcome of the join is affected. It is necessary to clone the key out of the join collector. A test for this should probably be added to both TestJoinDatamerge and TestDatamerge. That the value in RRs in the tree is also not restored may also be an issue. bq. I was not able to write a test case for this. I tried verifying by doing key.set(-1) in the mapper. I was seeing problem even if i clone. Thanks for the unit test Chris. I updated patch with unit test and fixed the bug. bq. The checks verifying type consistency for keys in general and for values in MultiFilterRecordReader have been removed. Are these not necessary? The checks verifying type consistency for values in MultiFilterRecordReader are not necessary. Because we can have a join such as override(inner(A,B),A). Removed the consistency checks for values in MultiFilterRecordReader. Also added a test TestJoinProperties(suggested by Chris) which tests 1. Outer join associativity : outer(outer(A, B), C) == outer(A, outer(B, C)) == outer(A, B, C) 2. Inner join associativity : inner(inner(A, B), C) == inner(A, inner(B,C)) == inner(A, B, C) 3. Override identity, inner consistency : override(inner(A,B),A) = A Also these tests use different value types in the sources. > Change org.apache.hadoop.mapred.join to use new api > --- > > Key: MAPREDUCE-355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-355 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-355-1.txt, patch-355-2.txt, patch-355.txt > > > To change org.apache.hadoop.examples.Join to use new api, we need to change > org.apache.hadoop.mapred.join to use new api. So, > Deprecate the code in org.apache.hadoop.mapred.join. > Copy the code to org.apache.hadoop.mapreduce.lib.join and Change it to use > new api. > Thoughts ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-355) Change org.apache.hadoop.mapred.join to use new api
[ https://issues.apache.org/jira/browse/MAPREDUCE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-355: -- Status: Patch Available (was: Open) > Change org.apache.hadoop.mapred.join to use new api > --- > > Key: MAPREDUCE-355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-355 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-355-1.txt, patch-355-2.txt, patch-355.txt > > > To change org.apache.hadoop.examples.Join to use new api, we need to change > org.apache.hadoop.mapred.join to use new api. So, > Deprecate the code in org.apache.hadoop.mapred.join. > Copy the code to org.apache.hadoop.mapreduce.lib.join and Change it to use > new api. > Thoughts ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-626) Modify TestLostTracker to improve execution time
[ https://issues.apache.org/jira/browse/MAPREDUCE-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729595#action_12729595 ] Devaraj Das commented on MAPREDUCE-626: --- This patch has a timing issue. What if the tracker-expiry thread hasn't run? Before asking for a task from the job, you should ensure that the TT is lost. One way is to factor out the body of the expiry thread's run method into a separate method within the expiry-thread class. Then you can manually invoke that method from the testcase. Also, we should avoid Thread.sleep in the test. > Modify TestLostTracker to improve execution time > > > Key: MAPREDUCE-626 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-626 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan >Priority: Minor > Attachments: mapred-626-v1.patch, mapred-626.patch, mapred-626.patch > > > This test can be made faster with a few modifications -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-626) Modify TestLostTracker to improve execution time
[ https://issues.apache.org/jira/browse/MAPREDUCE-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-626: -- Status: Open (was: Patch Available) > Modify TestLostTracker to improve execution time > > > Key: MAPREDUCE-626 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-626 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan >Priority: Minor > Attachments: mapred-626-v1.patch, mapred-626.patch, mapred-626.patch > > > This test can be made faster with a few modifications -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-467) Collect information about number of tasks succeeded / total per time unit for a tasktracker.
[ https://issues.apache.org/jira/browse/MAPREDUCE-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729596#action_12729596 ] Iyappan Srinivasan commented on MAPREDUCE-467: -- Tested some important scenarios and found them to pass: 1) After restarting a blacklisted tasktracker, it shd be made healthy and continue to receive task numbers. - In 5 node cluster. :Pass 2) After task tracker is killed and goes out of node list, otehr nodes recive these tasks and execute them. Number of tasks match. 3) Some task attempts are killed. The numbers captured reflects teh failures properly. 4) Do a job restart. task trackers should start receiveing tasks again and reflect it in their windows.. 5) For blacklisting scenarios, first MAPREDUCE-746 needs to be fixed. > Collect information about number of tasks succeeded / total per time unit for > a tasktracker. > - > > Key: MAPREDUCE-467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-467 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hemanth Yamijala >Assignee: Sharad Agarwal > Attachments: 467_v4.patch, 467_v5.patch, 467_v6.patch, 467_v7.patch, > 5931_v1.patch, 5931_v2.patch, 5931_v3.patch > > > Collecting information of number of tasks succeeded / total per tasktracker > and being able to see these counts per hour, day and since start time will > help reason about things like the blacklisting strategy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-467) Collect information about number of tasks succeeded / total per time unit for a tasktracker.
[ https://issues.apache.org/jira/browse/MAPREDUCE-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729601#action_12729601 ] Amareshwari Sriramadasu commented on MAPREDUCE-467: --- +1 for the patch > Collect information about number of tasks succeeded / total per time unit for > a tasktracker. > - > > Key: MAPREDUCE-467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-467 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hemanth Yamijala >Assignee: Sharad Agarwal > Attachments: 467_v4.patch, 467_v5.patch, 467_v6.patch, 467_v7.patch, > 5931_v1.patch, 5931_v2.patch, 5931_v3.patch > > > Collecting information of number of tasks succeeded / total per tasktracker > and being able to see these counts per hour, day and since start time will > help reason about things like the blacklisting strategy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-353) Allow shuffle read and connection timeouts to be configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-353: --- Attachment: MR-353.v1.patch Attaching new patch with suggested changes. > Allow shuffle read and connection timeouts to be configurable > - > > Key: MAPREDUCE-353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-353 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MR-353.patch, MR-353.v1.patch > > > It would be good for latency-sensitive applications to tune the shuffle > read/connection timeouts... in fact this made a huge difference to terasort > since we were seeing individual shuffles stuck for upwards of 60s and had to > have a very small read timeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-711: Status: Open (was: Patch Available) > Move Distributed Cache from Common to Map/Reduce > > > Key: MAPREDUCE-711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-711 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Vinod K V > Attachments: MAPREDUCE-711-20090709-common.txt, > MAPREDUCE-711-20090709-mapreduce.1.txt, MAPREDUCE-711-20090709-mapreduce.txt, > MAPREDUCE-711-20090710.txt > > > Distributed Cache logically belongs as part of map/reduce and not Common. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-711: Attachment: MAPREDUCE-711-20090710.txt Actually Philip, I am going ahead and am doing the package restructuring now itself. Things like these, it would be good to do as early as possible than later. In any case, you will just need another `sed` run over your patch. One more question: - We are knocking off DistributedCache completely from common. This would mean breaking of code that explicitly depends _only_ on the pre-split core jar. Two solutions are possible - 1) duplicate code across projects and 2) keep code in mapreduce and put placeholders in common, thus creating a dependency of mapreduce on common. None of the two seem feasible to me. > Move Distributed Cache from Common to Map/Reduce > > > Key: MAPREDUCE-711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-711 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Vinod K V > Attachments: MAPREDUCE-711-20090709-common.txt, > MAPREDUCE-711-20090709-mapreduce.1.txt, MAPREDUCE-711-20090709-mapreduce.txt, > MAPREDUCE-711-20090710.txt > > > Distributed Cache logically belongs as part of map/reduce and not Common. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729606#action_12729606 ] Vinod K V commented on MAPREDUCE-711: - Owen, Hudson will not be able to run the mapred patch till the changes to common patch are committed along with the common jar committed to mapreduce. How should I go ahead with this? Separate jira issue for common project first? Thanks. > Move Distributed Cache from Common to Map/Reduce > > > Key: MAPREDUCE-711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-711 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Vinod K V > Attachments: MAPREDUCE-711-20090709-common.txt, > MAPREDUCE-711-20090709-mapreduce.1.txt, MAPREDUCE-711-20090709-mapreduce.txt, > MAPREDUCE-711-20090710.txt > > > Distributed Cache logically belongs as part of map/reduce and not Common. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-656) Change org.apache.hadoop.mapred.SequenceFile* classes to use new api
[ https://issues.apache.org/jira/browse/MAPREDUCE-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-656: -- Status: Patch Available (was: Open) > Change org.apache.hadoop.mapred.SequenceFile* classes to use new api > > > Key: MAPREDUCE-656 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-656 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-656.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-656) Change org.apache.hadoop.mapred.SequenceFile* classes to use new api
[ https://issues.apache.org/jira/browse/MAPREDUCE-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-656: -- Attachment: patch-656.txt Patch depracating org.apache.hadoop.mapred.SequenceFile* classes and moving them to mapreduce package > Change org.apache.hadoop.mapred.SequenceFile* classes to use new api > > > Key: MAPREDUCE-656 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-656 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Attachments: patch-656.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-153) TestJobInProgressListener sometimes timesout
[ https://issues.apache.org/jira/browse/MAPREDUCE-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal resolved MAPREDUCE-153. -- Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] I committed this. Thanks Amar! > TestJobInProgressListener sometimes timesout > > > Key: MAPREDUCE-153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-153 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amar Kamat >Assignee: Amar Kamat > Fix For: 0.21.0 > > Attachments: MAPREDUCE-153-v1.0.patch, MAPREDUCE-153-v1.1.patch > > > It times out with "Could not find /taskTracker/jobcache/jobid/work in any of > the configured local directories". -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-632) Merge TestCustomOutputCommitter with TestCommandLineJobSubmission
[ https://issues.apache.org/jira/browse/MAPREDUCE-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-632: - Issue Type: Improvement (was: Bug) > Merge TestCustomOutputCommitter with TestCommandLineJobSubmission > - > > Key: MAPREDUCE-632 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-632 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Fix For: 0.21.0 > > Attachments: hadoop-5978.java > > > TestCommandLineJobSubmission tests job submisison with different command line > options. This can be easily enhanced to test custom output committer too and > we can do away with TestCustomOutputCommitter -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-632) Merge TestCustomOutputCommitter with TestCommandLineJobSubmission
[ https://issues.apache.org/jira/browse/MAPREDUCE-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sharad Agarwal updated MAPREDUCE-632: - Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this. Thanks Jothi! > Merge TestCustomOutputCommitter with TestCommandLineJobSubmission > - > > Key: MAPREDUCE-632 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-632 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Jothi Padmanabhan >Assignee: Jothi Padmanabhan > Fix For: 0.21.0 > > Attachments: hadoop-5978.java > > > TestCommandLineJobSubmission tests job submisison with different command line > options. This can be easily enhanced to test custom output committer too and > we can do away with TestCustomOutputCommitter -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-383) pipes combiner does not reset properly after a spill
[ https://issues.apache.org/jira/browse/MAPREDUCE-383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729619#action_12729619 ] Hadoop QA commented on MAPREDUCE-383: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12410966/patch.HADOOP-6070 against trunk revision 792839. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/373/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/373/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/373/console This message is automatically generated. > pipes combiner does not reset properly after a spill > > > Key: MAPREDUCE-383 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-383 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Christian Kunz >Assignee: Christian Kunz > Attachments: patch.HADOOP-6070 > > > When using a pipes combiner, the variable numBytes is not reset to 0 in > spillAll, effectively reducing the effect of running a combiner to the first > spill. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-339) JobTracker should give preference to failed tasks over virgin tasks so as to terminate the job ASAP if it is eventually going to fail.
[ https://issues.apache.org/jira/browse/MAPREDUCE-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-339: - Attachment: MAPREDUCE-339-v1.2.patch Attaching a patch that simply queues up failed task and first checks for failed task before scheduling any task. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Testing in progress. > JobTracker should give preference to failed tasks over virgin tasks so as to > terminate the job ASAP if it is eventually going to fail. > --- > > Key: MAPREDUCE-339 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-339 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Gautam Kowshik >Assignee: Devaraj Das > Attachments: MAPREDUCE-339-v1.2.patch > > > Case in point... I have 1585 maps and 160 slots (40 nodes). The job is such > that all maps fail within 2-3 minutes. The job takes forever to realise that > the job is bad. It took 2526 failures for it to reach 4 failed attempts for a > task. > As I understand, currently the JT prefers a failed task if and only if a task > tracker with a split replica for that map came asking for a task. In fact > there may not be a single TT at all in the mapred cluster which has a replica > for the splits used in this job (pre-0.20). This delays the job failure by a > lot and hence degrades cluster utilization as a whole. If i'm on a shared > cluster with many jobs waiting on it to fail, it's bad. > The JT should prefer a failed task a lot earlier than waiting for a data > local TT to come around asking. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-745) TestRecoveryManager fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-745: - Attachment: MAPREDUCE-745-v1.0.patch Attaching an example patch. The reason for failure is that MAPREDUCE-463 changes the job initialization code in RecoveryManager but failed to throw Exception upon init failure. > TestRecoveryManager fails sometimes > --- > > Key: MAPREDUCE-745 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-745 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: MAPREDUCE-745-v1.0.patch, > TEST-org.apache.hadoop.mapred.TestRecoveryManager.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-735) ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
[ https://issues.apache.org/jira/browse/MAPREDUCE-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-735: - Attachment: MAPREDUCE-735-v1.4-branch-0.20.patch MAPREDUCE-735-v1.4.patch Attaching a patch that modifies the mapreduce testcase. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Ant tests in progress. > ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner > > > Key: MAPREDUCE-735 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-735 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Suman Sehgal >Assignee: Amar Kamat > Attachments: HADOOP-6130-v1.0.patch, MAPREDUCE-735-v1.2.patch, > MAPREDUCE-735-v1.4-branch-0.20.patch, MAPREDUCE-735-v1.4.patch > > > KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of > the specified key is missing. > Scenario : > === > when value of num.key.fields.for.partition is greater than the separators > provided in the input. > Command: > > hadoop jar streaming.jar -Dmapred.reduce.tasks=3 > -Dnum.key.fields.for.partition=5 -input -output > -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer > org.apache.hadoop.mapred.lib.IdentityReducer -inputformat > org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner > org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881
[ https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729643#action_12729643 ] rahul k singh commented on MAPREDUCE-40: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] == > Memory management variables need a backwards compatibility option after > HADOOP-5881 > --- > > Key: MAPREDUCE-40 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-40 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Hemanth Yamijala >Assignee: rahul k singh >Priority: Blocker > Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, > hadoop-5919-11.patch, hadoop-5919-12.patch, hadoop-5919-2.patch, > hadoop-5919-3.patch, hadoop-5919-4.patch, hadoop-5919-5.patch, > hadoop-5919-6.patch, hadoop-5919-7.patch, hadoop-5919-8.patch, > hadoop-5919-9.patch > > > HADOOP-5881 modified variables related to memory management without looking > at the backwards compatibility angle. This JIRA is to adress the gap. Marking > it a blocker for 0.20.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner
[ https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729674#action_12729674 ] Tom White commented on MAPREDUCE-710: - +1 Minor nitpick: Can you use org.apache.hadoop.util.Shell instead of manipulating Process directly for running chmod? > Sqoop should read and transmit passwords in a more secure manner > > > Key: MAPREDUCE-710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-710 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.patch > > > Sqoop's current support for passwords involves reading passwords from the > command line "--password foo", which makes the password visible to other > users via 'ps'. An invisible-console approach should be taken. > Related, Sqoop transmits passwords to mysqldump in the same fashion, which is > also insecure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881
[ https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-40: --- Attachment: hadoop-5919-12-20.patch attached patch for 20 branch. > Memory management variables need a backwards compatibility option after > HADOOP-5881 > --- > > Key: MAPREDUCE-40 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-40 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Hemanth Yamijala >Assignee: rahul k singh >Priority: Blocker > Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, > hadoop-5919-11.patch, hadoop-5919-12-20.patch, hadoop-5919-12.patch, > hadoop-5919-2.patch, hadoop-5919-3.patch, hadoop-5919-4.patch, > hadoop-5919-5.patch, hadoop-5919-6.patch, hadoop-5919-7.patch, > hadoop-5919-8.patch, hadoop-5919-9.patch > > > HADOOP-5881 modified variables related to memory management without looking > at the backwards compatibility angle. This JIRA is to adress the gap. Marking > it a blocker for 0.20.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-153) TestJobInProgressListener sometimes timesout
[ https://issues.apache.org/jira/browse/MAPREDUCE-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-153: - Release Note: Only one MR cluster is brought up and hence there is no scope of jobid clashing. > TestJobInProgressListener sometimes timesout > > > Key: MAPREDUCE-153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-153 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amar Kamat >Assignee: Amar Kamat > Fix For: 0.21.0 > > Attachments: MAPREDUCE-153-v1.0.patch, MAPREDUCE-153-v1.1.patch > > > It times out with "Could not find /taskTracker/jobcache/jobid/work in any of > the configured local directories". -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729692#action_12729692 ] Philip Zeyliger commented on MAPREDUCE-711: --- Cool; I'll produce a new patch once you upload a new one here. Do consider changing the package name from filecache to distributedcache, since two names are more confusing than one. I think people who depended on the one-jar-to-rule-them-all (the pre-split world) will assume that they must depend on all three split jars for if they don't want to worry about what ended up where. So I'm not sure you're breaking code by moving it into another jar any more than the project split already has. -- Philip > Move Distributed Cache from Common to Map/Reduce > > > Key: MAPREDUCE-711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-711 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Vinod K V > Attachments: MAPREDUCE-711-20090709-common.txt, > MAPREDUCE-711-20090709-mapreduce.1.txt, MAPREDUCE-711-20090709-mapreduce.txt, > MAPREDUCE-711-20090710.txt > > > Distributed Cache logically belongs as part of map/reduce and not Common. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725) CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729713#action_12729713 ] rahul k singh commented on MAPREDUCE-725: - Looked at the JobInProgress's findSpeculativeTask , it doesn't change any state of TaskInProgress. We can refactor this method in order to find out if jip has any speculative tasks. We are using hasSpeculative to block a slot on the task tracker. As hasSpeculative is very dynamic in nature , there are chances that by the time tasktracker comes back for the task , we dont have any speculativeTasks to assign, this would mean we would waste some heartbeat and also would lead to under utilization. This problem is already there with the capacity-scheduler though. One way of reducing this is to have MIN_SPEC_CAP(in JobInProgress) kind of variable in capacity scheduler. Any thoughts? > CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141 > --- > > Key: MAPREDUCE-725 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-725 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched >Reporter: Arun C Murthy >Priority: Critical > > CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the > old speculation code (pre HADOOP-2141) which needs to be fixed in-order for > speculation to work correctly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-725) CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
[ https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729714#action_12729714 ] rahul k singh commented on MAPREDUCE-725: - Looked at the JobInProgress's findSpeculativeTask , it doesn't change any state of TaskInProgress. We can refactor this method in order to find out if jip has any speculative tasks. We are using hasSpeculative to block a slot on the task tracker. As hasSpeculative is very dynamic in nature , there are chances that by the time tasktracker comes back for the task , we dont have any speculativeTasks to assign, this would mean we would waste some heartbeat and also would lead to under utilization. This problem is already there with the capacity-scheduler though. One way of reducing this is to have MIN_SPEC_CAP(in JobInProgress) kind of variable in capacity scheduler. Any thoughts? > CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141 > --- > > Key: MAPREDUCE-725 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-725 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched >Reporter: Arun C Murthy >Priority: Critical > > CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the > old speculation code (pre HADOOP-2141) which needs to be fixed in-order for > speculation to work correctly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-711) Move Distributed Cache from Common to Map/Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729718#action_12729718 ] Owen O'Malley commented on MAPREDUCE-711: - Just post the result of the test-patch and post on the jira that the regressions still pass. > Move Distributed Cache from Common to Map/Reduce > > > Key: MAPREDUCE-711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-711 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Vinod K V > Attachments: MAPREDUCE-711-20090709-common.txt, > MAPREDUCE-711-20090709-mapreduce.1.txt, MAPREDUCE-711-20090709-mapreduce.txt, > MAPREDUCE-711-20090710.txt > > > Distributed Cache logically belongs as part of map/reduce and not Common. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner
[ https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-710: Status: Patch Available (was: Open) > Sqoop should read and transmit passwords in a more secure manner > > > Key: MAPREDUCE-710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-710 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.3.patch, > MAPREDUCE-710.patch > > > Sqoop's current support for passwords involves reading passwords from the > command line "--password foo", which makes the password visible to other > users via 'ps'. An invisible-console approach should be taken. > Related, Sqoop transmits passwords to mysqldump in the same fashion, which is > also insecure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner
[ https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-710: Attachment: MAPREDUCE-710.3.patch New patch including Tom's suggestion > Sqoop should read and transmit passwords in a more secure manner > > > Key: MAPREDUCE-710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-710 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.3.patch, > MAPREDUCE-710.patch > > > Sqoop's current support for passwords involves reading passwords from the > command line "--password foo", which makes the password visible to other > users via 'ps'. An invisible-console approach should be taken. > Related, Sqoop transmits passwords to mysqldump in the same fashion, which is > also insecure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner
[ https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-710: Status: Open (was: Patch Available) > Sqoop should read and transmit passwords in a more secure manner > > > Key: MAPREDUCE-710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-710 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.3.patch, > MAPREDUCE-710.patch > > > Sqoop's current support for passwords involves reading passwords from the > command line "--password foo", which makes the password visible to other > users via 'ps'. An invisible-console approach should be taken. > Related, Sqoop transmits passwords to mysqldump in the same fashion, which is > also insecure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-747) In Job Tracker logs, some host locations [either in SPLITS or in HOSTNAME subrecords] have numeric host locations
In Job Tracker logs, some host locations [either in SPLITS or in HOSTNAME subrecords] have numeric host locations - Key: MAPREDUCE-747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-747 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Dick King For example, instead of saying the normal HOSTNAME="/1\.2\.3\.192/node0123\.hadoop-cluster\.megacorp\.com" we might see the erroneous HOSTNAME="/1\.2\.3\.192/1\.2\.3\.197" where the IP address of node0123.hadoop-cluster.megacorp.com is in fact 1.2.3.197 . This is not a property of certain hosts. In our cluster, most hosts are occasionally reported in each of the two formats. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-748) In Job Tracker log map attempt failure reports, failed maps show a HOSTNAME without a rack ID.
In Job Tracker log map attempt failure reports, failed maps show a HOSTNAME without a rack ID. -- Key: MAPREDUCE-748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-748 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Dick King For example, from a job tracker log: MapAttempt TASK_TYPE="MAP" TASKID="task_200904211745_0002_m_02" TASK_ATTEMPT_ID="attempt_200904211745_0002_m_02_0" START_TIME="1240336754665" TRACKER_NAME="tracker_redacted1670\.redacted2\.com:localhost/127\.0\.0\.1:47698" HTTP_PORT="50060" . MapAttempt TASK_TYPE="MAP" TASKID="task_200904211745_0002_m_02" TASK_ATTEMPT_ID="attempt_200904211745_0002_m_02_0" TASK_STATUS="FAILED" FINISH_TIME="1240336777673" HOSTNAME="redacted1670\.redacted2\.com" ERROR="java\.io\.IOException: Task process exit with nonzero status of 15\. at org\.apache\.hadoop\.mapred\.TaskRunner\.run(TaskRunner\.java:424) ,java\.io\.IOException: Task process exit with nonzero status of 15\. at org\.apache\.hadoop\.mapred\.TaskRunner\.run(TaskRunner\.java:424) " . The hostname should have been [for example] HOSTNAME="/1\.2\.3\.192/redacted1670\.redacted2\.com" -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-742) Improv e the java comments for the π examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-742: - Attachment: m742_20090710.patch Thanks, Jakob for the review comments. m742_20090710.patch: rewrote the paragraphs about the limitation. > Improve the java comments for the π examples > > > Key: MAPREDUCE-742 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-742 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, examples >Affects Versions: 0.21.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 0.21.0 > > Attachments: m742_20090708.patch, m742_20090709.patch, > m742_20090710.patch > > > There are 3 examples, pi, bbp and distbbp for π computation. We should > tell the difference between them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-742) Impro ve the java comments for the π examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729805#action_12729805 ] Jakob Homan commented on MAPREDUCE-742: --- Looks great +1. > Improve the java comments for the π examples > > > Key: MAPREDUCE-742 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-742 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, examples >Affects Versions: 0.21.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 0.21.0 > > Attachments: m742_20090708.patch, m742_20090709.patch, > m742_20090710.patch > > > There are 3 examples, pi, bbp and distbbp for π computation. We should > tell the difference between them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-749) Make Sqoop unit tests more Hudson-friendly
Make Sqoop unit tests more Hudson-friendly -- Key: MAPREDUCE-749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-749 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Hudson servers (other than Apache's) need to be able to run the sqoop unit tests which depend on thirdparty JDBC drivers / database implementations. The build.xml needs some refactoring to make this happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-749) Make Sqoop unit tests more Hudson-friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-749: Status: Patch Available (was: Open) > Make Sqoop unit tests more Hudson-friendly > -- > > Key: MAPREDUCE-749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-749.2.patch, MAPREDUCE-749.patch > > > Hudson servers (other than Apache's) need to be able to run the sqoop unit > tests which depend on thirdparty JDBC drivers / database implementations. The > build.xml needs some refactoring to make this happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-749) Make Sqoop unit tests more Hudson-friendly
[
https://issues.apache.org/jira/browse/MAPREDUCE-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-749:
Attachment: MAPREDUCE-749.patch
Attaching patch for this. This adds a {{sqoop.thirdparty.lib.dir}} Ant property
which identifies a directory where JDBC libraries are kept. This keeps them out
from the main Hadoop ivy system so they can be placed statically on a test
server. Also added a {{ThirdPartyTests}} module in src/test/o.a.h.sqoop which
includes all the third-party tests. This is itself included by {{AllTests}}.
> Make Sqoop unit tests more Hudson-friendly
> --
>
> Key: MAPREDUCE-749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-749
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-749.2.patch, MAPREDUCE-749.patch
>
>
> Hudson servers (other than Apache's) need to be able to run the sqoop unit
> tests which depend on thirdparty JDBC drivers / database implementations. The
> build.xml needs some refactoring to make this happen.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-749) Make Sqoop unit tests more Hudson-friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-749: Attachment: MAPREDUCE-749.2.patch > Make Sqoop unit tests more Hudson-friendly > -- > > Key: MAPREDUCE-749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-749.2.patch, MAPREDUCE-749.patch > > > Hudson servers (other than Apache's) need to be able to run the sqoop unit > tests which depend on thirdparty JDBC drivers / database implementations. The > build.xml needs some refactoring to make this happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-742) Impro ve the java comments for the π examples
[
https://issues.apache.org/jira/browse/MAPREDUCE-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729875#action_12729875
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-742:
--
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no new tests are needed
for this patch.
[exec] Also please list what manual steps were
performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
The patch are mostly documentation changes. The code changes are all related
to output messages. So I did not add new unit tests.
Tested Util.millis2String(long) manually as shown below:
{code}
public static void main(String[] args) {
long n = (1L << 32) - 1;
System.out.println(Util.millis2String(n));
}
{code}
Before the patch, the output is "49 days 17:02:47.-01" which is incorrect.
After the patch, the correct value, "49 days 17:02:47.295", is printed.
> Improve the java comments for the π examples
>
>
> Key: MAPREDUCE-742
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-742
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation, examples
>Affects Versions: 0.21.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: m742_20090708.patch, m742_20090709.patch,
> m742_20090710.patch
>
>
> There are 3 examples, pi, bbp and distbbp for π computation. We should
> tell the difference between them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-739) Allow relative paths to be created inside archives.
[ https://issues.apache.org/jira/browse/MAPREDUCE-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-739: Status: Patch Available (was: Open) > Allow relative paths to be created inside archives. > --- > > Key: MAPREDUCE-739 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-739 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.21.0 > > Attachments: HADOOP-3663.patch, HADOOP-3663.patch, HADOOP-3663.patch, > MAPREDUCE-739.patch > > > Archives currently stores the full path from the input sources -- since it > allows multiple sources and regular expressions as inputs. So the created > archives have the full path of the input sources. This is un intuitive and a > user hassle. We should get rid of it and allow users to say that the created > archive should be relative to some absolute path and throw an excpetion if > the input does not confirm to the relative absolute path. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-739) Allow relative paths to be created inside archives.
[
https://issues.apache.org/jira/browse/MAPREDUCE-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-739:
Attachment: MAPREDUCE-739.patch
this patch adds an argument
{code}
archiveName <> -p paretnPath *
{code}
here all the files will be archived relative to parentPath. If the user has
only one directory and wants the archived files relative to that then
{code}
archiveName <> -p paretnPath can be used.
{code}
- the new argument is mandatory and is required. This is a change from old
versions but given that this will make usability much easier it will makes
things easy to understand.
- the old archives is still readalbe from this new code (the old archives that
were created pre 0.21 will be readable still)
- also incorpiorated hairong's comments to make the code much easier to read.
- update forrest docs with better examples in common jira HADOOP-6142.
> Allow relative paths to be created inside archives.
> ---
>
> Key: MAPREDUCE-739
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-739
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
>Reporter: Mahadev konar
>Assignee: Mahadev konar
> Fix For: 0.21.0
>
> Attachments: HADOOP-3663.patch, HADOOP-3663.patch, HADOOP-3663.patch,
> MAPREDUCE-739.patch
>
>
> Archives currently stores the full path from the input sources -- since it
> allows multiple sources and regular expressions as inputs. So the created
> archives have the full path of the input sources. This is un intuitive and a
> user hassle. We should get rid of it and allow users to say that the created
> archive should be relative to some absolute path and throw an excpetion if
> the input does not confirm to the relative absolute path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-742) Impro ve the java comments for the π examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved MAPREDUCE-742. -- Resolution: Fixed Hadoop Flags: [Reviewed] I have committed this. > Improve the java comments for the π examples > > > Key: MAPREDUCE-742 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-742 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, examples >Affects Versions: 0.21.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 0.21.0 > > Attachments: m742_20090708.patch, m742_20090709.patch, > m742_20090710.patch > > > There are 3 examples, pi, bbp and distbbp for π computation. We should > tell the difference between them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-740) Provide summary information per job once a job is finished.
[ https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reassigned MAPREDUCE-740: --- Assignee: Arun C Murthy > Provide summary information per job once a job is finished. > --- > > Key: MAPREDUCE-740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-740 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hong Tang >Assignee: Arun C Murthy >Priority: Minor > > It would be nice if JobTracker can output a one line summary information per > job once a job is finished. Otherwise, users or system administrators would > end up scraping individual job history logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-740) Provide summary information per job once a job is finished.
[ https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-740: Component/s: jobtracker Priority: Major (was: Minor) Fix Version/s: 0.21.0 Issue Type: New Feature (was: Improvement) > Provide summary information per job once a job is finished. > --- > > Key: MAPREDUCE-740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-740 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Hong Tang >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-740_0_20090709.patch > > > It would be nice if JobTracker can output a one line summary information per > job once a job is finished. Otherwise, users or system administrators would > end up scraping individual job history logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-740) Provide summary information per job once a job is finished.
[ https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-740: Attachment: MAPREDUCE-740_0_20090709.patch Straight-forward patch which allows for a new (configurable) appender which can be used to direct job-summary (one line summary per job) to the desired location. > Provide summary information per job once a job is finished. > --- > > Key: MAPREDUCE-740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-740 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Hong Tang >Assignee: Arun C Murthy >Priority: Minor > Fix For: 0.21.0 > > Attachments: MAPREDUCE-740_0_20090709.patch > > > It would be nice if JobTracker can output a one line summary information per > job once a job is finished. Otherwise, users or system administrators would > end up scraping individual job history logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-740) Provide summary information per job once a job is finished.
[
https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729893#action_12729893
]
Arun C Murthy commented on MAPREDUCE-740:
-
Example log:
{noformat}
09/07/10 16:39:39 INFO mapred.JobInProgress$JobSummary:
jobId=job_200907101638_0001,submitTime=1247269137321,launchTime=1247269137920,finishTime=1247269179380,numMaps=10,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=arunc,queue=default,status=SUCCEEDED,mapSlotSeconds=39,reduceSlotsSeconds=0,clusterMapCapacity=4,clusterReduceCapacity=4
{noformat}
> Provide summary information per job once a job is finished.
> ---
>
> Key: MAPREDUCE-740
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-740
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobtracker
>Reporter: Hong Tang
>Assignee: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-740_0_20090709.patch
>
>
> It would be nice if JobTracker can output a one line summary information per
> job once a job is finished. Otherwise, users or system administrators would
> end up scraping individual job history logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner
[ https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729895#action_12729895 ] Hadoop QA commented on MAPREDUCE-710: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413143/MAPREDUCE-710.3.patch against trunk revision 792901. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/376/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/376/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/376/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/376/console This message is automatically generated. > Sqoop should read and transmit passwords in a more secure manner > > > Key: MAPREDUCE-710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-710 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.3.patch, > MAPREDUCE-710.patch > > > Sqoop's current support for passwords involves reading passwords from the > command line "--password foo", which makes the password visible to other > users via 'ps'. An invisible-console approach should be taken. > Related, Sqoop transmits passwords to mysqldump in the same fashion, which is > also insecure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-710) Sqoop should read and transmit passwords in a more secure manner
[ https://issues.apache.org/jira/browse/MAPREDUCE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729902#action_12729902 ] Aaron Kimball commented on MAPREDUCE-710: - Test failures are still unrelated to the patch. :) > Sqoop should read and transmit passwords in a more secure manner > > > Key: MAPREDUCE-710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-710 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-710.2.patch, MAPREDUCE-710.3.patch, > MAPREDUCE-710.patch > > > Sqoop's current support for passwords involves reading passwords from the > command line "--password foo", which makes the password visible to other > users via 'ps'. An invisible-console approach should be taken. > Related, Sqoop transmits passwords to mysqldump in the same fashion, which is > also insecure. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-750) Extensible ConnManager factory API
Extensible ConnManager factory API -- Key: MAPREDUCE-750 URL: https://issues.apache.org/jira/browse/MAPREDUCE-750 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Sqoop uses the ConnFactory class to instantiate a ConnManager implementation based on the connect string and other arguments supplied by the user. This allows per-database logic to be encapsulated in different ConnManager instances, and dynamically chosen based on which database the user is actually importing from. But adding new ConnManager implementations requires modifying the source of a common ConnFactory class. An indirection layer should be used to delegate instantiation to a number of factory implementations which can be specified in the static configuration or at runtime. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-750) Extensible ConnManager factory API
[ https://issues.apache.org/jira/browse/MAPREDUCE-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-750: Attachment: MAPREDUCE-750.patch > Extensible ConnManager factory API > -- > > Key: MAPREDUCE-750 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-750 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-750.patch > > > Sqoop uses the ConnFactory class to instantiate a ConnManager implementation > based on the connect string and other arguments supplied by the user. This > allows per-database logic to be encapsulated in different ConnManager > instances, and dynamically chosen based on which database the user is > actually importing from. But adding new ConnManager implementations requires > modifying the source of a common ConnFactory class. An indirection layer > should be used to delegate instantiation to a number of factory > implementations which can be specified in the static configuration or at > runtime. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-750) Extensible ConnManager factory API
[
https://issues.apache.org/jira/browse/MAPREDUCE-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729907#action_12729907
]
Aaron Kimball commented on MAPREDUCE-750:
-
This patch creates this public API. The {{sqoop.conn.factories}} parameter has
been added to mapred-default.xml. This parameter defaults to the
{{DefaultManagerFactory}} implementation supplied in Sqoop, which contains much
of the logic previously in {{ConnManager}}.
The names of additional classes implementing {{ManagerFactory}} may be added to
the config parameter. The ConnFactory implementation will instantiate all such
{{ManagerFactory}} implementations. When {{ConnFactory.getManager()}} is
called, it will consult the {{accept()}} all {{ManagerFactory}} implementations
in order, returning the first non-null value returned by an {{accept()}} method
(similar to the delegation mechanism used by the Serializations framework).
> Extensible ConnManager factory API
> --
>
> Key: MAPREDUCE-750
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-750
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-750.patch
>
>
> Sqoop uses the ConnFactory class to instantiate a ConnManager implementation
> based on the connect string and other arguments supplied by the user. This
> allows per-database logic to be encapsulated in different ConnManager
> instances, and dynamically chosen based on which database the user is
> actually importing from. But adding new ConnManager implementations requires
> modifying the source of a common ConnFactory class. An indirection layer
> should be used to delegate instantiation to a number of factory
> implementations which can be specified in the static configuration or at
> runtime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-750) Extensible ConnManager factory API
[ https://issues.apache.org/jira/browse/MAPREDUCE-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-750: Status: Patch Available (was: Open) > Extensible ConnManager factory API > -- > > Key: MAPREDUCE-750 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-750 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-750.patch > > > Sqoop uses the ConnFactory class to instantiate a ConnManager implementation > based on the connect string and other arguments supplied by the user. This > allows per-database logic to be encapsulated in different ConnManager > instances, and dynamically chosen based on which database the user is > actually importing from. But adding new ConnManager implementations requires > modifying the source of a common ConnFactory class. An indirection layer > should be used to delegate instantiation to a number of factory > implementations which can be specified in the static configuration or at > runtime. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-3) Set mapred.child.ulimit automatically to the value of the RAM limits for a job, if they are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729916#action_12729916
]
Aaron Kimball commented on MAPREDUCE-3:
---
A caution here is that {{mapred.child.ulimit}} needs to account for the memory
overhead of the JVM itself. Merely setting {{mapred.child.ulimit}} to the same
value as the {{-Xmx}}__{{m}} in {{mapred.child.java.opts}} will fail to
launch child tasks. You'll need some overhead room; I don't know exactly how
much.
> Set mapred.child.ulimit automatically to the value of the RAM limits for a
> job, if they are set
> ---
>
> Key: MAPREDUCE-3
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>
> Memory based monitoring and scheduling allow users to set memory limits for
> the tasks of their jobs. This parameter is the total memory taken by the
> task, and any children it may launch (for e.g. in the case of streaming). A
> related parameter is mapred.child.ulimit which is a hard limit on the memory
> used by a single process of the entire task tree. For user convenience, it
> would be sensible for the system to set the ulimit to atleast the memory
> required by the task, if the user has specified the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-740) Provide summary information per job once a job is finished.
[ https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729922#action_12729922 ] Rajiv Chittajallu commented on MAPREDUCE-740: - +1 for the log format. > Provide summary information per job once a job is finished. > --- > > Key: MAPREDUCE-740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-740 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Hong Tang >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-740_0_20090709.patch > > > It would be nice if JobTracker can output a one line summary information per > job once a job is finished. Otherwise, users or system administrators would > end up scraping individual job history logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-740) Provide summary information per job once a job is finished.
[ https://issues.apache.org/jira/browse/MAPREDUCE-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729921#action_12729921 ] Hong Tang commented on MAPREDUCE-740: - +1. Patch looks good. > Provide summary information per job once a job is finished. > --- > > Key: MAPREDUCE-740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-740 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Hong Tang >Assignee: Arun C Murthy > Fix For: 0.21.0 > > Attachments: MAPREDUCE-740_0_20090709.patch > > > It would be nice if JobTracker can output a one line summary information per > job once a job is finished. Otherwise, users or system administrators would > end up scraping individual job history logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-705) User-configurable quote and delimiter characters for Sqoop records and record reparsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729943#action_12729943 ] Hadoop QA commented on MAPREDUCE-705: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413046/MAPREDUCE-705.2.patch against trunk revision 793136. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 45 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 316 release audit warnings (more than the trunk's current 315 warnings). -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/377/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/377/artifact/trunk/current/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/377/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/377/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/377/console This message is automatically generated. > User-configurable quote and delimiter characters for Sqoop records and record > reparsing > --- > > Key: MAPREDUCE-705 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-705 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-705.2.patch, MAPREDUCE-705.patch > > > Sqoop needs a mechanism for users to govern how fields are quoted and what > delimiter characters separate fields and records. With delimiters providing > an unambiguous format, a parse method can reconstitute the generated record > data object from a text-based representation of the same record. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-421) mapred pipes might return exit code 0 even when failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729954#action_12729954 ] Hadoop QA commented on MAPREDUCE-421: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413047/MAPREDUCE-421.patch against trunk revision 793136. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/378/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/378/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/378/console This message is automatically generated. > mapred pipes might return exit code 0 even when failing > --- > > Key: MAPREDUCE-421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.1 > > Attachments: MAPREDUCE-421.patch > > > up to hadoop 0.18.3 org.apache.hadoop.mapred.JobShell ensured that 'hadoop > jar' returns non-zero exit code when the job fails. > This is no longer true after moving this to org.apache.hadoop.util.RunJar. > Pipes jobs submitted through cli never returned proper exit code. > The main methods in org.apache.hadoop.util.RunJar. and > org.apache.hadoop.mapred.pipes.Submitter should be modified to return an exit > code similar to how org.apache.hadoop.mapred.JobShell did it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
