[jira] [Commented] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036713#comment-13036713 ] Hadoop QA commented on MAPREDUCE-2407: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478943/2407.patch against trunk revision 1125223. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 2 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//testReport/ Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/283//console This message is automatically generated. Make Gridmix emulate usage of Distributed Cache files - Key: MAPREDUCE-2407 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 2407.patch Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-2407: Attachment: 2407.v1.patch Attaching new patch fixing the release audit warning. Make Gridmix emulate usage of Distributed Cache files - Key: MAPREDUCE-2407 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 2407.patch, 2407.v1.patch Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-2407: Status: Open (was: Patch Available) Make Gridmix emulate usage of Distributed Cache files - Key: MAPREDUCE-2407 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 2407.patch, 2407.v1.patch Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2512) wait(5000) and notify() mechanism can be implemented instead of sleep(5000) in reduce task when there are no copies in progress and no new copies to schedule
[
https://issues.apache.org/jira/browse/MAPREDUCE-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036751#comment-13036751
]
Devaraj K commented on MAPREDUCE-2512:
--
This try/catch is not related to the any stuck tasks. If there are no map
outputs ready to copy, this thread goes to sleep state for 5000 millis and then
continue execution after timeout. When this thread is in sleep state, if any
thread performs some operation (like interrupt (), stop ()) on this thread, it
will throw InterruptedException and comes out from the sleep state. After
getting InterruptedException, it will ignore and continue execution normally.
This thread waits for 5000 millis even If it gets new map completion events
before 5000 millis. This can be optimized such that,
{code:title=ReduceTask.java|borderStyle=solid}
reporter.progress();
- Thread.sleep(5000);
+ synchronized (lockCopy) {
+lockCopy.wait(5000);
+ }
}
} catch (InterruptedException e) { } // IGNORE
int numNewMaps = getMapCompletionEvents();
if (numNewMaps 0) {
+ synchronized (lockCopy) {
+lockCopy.notify();
+ }
LOG.info(reduceTask.getTaskID() + : +
Got + numNewMaps + new map-outputs);
{code}
wait(5000) and notify() mechanism can be implemented instead of sleep(5000)
in reduce task when there are no copies in progress and no new copies to
schedule
-
Key: MAPREDUCE-2512
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2512
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.20.2
Reporter: Devaraj K
Assignee: Devaraj K
{code:title=ReduceTask.java|borderStyle=solid}
try {
if (numInFlight == 0 numScheduled == 0) {
// we should indicate progress as we don't want TT to think
// we're stuck and kill us
reporter.progress();
Thread.sleep(5000);
}
} catch (InterruptedException e) { } // IGNORE
{code}
Here if we have no copies in flight and we can't schedule anything new, it is
going to wait for 5000 millis. Instead of waiting for 5000 millis, this
thread can wait with timeout and GetMapEventsThread can notify it if gets new
map completion events earlier than 5000 millis time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036777#comment-13036777 ] Hadoop QA commented on MAPREDUCE-2407: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479884/2407.v1.patch against trunk revision 1125223. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/284//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/284//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/284//console This message is automatically generated. Make Gridmix emulate usage of Distributed Cache files - Key: MAPREDUCE-2407 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 2407.patch, 2407.v1.patch Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2516) option to control sensitive web actions
[ https://issues.apache.org/jira/browse/MAPREDUCE-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036808#comment-13036808 ] Hudson commented on MAPREDUCE-2516: --- Integrated in Hadoop-Mapreduce-22-branch #57 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/57/]) MAPREDUCE-2516. Rename webinterface.private.actions to mapreduce.jobtracker.webinterface.trusted. Contributed by Ari Rabkin. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125222 Files : * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java * /hadoop/mapreduce/branches/branch-0.22/src/java/mapred-default.xml * /hadoop/mapreduce/branches/branch-0.22/CHANGES.txt * /hadoop/mapreduce/branches/branch-0.22/src/java/org/apache/hadoop/mapred/JSPUtil.java * /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java option to control sensitive web actions --- Key: MAPREDUCE-2516 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2516 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Ari Rabkin Assignee: Ari Rabkin Priority: Minor Fix For: 0.22.0 Attachments: MAPREDUCE-2516.txt as per HADOOP-7302, webinterface.private.actions should not be in trunk. But it should be here, and should have a clearer name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036811#comment-13036811 ] Robert Joseph Evans commented on MAPREDUCE-2470: The difference is that with Job.getStatus it calls internally updateStatus which will throw an IOException if it is unable to get an updated status from the JobTracker. The only time you can get an NPE from that is if you catch the IOException, ignore it, and keep trying to use the Job object what was originally created. We can fix that, by modifying updateStatus to not overwrite Job.status until it knows that the updated status is not null. But that was not explicitly part of this JIRA, and I could not find any other JIRA to cover it so I though it was not a big deal, and most likely worked as designed. Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036814#comment-13036814 ] Robert Joseph Evans commented on MAPREDUCE-2470: Oh I forgot to add that all of the other downgrade methods in JobClient.java are similar to Job.getStatus, either they return values from the cached JobStatus which should never be null except if an IOException was caught and ignored, so they are not likely to cause an NPE. Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2297) All map reduce tasks are failing if we give invalid path jar file for Job
[
https://issues.apache.org/jira/browse/MAPREDUCE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036831#comment-13036831
]
Devaraj K commented on MAPREDUCE-2297:
--
Hi Todd, I agree with you for job specific configuration. If we configure
archives (contains invalid path) common for all jobs then all jobs will fail.
Here all jobs may not depend on all the archives.
Consider the case of Hive, if we configure any invalid path for the property
hive.aux.jars.path all the jobs will fail which is not using that jar also.
All map reduce tasks are failing if we give invalid path jar file for Job
-
Key: MAPREDUCE-2297
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2297
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.20.2
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Fix For: 0.20.4
Attachments: MAPREDUCE-2297.patch
This can be reproduced by giving the invalid jar file for the Job or it can
be reproduced from hive.
In hive-default.xml
property
namehive.aux.jars.path/name
value/value
descriptionProvided for adding auxillaryjarsPath/description
/property
If we configure an invalid path for jar file, It is making all map reduce
tasks to fail even those jobs are not depending on this jar file and it is
giving the below exception.
{code:xml}
hive select * from a join b on(a.b=b.c);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapred.reduce.tasks=number
java.io.FileNotFoundException: File does not exist: /user/root/grade.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:495)
at
org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:651)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:783)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:698)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2524) Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle -- Key: MAPREDUCE-2524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Fix For: 0.20.205.0 The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2525) State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-2525: Assignee: Devaraj K State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp --- Key: MAPREDUCE-2525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2525 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.23.0 Environment: *Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3* Reporter: Devaraj K Assignee: Devaraj K Priority: Minor These are the steps to reproduce, # Select all the running jobs. # Refresh the jobtracker.jsp page. # After refreshing, *Deselect All* button becomes *Select All*, however all the selected check boxes remain in the select state only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2525) State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp
State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp --- Key: MAPREDUCE-2525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2525 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.23.0 Environment: *Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3* Reporter: Devaraj K Priority: Minor These are the steps to reproduce, # Select all the running jobs. # Refresh the jobtracker.jsp page. # After refreshing, *Deselect All* button becomes *Select All*, however all the selected check boxes remain in the select state only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Naisbitt updated MAPREDUCE-2489: Attachment: MAPREDUCE-2489-mapred.patch Jobsplits with random hostnames can make the queue unusable --- Key: MAPREDUCE-2489 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Attachments: MAPREDUCE-2489-mapred.patch We saw an issue where a custom InputSplit was returning invalid hostnames for the splits that were then causing the JobTracker to attempt to excessively resolve host names. This caused a major slowdown for the JobTracker. We should prevent invalid InputSplit hostnames from affecting everyone else. I propose we implement some verification for the hostnames to try to ensure that we only do DNS lookups on valid hostnames (and fail otherwise). We could also fail the job after a certain number of failures in the resolve. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036847#comment-13036847 ] Jeffrey Naisbitt commented on MAPREDUCE-2489: - @Allen, apologies for not being clear in the description. Also, I expect this patch will fail in the Hudson build until HADOOP-7314 has been committed. Jobsplits with random hostnames can make the queue unusable --- Key: MAPREDUCE-2489 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Attachments: MAPREDUCE-2489-mapred.patch We saw an issue where a custom InputSplit was returning invalid hostnames for the splits that were then causing the JobTracker to attempt to excessively resolve host names. This caused a major slowdown for the JobTracker. We should prevent invalid InputSplit hostnames from affecting everyone else. I propose we implement some verification for the hostnames to try to ensure that we only do DNS lookups on valid hostnames (and fail otherwise). We could also fail the job after a certain number of failures in the resolve. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2525) State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2525: - Attachment: MR-UI.jpg State of the checkboxes are not matching with Select All / Deselect All button, after refreshing jobtracker.jsp --- Key: MAPREDUCE-2525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2525 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.23.0 Environment: *Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3* Reporter: Devaraj K Assignee: Devaraj K Priority: Minor Attachments: MR-UI.jpg These are the steps to reproduce, # Select all the running jobs. # Refresh the jobtracker.jsp page. # After refreshing, *Deselect All* button becomes *Select All*, however all the selected check boxes remain in the select state only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2459) Cache HAR filesystem metadata
[ https://issues.apache.org/jira/browse/MAPREDUCE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036867#comment-13036867 ] Mahadev konar commented on MAPREDUCE-2459: -- +1 lgtm. Ill commit it to trunk. Cache HAR filesystem metadata - Key: MAPREDUCE-2459 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2459 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Mac Yang Assignee: Mac Yang Fix For: 0.23.0 Attachments: MAPREDUCE-2459.1.patch, MAPREDUCE-2459.2.patch Each HAR file system has two index files that contains information on how files are stored in the part files. During the block location calculation, these indexes are reread for every file in the archive. Caching the indexes and the status of the part files will greatly reduce the number of name node operations during the job setup time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2463) Job History files are not moving to done folder when job history location is hdfs location
[
https://issues.apache.org/jira/browse/MAPREDUCE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2463:
-
Attachment: MAPREDUCE-2463-2.patch
Job History files are not moving to done folder when job history location is
hdfs location
--
Key: MAPREDUCE-2463
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2463
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 0.23.0
Attachments: MAPREDUCE-2463-1.patch, MAPREDUCE-2463-2.patch,
MAPREDUCE-2463.patch
If mapreduce.jobtracker.jobhistory.location is configured as HDFS location
then either during initialization of Job Tracker (while moving old job
history files) or after completion of the job, history files are not moving
to done and giving following exception.
{code:xml}
2011-04-29 15:27:27,813 ERROR
org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history
file to DONE folder.
java.lang.IllegalArgumentException: Wrong FS:
hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected:
file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
at
org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2264: - Attachment: MAPREDUCE-2264-trunk.patch MAPREDUCE-2264-0.20.3.patch Job status exceeds 100% in some cases -- Key: MAPREDUCE-2264 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Adam Kramer Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, more than 100%.bmp I'm looking now at my jobtracker's list of running reduce tasks. One of them is 120.05% complete, the other is 107.28% complete. I understand that these numbers are estimates, but there is no case in which an estimate of 100% for a non-complete task is better than an estimate of 99.99%, nor is there any case in which an estimate greater than 100% is valid. I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-2264: Assignee: Devaraj K Job status exceeds 100% in some cases -- Key: MAPREDUCE-2264 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Adam Kramer Assignee: Devaraj K Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, more than 100%.bmp I'm looking now at my jobtracker's list of running reduce tasks. One of them is 120.05% complete, the other is 107.28% complete. I understand that these numbers are estimates, but there is no case in which an estimate of 100% for a non-complete task is better than an estimate of 99.99%, nor is there any case in which an estimate greater than 100% is valid. I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2459) Cache HAR filesystem metadata
[ https://issues.apache.org/jira/browse/MAPREDUCE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2459: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk. Thanks mac! Cache HAR filesystem metadata - Key: MAPREDUCE-2459 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2459 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Mac Yang Assignee: Mac Yang Fix For: 0.23.0 Attachments: MAPREDUCE-2459.1.patch, MAPREDUCE-2459.2.patch Each HAR file system has two index files that contains information on how files are stored in the part files. During the block location calculation, these indexes are reread for every file in the archive. Caching the indexes and the status of the part files will greatly reduce the number of name node operations during the job setup time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2463) Job History files are not moving to done folder when job history location is hdfs location
[
https://issues.apache.org/jira/browse/MAPREDUCE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036876#comment-13036876
]
Devaraj K commented on MAPREDUCE-2463:
--
Thanks Todd for reviewing.
Provided patch by addressing the above comments.
Job History files are not moving to done folder when job history location is
hdfs location
--
Key: MAPREDUCE-2463
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2463
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 0.23.0
Attachments: MAPREDUCE-2463-1.patch, MAPREDUCE-2463-2.patch,
MAPREDUCE-2463.patch
If mapreduce.jobtracker.jobhistory.location is configured as HDFS location
then either during initialization of Job Tracker (while moving old job
history files) or after completion of the job, history files are not moving
to done and giving following exception.
{code:xml}
2011-04-29 15:27:27,813 ERROR
org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history
file to DONE folder.
java.lang.IllegalArgumentException: Wrong FS:
hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected:
file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
at
org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036887#comment-13036887 ] Devaraj K commented on MAPREDUCE-2264: -- It was considering compressed bytes size for calculating the totalBytes because of this it is showing the progress as more than 100%. Provided patch consider the uncompressed size of the data for calculating the progress. Job status exceeds 100% in some cases -- Key: MAPREDUCE-2264 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Adam Kramer Assignee: Devaraj K Fix For: 0.20.4, 0.23.0 Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, more than 100%.bmp I'm looking now at my jobtracker's list of running reduce tasks. One of them is 120.05% complete, the other is 107.28% complete. I understand that these numbers are estimates, but there is no case in which an estimate of 100% for a non-complete task is better than an estimate of 99.99%, nor is there any case in which an estimate greater than 100% is valid. I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-2264: - Fix Version/s: 0.23.0 0.20.4 Status: Patch Available (was: Open) Job status exceeds 100% in some cases -- Key: MAPREDUCE-2264 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Adam Kramer Assignee: Devaraj K Fix For: 0.20.4, 0.23.0 Attachments: MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, more than 100%.bmp I'm looking now at my jobtracker's list of running reduce tasks. One of them is 120.05% complete, the other is 107.28% complete. I understand that these numbers are estimates, but there is no case in which an estimate of 100% for a non-complete task is better than an estimate of 99.99%, nor is there any case in which an estimate greater than 100% is valid. I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2459) Cache HAR filesystem metadata
[ https://issues.apache.org/jira/browse/MAPREDUCE-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036890#comment-13036890 ] Hudson commented on MAPREDUCE-2459: --- Integrated in Hadoop-Mapreduce-trunk-Commit #690 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/690/]) MAPREDUCE-2459. Cache HAR filesystem metadata. (Mac Yang via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125428 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/fs/HarFileSystem.java Cache HAR filesystem metadata - Key: MAPREDUCE-2459 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2459 Project: Hadoop Map/Reduce Issue Type: Improvement Components: harchive Reporter: Mac Yang Assignee: Mac Yang Fix For: 0.23.0 Attachments: MAPREDUCE-2459.1.patch, MAPREDUCE-2459.2.patch Each HAR file system has two index files that contains information on how files are stored in the part files. During the block location calculation, these indexes are reread for every file in the archive. Caching the indexes and the status of the part files will greatly reduce the number of name node operations during the job setup time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2483) Clean up duplication of dependent jar files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036892#comment-13036892 ] Hudson commented on MAPREDUCE-2483: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects from build artifacts. (Eric Yang via omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125017 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/ivy.xml * /hadoop/mapreduce/trunk/src/contrib/mumak/build.xml * /hadoop/mapreduce/trunk/build.xml Clean up duplication of dependent jar files --- Key: MAPREDUCE-2483 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2483 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.23.0 Attachments: MAPREDUCE-2483.patch For trunk, the build and deployment tree look like this: hadoop-common-0.2x.y hadoop-hdfs-0.2x.y hadoop-mapred-0.2x.y Technically, mapred's the third party dependent jar files should be fetch from hadoop-common and hadoop-hdfs. However, it is currently fetching from hadoop-mapred/lib only. It would be nice to eliminate the need to repeat duplicated jar files at build time. There are two options to manage this dependency list, continue to enhance ant build structure to fetch and filter jar file dependencies using ivy. On the other hand, it would be a good opportunity to convert the build structure to maven, and use maven to manage the provided jar files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2372) TaskLogAppender mechanism shouldn't be set in log4j.properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036891#comment-13036891 ] Hudson commented on MAPREDUCE-2372: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2372. TaskLogAppender mechanism shouldn't be set up in log4j.properties. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125082 Files : * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/TaskLogAppender.java * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/TaskRunner.java TaskLogAppender mechanism shouldn't be set in log4j.properties -- Key: MAPREDUCE-2372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2372 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: mapreduce-2372.txt, mapreduce-2372.txt, mapreduce-2372.txt The TaskLogAppender log4j appender relies on using log4j.properties to pass in some Java system properties into properties of the logger. This is problematic since we've often found that users have customized log4j.properties and don't upgrade it when they upgrade the version of Hadoop. Since this is really an internal mechanism of how the task runner passes task info to the TLA, we shouldn't rely on these settings in log4j.properties at all. Rather, we should just get the system properties directly from System.getProperty. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2518) missing t flag in distcp help message '-p[rbugp]'
[ https://issues.apache.org/jira/browse/MAPREDUCE-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036893#comment-13036893 ] Hudson commented on MAPREDUCE-2518: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2518. The t flag is missing in distcp help message. Contributed by Wei Yongjun szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125148 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/tools/org/apache/hadoop/tools/DistCp.java missing t flag in distcp help message '-p[rbugp]' - Key: MAPREDUCE-2518 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2518 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.0 Reporter: Wei Yongjun Assignee: Wei Yongjun Fix For: 0.23.0 Attachments: MAPREDUCE-2518.patch 't: modification and access times' flag is defined but missing in distcp help message '-p[rbugp]'. should be changed to -p[rbugpt]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2516) option to control sensitive web actions
[ https://issues.apache.org/jira/browse/MAPREDUCE-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13036894#comment-13036894 ] Hudson commented on MAPREDUCE-2516: --- Integrated in Hadoop-Mapreduce-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/685/]) MAPREDUCE-2516. Rename webinterface.private.actions to mapreduce.jobtracker.webinterface.trusted. Contributed by Ari Rabkin. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125223 Files : * /hadoop/mapreduce/trunk/src/java/mapred-default.xml * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/server/jobtracker/JTConfig.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JSPUtil.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java option to control sensitive web actions --- Key: MAPREDUCE-2516 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2516 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Ari Rabkin Assignee: Ari Rabkin Priority: Minor Fix For: 0.22.0 Attachments: MAPREDUCE-2516.txt as per HADOOP-7302, webinterface.private.actions should not be in trunk. But it should be here, and should have a clearer name. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: (was: MAPREDUCE-2490-branch-0.20-security-v2.patch) Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: MAPREDUCE-2490-branch-0.20-security-v2.patch Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: MAPREDUCE-2490-trunk-v2.patch Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Attachment: MAPREDUCE-2490-branch-0.20-security-v2.patch Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Status: Open (was: Patch Available) Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2490: --- Status: Patch Available (was: Open) Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2297) All map reduce tasks are failing if we give invalid path jar file for Job
[
https://issues.apache.org/jira/browse/MAPREDUCE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037013#comment-13037013
]
Todd Lipcon commented on MAPREDUCE-2297:
bq. Consider the case of Hive, if we configure any invalid path for the
property hive.aux.jars.path all the jobs will fail which is not using that
jar also.
I would consider it a broken config if you configure hive.aux.jars.path to
point to a jar which doesn't exist.
In your patch, if you accidentally make a typo in your DistributedCache
entries, you'll see NoClassDefFound exceptions or other much scarier errors. I
think it's better to fail with the File does not exist error during
localization.
All map reduce tasks are failing if we give invalid path jar file for Job
-
Key: MAPREDUCE-2297
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2297
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.20.2
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Fix For: 0.20.4
Attachments: MAPREDUCE-2297.patch
This can be reproduced by giving the invalid jar file for the Job or it can
be reproduced from hive.
In hive-default.xml
property
namehive.aux.jars.path/name
value/value
descriptionProvided for adding auxillaryjarsPath/description
/property
If we configure an invalid path for jar file, It is making all map reduce
tasks to fail even those jobs are not depending on this jar file and it is
giving the below exception.
{code:xml}
hive select * from a join b on(a.b=b.c);
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number
In order to set a constant number of reducers:
set mapred.reduce.tasks=number
java.io.FileNotFoundException: File does not exist: /user/root/grade.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:495)
at
org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:651)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:783)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:698)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037027#comment-13037027 ] Hadoop QA commented on MAPREDUCE-2490: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479926/MAPREDUCE-2490-trunk-v2.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//console This message is automatically generated. Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037042#comment-13037042 ] Jonathan Eagles commented on MAPREDUCE-2490: This most recent patch version still logs the black/gray counts, but has also added some debug info at the caller to better aid in debug the root cause in the case for debugging black/gray list failures. The gridmix contrib test failure is unrelated to my change. No new unit tests were add due to the nature of this change (only affects logging). Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2526) Forward port MAPREDUCE-1966 Add task tracker graylisting
[ https://issues.apache.org/jira/browse/MAPREDUCE-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-2526: --- Affects Version/s: 0.23.0 Assignee: (was: Greg Roelofs) Issue Type: New Feature (was: Bug) Forward port MAPREDUCE-1966 Add task tracker graylisting Key: MAPREDUCE-2526 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2526 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Jonathan Eagles The current heuristic of rolling up fixed number of job failures per tracker isn't working well, we need better design/heuristics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2526) Forward port MAPREDUCE-1966 Add task tracker graylisting
Forward port MAPREDUCE-1966 Add task tracker graylisting Key: MAPREDUCE-2526 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2526 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Jonathan Eagles Assignee: Greg Roelofs The current heuristic of rolling up fixed number of job failures per tracker isn't working well, we need better design/heuristics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037056#comment-13037056 ] Jonathan Eagles commented on MAPREDUCE-2490: Filed MAPREDUCE-2526 Add task tracker graylisting to aid in the discussion to forward port task tracker graylisting to trunk to address comments above. Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2495) The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037080#comment-13037080 ] Robert Joseph Evans commented on MAPREDUCE-2495: Just like before. The contrib test issues are with RAID, and appear to be a known issue. The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason - Key: MAPREDUCE-2495 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2495 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Affects Versions: 0.21.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Minor Attachments: MAPREDUCE-2495-20.20X-V1.patch, MAPREDUCE-2495-20.20X-V2.patch, MAPREDUCE-2495-20.20X-V3.patch, MAPREDUCE-2495-v1.patch, MAPREDUCE-2495-v2.patch, MAPREDUCE-2495-v3.patch The cleanup thread in the distributed cache handles IOExceptions and the like correctly, but just to be a bit more defensive it would be good to monitor the thread, and check that it is still alive regularly, so that the distributed cache does not fill up the entire disk on the node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2470: - Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) bq. The difference is that with Job.getStatus it calls internally updateStatus which will throw an IOException if it is unable to get an updated status from the JobTracker. The only time you can get an NPE from that is if you catch the IOException, ignore it, and keep trying to use the Job object what was originally created. Got it. Thanks for the explanation. +1 I committed this. Thanks! Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Fix For: 0.23.0 Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2514) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2514: - Resolution: Fixed Fix Version/s: 0.23.0 0.20.205.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Jon! ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log Key: MAPREDUCE-2514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2514 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2514-branch-0.20-security.patch, MAPREDUCE-2514-trunk.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-2490: - Resolution: Fixed Fix Version/s: 0.23.0 0.20.205.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks, Jon! Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037148#comment-13037148 ] Todd Lipcon commented on MAPREDUCE-2470: This seems to have broken the MR trunk build: https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/691/ Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Fix For: 0.23.0 Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037161#comment-13037161 ] Chris Douglas commented on MAPREDUCE-2470: -- *sigh* Yea, it looks like the fault injection is broken. I'll revert it. Not sure why Hudson didn't flag that... Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Fix For: 0.23.0 Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas reopened MAPREDUCE-2470: -- Reverted while FI breakage is investigated Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Fix For: 0.23.0 Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2527) MR-279: Metrics for MRAppMaster
MR-279: Metrics for MRAppMaster --- Key: MAPREDUCE-2527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2527 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Fix For: 0.23.0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2527) MR-279: Metrics for MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-2527: --- Attachment: mr-2527-am-metrics-v1.patch MR-279: Metrics for MRAppMaster --- Key: MAPREDUCE-2527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2527 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Fix For: 0.23.0 Attachments: mr-2527-am-metrics-v1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2490) Log blacklist debug count
[ https://issues.apache.org/jira/browse/MAPREDUCE-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037182#comment-13037182 ] Hudson commented on MAPREDUCE-2490: --- Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/]) Log blacklist debug count - Key: MAPREDUCE-2490 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2490 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.204.0, 0.22.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2490-branch-0.20-security-v2.patch, MAPREDUCE-2490-branch-0.20-security.patch, MAPREDUCE-2490-trunk-v2.patch, MAPREDUCE-2490-trunk.patch Gain some insight into blacklist increments/decrements by enhancing the debug logging -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2514) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037183#comment-13037183 ] Hudson commented on MAPREDUCE-2514: --- Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/]) ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log Key: MAPREDUCE-2514 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2514 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Trivial Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2514-branch-0.20-security.patch, MAPREDUCE-2514-trunk.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037181#comment-13037181 ] Hudson commented on MAPREDUCE-2470: --- Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/]) Revert MAPREDUCE-2470 cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1125599 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/TestNetworkedJob.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/protocol/ClientProtocol.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/RunningJob.java * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JobClient.java Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff Assignee: Robert Joseph Evans Fix For: 0.23.0 Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, counters_null_data.pcap This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2407) Make Gridmix emulate usage of Distributed Cache files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037191#comment-13037191 ] Amar Kamat commented on MAPREDUCE-2407: --- The latest patch looks good to me. I have some minor comments (mostly alignment, refactoring and parameter naming) which I have discussed with Ravi offline. I don't want to block the patch just for some minor comments. +1. Make Gridmix emulate usage of Distributed Cache files - Key: MAPREDUCE-2407 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2407 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/gridmix Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: 2407.patch, 2407.v1.patch Currently Gridmix emulates disk IO load only. This JIRA is to make Gridmix emulate Distributed Cache load as defined by the job-trace. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2492) [MAPREDUCE] The new MapReduce API should make available task's progress to the task
[
https://issues.apache.org/jira/browse/MAPREDUCE-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13037231#comment-13037231
]
Chris Douglas commented on MAPREDUCE-2492:
--
Adding progress to {{TaskAttemptContext}} makes sense. Minor points on the
tests:
* Please use JUnit4 annotations instead of extending {{TestCase}} for new tests
* Using {{@BeforeClass}} instead of an instance initializer block would be
easier to read and would fail appropriately
* Deleting test.build.data (defaulting to /tmp) recursively when the test
runs could have bad side-effects. Please use a subdir of {{rootTestDir}} instead
* Cleaning up after the test would also be a good idea ({{@AfterClass}}). Right
now, many of the tests will not delete output when the test fails.
[MAPREDUCE] The new MapReduce API should make available task's progress to
the task
---
Key: MAPREDUCE-2492
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2492
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.23.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Attachments: MAPREDUCE-2492-v1.3.patch
There is no way to get the task's current progress in the new MapReduce API.
It would be nice to make it available so that the task (map/reduce) can use
it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
