[jira] Commented: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860479#action_12860479 ] Hadoop QA commented on MAPREDUCE-1545: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12440269/mr-1545-trunk-v2.patch against trunk revision 937201. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/134/console This message is automatically generated. > Add 'first-task-launched' to job-summary > > > Key: MAPREDUCE-1545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Arun C Murthy >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-1545-trunk-v1.patch, mr-1545-trunk-v2.patch, > mr-1545-y20s-v1.patch, mr-1545-y20s-v2.patch, mr-1545-y20s-v3.patch > > > It would be useful to track 'first-task-launched' time to job-summary for > better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1718) job conf key for the services name of DelegationToken for HFTP url is constructed incorrectly in HFTPFileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated MAPREDUCE-1718: -- Attachment: MAPREDUCE-1718-BP20-2.patch moved utility method to SecurityUtil.java > job conf key for the services name of DelegationToken for HFTP url is > constructed incorrectly in HFTPFileSystem > --- > > Key: MAPREDUCE-1718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1718 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1718-BP20-1.patch, MAPREDUCE-1718-BP20-2.patch > > > the key (build in TokenCache) is hdfs.service.host_HOSTNAME.PORT, but > in HftpFileSystem it is sometimes built as hdfs.service.host_IP.PORT. > Fix. change it to always be IP. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1718) job conf key for the services name of DelegationToken for HFTP url is constructed incorrectly in HFTPFileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860461#action_12860461 ] Boris Shkolnik commented on MAPREDUCE-1718: --- shouldn't. it is used as a key, so it just needs to be the same in both cases. > job conf key for the services name of DelegationToken for HFTP url is > constructed incorrectly in HFTPFileSystem > --- > > Key: MAPREDUCE-1718 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1718 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: MAPREDUCE-1718-BP20-1.patch, MAPREDUCE-1718-BP20-2.patch > > > the key (build in TokenCache) is hdfs.service.host_HOSTNAME.PORT, but > in HftpFileSystem it is sometimes built as hdfs.service.host_IP.PORT. > Fix. change it to always be IP. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1641) Job submission should fail if same uri is added for mapred.cache.files and mapred.cache.archives
[ https://issues.apache.org/jira/browse/MAPREDUCE-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dick King updated MAPREDUCE-1641: - Attachment: duped-files-archives--off-0-20-101--2010-04-23--1819.patch This patch applies to 0.20.101, not to trunk, and needs to be forward ported. However, it is licensed. > Job submission should fail if same uri is added for mapred.cache.files and > mapred.cache.archives > > > Key: MAPREDUCE-1641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1641 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distributed-cache >Reporter: Amareshwari Sriramadasu >Assignee: Dick King > Fix For: 0.22.0 > > Attachments: BZ-3539321--off-0-20-101--2010-04-20.patch, > duped-files-archives--off-0-20-101--2010-04-21.patch, > duped-files-archives--off-0-20-101--2010-04-23--1819.patch > > > The behavior of mapred.cache.files and mapred.cache.archives is different > during localization in the following way: > If a jar file is added to mapred.cache.files, it will be localized under > TaskTracker under a unique path. > If a jar file is added to mapred.cache.archives, it will be localized under a > unique path in a directory named the jar file name, and will be unarchived > under the same directory. > If same jar file is passed for both the configurations, the behavior > undefined. Thus the job submission should fail. > Currently, since distributed cache processes files before archives, the jar > file will be just localized and not unarchived. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1545) Add 'first-task-launched' to job-summary
[ https://issues.apache.org/jira/browse/MAPREDUCE-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-1545: --- Status: Patch Available (was: Open) > Add 'first-task-launched' to job-summary > > > Key: MAPREDUCE-1545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1545 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Arun C Murthy >Assignee: Luke Lu > Fix For: 0.22.0 > > Attachments: mr-1545-trunk-v1.patch, mr-1545-trunk-v2.patch, > mr-1545-y20s-v1.patch, mr-1545-y20s-v2.patch, mr-1545-y20s-v3.patch > > > It would be useful to track 'first-task-launched' time to job-summary for > better reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1650) Exclude Private elements from generated MapReduce Javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860394#action_12860394 ] Hadoop QA commented on MAPREDUCE-1650: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442685/MAPREDUCE-1650.patch against trunk revision 937201. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/133/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/133/console This message is automatically generated. > Exclude Private elements from generated MapReduce Javadoc > - > > Key: MAPREDUCE-1650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1650 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1650.patch, MAPREDUCE-1650.patch, > MAPREDUCE-1650.patch > > > Exclude elements annotated with InterfaceAudience.Private or > InterfaceAudience.LimitedPrivate from Javadoc and JDiff. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1694) streaming documentation appears to be wrong on overriding settings w/-D
[ https://issues.apache.org/jira/browse/MAPREDUCE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved MAPREDUCE-1694. - Fix Version/s: 0.21.0 Resolution: Fixed Marking this as fixed in 0.21, hoping that it comes out sooner rather than later. > streaming documentation appears to be wrong on overriding settings w/-D > --- > > Key: MAPREDUCE-1694 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1694 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming, documentation >Affects Versions: 0.20.2 >Reporter: Allen Wittenauer > Fix For: 0.21.0 > > > Throughout http://hadoop.apache.org/common/docs/current/streaming.html , > there are many examples that do "hadoop jar streaming blah > -Dsomething=something". None of these examples appear to work anymore. > Moving the "-Dsomething=something" to be after "hadoop jar streaming" works. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1725) Fix MapReduce API incompatibilities between 0.20 and 0.21
Fix MapReduce API incompatibilities between 0.20 and 0.21 - Key: MAPREDUCE-1725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1725 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Reporter: Tom White Assignee: Tom White Priority: Blocker A few API compatibilities have crept in since 0.20 (they are being tracked in MAPREDUCE-1623). These should be fixed before 0.21 is released. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1568) TrackerDistributedCacheManager should do deleteLocalPath asynchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860354#action_12860354 ] Scott Chen commented on MAPREDUCE-1568: --- Amareshwari, Actually, I do not have a strong preference on whether to use another cleanup thread or not. I think it is more elegant because getLocalCache() will actually just do get local cache instead of checking and start the cleaning thread. But on the other hand the current change is simple and safe. What do you think? > TrackerDistributedCacheManager should do deleteLocalPath asynchronously > --- > > Key: MAPREDUCE-1568 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1568 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1568.txt > > > TrackerDistributedCacheManager.deleteCache() has been improved: > MAPREDUCE-1302 makes TrackerDistributedCacheManager rename the caches in the > main thread and then delete them in the background > MAPREDUCE-1098 avoids global locking while do the renaming (renaming lots of > directories can also takes a long time) > But the deleteLocalCache is still in the main thread of TaskRunner.run(). So > it will still slow down the task which triggers the deletion (originally this > will blocks all tasks, but it is fixed by MAPREDUCE-1098). Other tasks do not > wait for the deletion. The task which triggers the deletion should not wait > for this either. TrackerDistributedCacheManager should do deleteLocalPath() > asynchronously. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1568) TrackerDistributedCacheManager should do deleteLocalPath asynchronously
[
https://issues.apache.org/jira/browse/MAPREDUCE-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860350#action_12860350
]
Scott Chen commented on MAPREDUCE-1568:
---
Hey Amareshwari,
deleteCache will first get the global lock of all cache and put the one needs
with zero reference count in toBeDeleted (this is done by you guys in
MAPREDUCE-1098). And the asynchronous deletion will start from there.
When the deletion condition is valid, only one task will get the global lock
and after it comes out of the global lock the deletion condition will no longer
valid. So there cannot be two threads deleting same set of cache at the same
moment.
{code}
private void deleteCache(Configuration conf) throws IOException {
Collection toBeDeleted = new LinkedList();
synchronized (cachedArchives) { // Global lock of all caches
// Find cache Status with refcount of zero and put them in to toBeDeleted
}
// do the deletion asynchronously, after releasing the global lock
...
cacheFileCleaner.start();
}
{code}
A separate cleanup thread is another option. I think that will work fine as
well. But that will require more change. I think the good thing about the
current patch is that it is simple and safe.
> TrackerDistributedCacheManager should do deleteLocalPath asynchronously
> ---
>
> Key: MAPREDUCE-1568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1568
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.22.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1568.txt
>
>
> TrackerDistributedCacheManager.deleteCache() has been improved:
> MAPREDUCE-1302 makes TrackerDistributedCacheManager rename the caches in the
> main thread and then delete them in the background
> MAPREDUCE-1098 avoids global locking while do the renaming (renaming lots of
> directories can also takes a long time)
> But the deleteLocalCache is still in the main thread of TaskRunner.run(). So
> it will still slow down the task which triggers the deletion (originally this
> will blocks all tasks, but it is fixed by MAPREDUCE-1098). Other tasks do not
> wait for the deletion. The task which triggers the deletion should not wait
> for this either. TrackerDistributedCacheManager should do deleteLocalPath()
> asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1664) Job Acls affect Queue Acls
[
https://issues.apache.org/jira/browse/MAPREDUCE-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-1664:
Attachment: 1664.20S.3.4.patch
Attaching patch for earlier version of hadoop. Not for commit here.
> Job Acls affect Queue Acls
> --
>
> Key: MAPREDUCE-1664
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1664
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Affects Versions: 0.22.0
>Reporter: Ravi Gummadi
>Assignee: Ravi Gummadi
> Fix For: 0.22.0
>
> Attachments: 1664.20S.3.4.patch
>
>
> MAPREDUCE-1307 introduced job ACLs for securing job level operations. So in
> current trunk, queue ACLs and job ACLs are checked(with AND for both acls)
> for allowing job level operations. So for doing operations like killJob,
> killTask and setJobPriority user should be part of both
> mapred.queue.{queuename}.acl-administer-jobs and in
> mapreduce.job.acl-modify-job. This needs to change so that users who are part
> of mapred.queue.{queuename}.acl-administer-jobs will be able to do
> killJob,killTask,setJobPriority and users part of
> mapreduce.job.acl-modify-job will be able to do
> killJob,killTask,setJobPriority.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1612) job conf file is not accessible from job history web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860346#action_12860346 ] Hadoop QA commented on MAPREDUCE-1612: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442668/MR-1619-1.patch against trunk revision 937201. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/132/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/132/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/132/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/132/console This message is automatically generated. > job conf file is not accessible from job history web page > - > > Key: MAPREDUCE-1612 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1612 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: jobconf_history_jsp.fix.20S.patch, MR-1619-1.patch > > > Clicking on conf file link from job history web page is causing an NPE if > history file(and the job conf file) are stored on DFS. This NPE is from > jobconf_history.jsp because jobConf built from path on DFS is not having any > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1723) Capacity Scheduler should allow configuration of Map & Reduce task slots independently per queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860340#action_12860340 ] Allen Wittenauer commented on MAPREDUCE-1723: - FWIW, if pre-emption = kill rather than freeze, then I don't think that is efficient from a system throughput perspective. > Capacity Scheduler should allow configuration of Map & Reduce task slots > independently per queue > > > Key: MAPREDUCE-1723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1723 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/capacity-sched >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan > Fix For: 0.20.3 > > > The Capacity Scheduler allows configuration of percentage of task slots per > queue. We have a scenario in which our biggest queue (50% quota) has Jobs > with mainly Map tasks & we need to enforce strict capacity limits per queue > due to SLA requirements. So other smaller queues which require Reduce tasks > gets starved even though the Reduce slots are idle. The Grid can be more > efficiently utilized if Capacity Scheduler allows configuration of Map & > Reduce task slots capacity independently per queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1288) DistributedCache localizes only once per cache URI
[ https://issues.apache.org/jira/browse/MAPREDUCE-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860339#action_12860339 ] Allen Wittenauer commented on MAPREDUCE-1288: - That sounds like really bad behavior. Why should an old job fail because of what is, essentially, an external event? This still sounds like a blocker to me. > DistributedCache localizes only once per cache URI > -- > > Key: MAPREDUCE-1288 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1288 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security, tasktracker >Affects Versions: 0.21.0 >Reporter: Devaraj Das >Priority: Blocker > Fix For: 0.21.0 > > > As part of the file localization the distributed cache localizer creates a > copy of the file in the corresponding user's private directory. The > localization in DistributedCache assumes the key as the URI of the cachefile > and if it already exists in the map, the localization is not done again. This > means that another user cannot access the same distributed cache file. We > should change the key to include the username so that localization is done > for every user. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1687) Stress submission policy does not always stress the cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1687: - Attachment: mr-1687-yhadoop-20.1xx-20100423-2.patch Fixed a minor bug in StressReaderThread.run() where the condition of the while loop waiting for cluster to be underloaded is incorrect. > Stress submission policy does not always stress the cluster. > > > Key: MAPREDUCE-1687 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1687 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Reporter: Hong Tang > Attachments: mr-1687-yhadoop-20.1xx-20100416.patch, > mr-1687-yhadoop-20.1xx-20100423-2.patch, mr-1687-yhadoop-20.1xx-20100423.patch > > > Currently, the rough idea of stress submission policy is to continue > submitting jobs until the pending map tasks reach 2x of the cluster capacity. > This proves to be inadequate and we saw a large job could monopolize the > whole cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1650) Exclude Private elements from generated MapReduce Javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1650: - Attachment: MAPREDUCE-1650.patch > Exclude Private elements from generated MapReduce Javadoc > - > > Key: MAPREDUCE-1650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1650 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1650.patch, MAPREDUCE-1650.patch, > MAPREDUCE-1650.patch > > > Exclude elements annotated with InterfaceAudience.Private or > InterfaceAudience.LimitedPrivate from Javadoc and JDiff. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1650) Exclude Private elements from generated MapReduce Javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1650: - Status: Patch Available (was: In Progress) Fix Version/s: 0.21.0 > Exclude Private elements from generated MapReduce Javadoc > - > > Key: MAPREDUCE-1650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1650 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1650.patch, MAPREDUCE-1650.patch, > MAPREDUCE-1650.patch > > > Exclude elements annotated with InterfaceAudience.Private or > InterfaceAudience.LimitedPrivate from Javadoc and JDiff. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (MAPREDUCE-1650) Exclude Private elements from generated MapReduce Javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-1650 started by Tom White. > Exclude Private elements from generated MapReduce Javadoc > - > > Key: MAPREDUCE-1650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1650 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation >Reporter: Tom White >Assignee: Tom White > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1650.patch, MAPREDUCE-1650.patch, > MAPREDUCE-1650.patch > > > Exclude elements annotated with InterfaceAudience.Private or > InterfaceAudience.LimitedPrivate from Javadoc and JDiff. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860276#action_12860276 ] Hadoop QA commented on MAPREDUCE-1219: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442659/MR-1219-2.patch against trunk revision 937201. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/131/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/131/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/131/console This message is automatically generated. > JobTracker Metrics causes undue load on JobTracker > -- > > Key: MAPREDUCE-1219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Jothi Padmanabhan >Assignee: Sreekanth Ramakrishnan > Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, MR-1219-2.patch, > patch-1219-ydist.txt > > > JobTrackerMetricsInst.doUpdates updates job-level counters of all running > jobs into JobTracker's metrics causing very bad performance and hampers > heartbeats. Since Job level metrics are better served by JobHistory, it may > be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1397) NullPointerException observed during task failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860230#action_12860230
]
Hadoop QA commented on MAPREDUCE-1397:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442658/patch-1397-3.txt
against trunk revision 936166.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/130/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/130/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/130/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/130/console
This message is automatically generated.
> NullPointerException observed during task failures
> --
>
> Key: MAPREDUCE-1397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: patch-1397-1.txt, patch-1397-2.txt, patch-1397-3.txt,
> patch-1397-ydist.txt, patch-1397.txt
>
>
> In an environment where many jobs are killed simultaneously, NPEs are
> observed in the TT/JT logs when a task fails. The situation is aggravated
> when the taskcontroller.cfg is not configured properly. Below is the
> exception obtained:
> {noformat}
> INFO org.apache.hadoop.mapred.TaskInProgress: Error from :
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:529)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:329)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:315)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:146)
> at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:109)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:502)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1612) job conf file is not accessible from job history web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1612: -- Attachment: MR-1619-1.patch Attaching patch for trunk. > job conf file is not accessible from job history web page > - > > Key: MAPREDUCE-1612 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1612 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: jobconf_history_jsp.fix.20S.patch, MR-1619-1.patch > > > Clicking on conf file link from job history web page is causing an NPE if > history file(and the job conf file) are stored on DFS. This NPE is from > jobconf_history.jsp because jobConf built from path on DFS is not having any > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1612) job conf file is not accessible from job history web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1612: -- Status: Patch Available (was: Open) > job conf file is not accessible from job history web page > - > > Key: MAPREDUCE-1612 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1612 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: jobconf_history_jsp.fix.20S.patch, MR-1619-1.patch > > > Clicking on conf file link from job history web page is causing an NPE if > history file(and the job conf file) are stored on DFS. This NPE is from > jobconf_history.jsp because jobConf built from path on DFS is not having any > properties. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1724) JobTracker balks at empty String for locations
JobTracker balks at empty String for locations -- Key: MAPREDUCE-1724 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1724 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.1 Environment: CDH 2 hadoop-0.20.1+169.68 Reporter: Craig Macdonald Priority: Minor If a split has locations which are "" (empty String), then the JobTracker will get upset during initialization: 2010-04-22 19:09:20,395 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.charAt(String.java:687) at org.apache.hadoop.net.NetUtils.normalizeHostName(NetUtils.java:420) at org.apache.hadoop.net.NetUtils.normalizeHostNames(NetUtils.java:443) at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:42) at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2411) at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:360) at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:462) at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3183) at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:637) Two key points: * This is different from Hadoop 0.18 * CombineFileSplit has a constructor where String[] location is not specified, and hence the location array is populated with empty Strings. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1693) Process tree clean up of either a failed task or killed task tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1693: - Attachment: taskchildskilling_1693.patch Method name changes in utility class based on Iyappan review comments and this patch got effected.So uploading the latest path. > Process tree clean up of either a failed task or killed task tests. > --- > > Key: MAPREDUCE-1693 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1693 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: taskchildskilling_1693.diff, > taskchildskilling_1693.diff, taskchildskilling_1693.patch, > taskchildskilling_1693.patch, taskchildskilling_1693.patch, > taskchildskilling_1693.patch, taskchildskilling_1693.patch > > > The following scenarios covered in the test. > 1. Run a job which spawns subshells in the tasks. Kill one of the task. All > the child process of the killed task must be killed. > 2. Run a job which spawns subshells in tasks. Fail one of the task. All the > child process of the killed task must be killed along with the task after its > failure. > 3. Check process tree cleanup on paritcular task-tracker when we use > -kill-task and -fail-task with both map and reduce. > 4. Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Let the job complete . Check if all the > child processes are killed, the overall job should fail. > l)Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Kill/fail the job while in progress. > Check if all the child processes are killed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1710) Process tree clean up of exceeding memory limit tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1710: - Attachment: memorylimittask_1710.patch method signatures changed in Utility class based on Iyappan review comments and and it effects to this patch.So uploading the new patch for it. > Process tree clean up of exceeding memory limit tasks. > -- > > Key: MAPREDUCE-1710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1710 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: memorylimittask_1710.patch, memorylimittask_1710.patch, > memorylimittask_1710.patch, memorylimittask_1710.patch > > > 1. Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Let the job complete . Check if all the > child processes are killed, the overall job should fail. > 2. Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Kill/fail the job while in progress. > Check if all the child processes are killed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1711) Gridmix should provide an option to submit jobs to the same queues as specified in the trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-1711: - Attachment: MR-1711-yhadoop-20-1xx-4.patch > Gridmix should provide an option to submit jobs to the same queues as > specified in the trace. > - > > Key: MAPREDUCE-1711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1711 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix >Reporter: Hong Tang > Attachments: MR-1711-yhadoop-20-1xx-2.patch, > MR-1711-yhadoop-20-1xx-3.patch, MR-1711-yhadoop-20-1xx-4.patch, > MR-1711-yhadoop-20-1xx.patch, mr-1711-yhadoop-20.1xx-20100416.patch > > > Gridmix should provide an option to submit jobs to the same queues as > specified in the trace. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1711) Gridmix should provide an option to submit jobs to the same queues as specified in the trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-1711: - Attachment: MR-1711-yhadoop-20-1xx-3.patch Implemented the changes suggested by hong > Gridmix should provide an option to submit jobs to the same queues as > specified in the trace. > - > > Key: MAPREDUCE-1711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1711 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix >Reporter: Hong Tang > Attachments: MR-1711-yhadoop-20-1xx-2.patch, > MR-1711-yhadoop-20-1xx-3.patch, MR-1711-yhadoop-20-1xx.patch, > mr-1711-yhadoop-20.1xx-20100416.patch > > > Gridmix should provide an option to submit jobs to the same queues as > specified in the trace. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1723) Capacity Scheduler should allow configuration of Map & Reduce task slots independently per queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860198#action_12860198 ] Subramaniam Krishnan commented on MAPREDUCE-1723: - Yes...I was just about to update my comment clarifying the same...:) > Capacity Scheduler should allow configuration of Map & Reduce task slots > independently per queue > > > Key: MAPREDUCE-1723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1723 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/capacity-sched >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan > Fix For: 0.20.3 > > > The Capacity Scheduler allows configuration of percentage of task slots per > queue. We have a scenario in which our biggest queue (50% quota) has Jobs > with mainly Map tasks & we need to enforce strict capacity limits per queue > due to SLA requirements. So other smaller queues which require Reduce tasks > gets starved even though the Reduce slots are idle. The Grid can be more > efficiently utilized if Capacity Scheduler allows configuration of Map & > Reduce task slots capacity independently per queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1723) Capacity Scheduler should allow configuration of Map & Reduce task slots independently per queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860197#action_12860197 ] Hemanth Yamijala commented on MAPREDUCE-1723: - bq. the proportion of Map & Reduce slots in our cluster would ideally be 3:4 You mean, 3 maps out of 4 slots ? > Capacity Scheduler should allow configuration of Map & Reduce task slots > independently per queue > > > Key: MAPREDUCE-1723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1723 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/capacity-sched >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan > Fix For: 0.20.3 > > > The Capacity Scheduler allows configuration of percentage of task slots per > queue. We have a scenario in which our biggest queue (50% quota) has Jobs > with mainly Map tasks & we need to enforce strict capacity limits per queue > due to SLA requirements. So other smaller queues which require Reduce tasks > gets starved even though the Reduce slots are idle. The Grid can be more > efficiently utilized if Capacity Scheduler allows configuration of Map & > Reduce task slots capacity independently per queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1723) Capacity Scheduler should allow configuration of Map & Reduce task slots independently per queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860195#action_12860195 ] Subramaniam Krishnan commented on MAPREDUCE-1723: - Hemanth, the proportion of Map & Reduce slots in our cluster would ideally be 3:4 (tending to even 4:5) but right now we have configured it at 2:3 to mitigate the effect described above. > Capacity Scheduler should allow configuration of Map & Reduce task slots > independently per queue > > > Key: MAPREDUCE-1723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1723 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/capacity-sched >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan > Fix For: 0.20.3 > > > The Capacity Scheduler allows configuration of percentage of task slots per > queue. We have a scenario in which our biggest queue (50% quota) has Jobs > with mainly Map tasks & we need to enforce strict capacity limits per queue > due to SLA requirements. So other smaller queues which require Reduce tasks > gets starved even though the Reduce slots are idle. The Grid can be more > efficiently utilized if Capacity Scheduler allows configuration of Map & > Reduce task slots capacity independently per queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1695) capacity scheduler is not included in findbugs/javadoc targets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1695: Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Hong ! > capacity scheduler is not included in findbugs/javadoc targets > -- > > Key: MAPREDUCE-1695 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1695 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1695-2.patch, mapreduce-1695-20100420-2.patch, > mapreduce-1695-20100420-2.patch, mapreduce-1695-20100420.patch, > mapreduce-1695-20100421.patch, mapreduce-1695-20100422.patch, > MAPREDUCE-1695-3.patch, MAPREDUCE-1695.patch, > mr1695-hadoop-findbugs-report-1.html, mr1695-hadoop-findbugs-report-2.html > > > Capacity Scheduler is not included in findbugs/javadoc targets. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1695) capacity scheduler is not included in findbugs/javadoc targets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860184#action_12860184 ] Hemanth Yamijala commented on MAPREDUCE-1695: - +1. Looks good to me. This is ready to go. > capacity scheduler is not included in findbugs/javadoc targets > -- > > Key: MAPREDUCE-1695 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1695 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched >Reporter: Hong Tang >Assignee: Hong Tang > Attachments: MAPREDUCE-1695-2.patch, mapreduce-1695-20100420-2.patch, > mapreduce-1695-20100420-2.patch, mapreduce-1695-20100420.patch, > mapreduce-1695-20100421.patch, mapreduce-1695-20100422.patch, > MAPREDUCE-1695-3.patch, MAPREDUCE-1695.patch, > mr1695-hadoop-findbugs-report-1.html, mr1695-hadoop-findbugs-report-2.html > > > Capacity Scheduler is not included in findbugs/javadoc targets. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860180#action_12860180 ] Amareshwari Sriramadasu commented on MAPREDUCE-1219: +1 patch looks good. > JobTracker Metrics causes undue load on JobTracker > -- > > Key: MAPREDUCE-1219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Jothi Padmanabhan >Assignee: Sreekanth Ramakrishnan > Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, MR-1219-2.patch, > patch-1219-ydist.txt > > > JobTrackerMetricsInst.doUpdates updates job-level counters of all running > jobs into JobTracker's metrics causing very bad performance and hampers > heartbeats. Since Job level metrics are better served by JobHistory, it may > be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1219: -- Status: Patch Available (was: Open) > JobTracker Metrics causes undue load on JobTracker > -- > > Key: MAPREDUCE-1219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Jothi Padmanabhan >Assignee: Sreekanth Ramakrishnan > Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, MR-1219-2.patch, > patch-1219-ydist.txt > > > JobTrackerMetricsInst.doUpdates updates job-level counters of all running > jobs into JobTracker's metrics causing very bad performance and hampers > heartbeats. Since Job level metrics are better served by JobHistory, it may > be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1219: -- Status: Open (was: Patch Available) > JobTracker Metrics causes undue load on JobTracker > -- > > Key: MAPREDUCE-1219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Jothi Padmanabhan >Assignee: Sreekanth Ramakrishnan > Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, MR-1219-2.patch, > patch-1219-ydist.txt > > > JobTrackerMetricsInst.doUpdates updates job-level counters of all running > jobs into JobTracker's metrics causing very bad performance and hampers > heartbeats. Since Job level metrics are better served by JobHistory, it may > be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreekanth Ramakrishnan updated MAPREDUCE-1219: -- Attachment: MR-1219-2.patch Incorporating Amareshwaris comment. > JobTracker Metrics causes undue load on JobTracker > -- > > Key: MAPREDUCE-1219 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Jothi Padmanabhan >Assignee: Sreekanth Ramakrishnan > Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, MR-1219-2.patch, > patch-1219-ydist.txt > > > JobTrackerMetricsInst.doUpdates updates job-level counters of all running > jobs into JobTracker's metrics causing very bad performance and hampers > heartbeats. Since Job level metrics are better served by JobHistory, it may > be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1397) NullPointerException observed during task failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1397:
---
Attachment: patch-1397-3.txt
Patch for trunk with changes in testcase.
> NullPointerException observed during task failures
> --
>
> Key: MAPREDUCE-1397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: patch-1397-1.txt, patch-1397-2.txt, patch-1397-3.txt,
> patch-1397-ydist.txt, patch-1397.txt
>
>
> In an environment where many jobs are killed simultaneously, NPEs are
> observed in the TT/JT logs when a task fails. The situation is aggravated
> when the taskcontroller.cfg is not configured properly. Below is the
> exception obtained:
> {noformat}
> INFO org.apache.hadoop.mapred.TaskInProgress: Error from :
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:529)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:329)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:315)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:146)
> at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:109)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:502)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1397) NullPointerException observed during task failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1397:
---
Status: Patch Available (was: Open)
> NullPointerException observed during task failures
> --
>
> Key: MAPREDUCE-1397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: patch-1397-1.txt, patch-1397-2.txt, patch-1397-3.txt,
> patch-1397-ydist.txt, patch-1397.txt
>
>
> In an environment where many jobs are killed simultaneously, NPEs are
> observed in the TT/JT logs when a task fails. The situation is aggravated
> when the taskcontroller.cfg is not configured properly. Below is the
> exception obtained:
> {noformat}
> INFO org.apache.hadoop.mapred.TaskInProgress: Error from :
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:529)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:329)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:315)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:146)
> at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:109)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:502)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1397) NullPointerException observed during task failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1397:
---
Attachment: patch-1397-ydist.txt
Patch for yahoo! distribution with a couple of bug fixes in test case.
> NullPointerException observed during task failures
> --
>
> Key: MAPREDUCE-1397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: patch-1397-1.txt, patch-1397-2.txt,
> patch-1397-ydist.txt, patch-1397.txt
>
>
> In an environment where many jobs are killed simultaneously, NPEs are
> observed in the TT/JT logs when a task fails. The situation is aggravated
> when the taskcontroller.cfg is not configured properly. Below is the
> exception obtained:
> {noformat}
> INFO org.apache.hadoop.mapred.TaskInProgress: Error from :
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:529)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:329)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:315)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:146)
> at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:109)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:502)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1397) NullPointerException observed during task failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1397:
---
Status: Open (was: Patch Available)
> NullPointerException observed during task failures
> --
>
> Key: MAPREDUCE-1397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: patch-1397-1.txt, patch-1397-2.txt, patch-1397.txt
>
>
> In an environment where many jobs are killed simultaneously, NPEs are
> observed in the TT/JT logs when a task fails. The situation is aggravated
> when the taskcontroller.cfg is not configured properly. Below is the
> exception obtained:
> {noformat}
> INFO org.apache.hadoop.mapred.TaskInProgress: Error from :
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:529)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:329)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:315)
> at
> org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:146)
> at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:109)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:502)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1225) TT successfully localizes a task even though the corresponding cache-file has already changed on DFS.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860160#action_12860160 ] Hadoop QA commented on MAPREDUCE-1225: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442402/MAPREDUCE-1225.patch against trunk revision 936166. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/console This message is automatically generated. > TT successfully localizes a task even though the corresponding cache-file has > already changed on DFS. > - > > Key: MAPREDUCE-1225 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1225 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0, 0.22.0 >Reporter: Vinod K V > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1225.patch, MAPREDUCE-1225.patch, > MAPREDUCE-1225.patch, MAPREDUCE-1225.patch > > > This happens with the first task of a job being localized on this TT. TT > doesn't check if the file on DFS is fresh according to the timestamps set in > job-conf during submission. After the first task incorrectly gets localized, > all further tasks fail on this TT as expected. > Found this issue while trying to improve test-case for MAPREUCE-913. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1687) Stress submission policy does not always stress the cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1687: - Attachment: mr-1687-yhadoop-20.1xx-20100423.patch Patch that fine tunes the overload determination heuristics. > Stress submission policy does not always stress the cluster. > > > Key: MAPREDUCE-1687 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1687 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Reporter: Hong Tang > Attachments: mr-1687-yhadoop-20.1xx-20100416.patch, > mr-1687-yhadoop-20.1xx-20100423.patch > > > Currently, the rough idea of stress submission policy is to continue > submitting jobs until the pending map tasks reach 2x of the cluster capacity. > This proves to be inadequate and we saw a large job could monopolize the > whole cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1711) Gridmix should provide an option to submit jobs to the same queues as specified in the trace.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860152#action_12860152
]
Hong Tang commented on MAPREDUCE-1711:
--
{noformat}
+assertEquals(" Improper queue for " + job.getJobName() + " " ,
+ job.getConfiguration().get("mapred.job.queue.name"),
+ "q" + (Integer.valueOf(job.getJobName().substring(
+ job.getJobName().length() - 1))+1));
{noformat}
This code is fragile, why not using GridmixJob.getJobSeqId(job)?
> Gridmix should provide an option to submit jobs to the same queues as
> specified in the trace.
> -
>
> Key: MAPREDUCE-1711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1711
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
>Reporter: Hong Tang
> Attachments: MR-1711-yhadoop-20-1xx-2.patch,
> MR-1711-yhadoop-20-1xx.patch, mr-1711-yhadoop-20.1xx-20100416.patch
>
>
> Gridmix should provide an option to submit jobs to the same queues as
> specified in the trace.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1711) Gridmix should provide an option to submit jobs to the same queues as specified in the trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860150#action_12860150 ] Hong Tang commented on MAPREDUCE-1711: -- - We should generate data only once. - We may want to increase the # of jobs submitted more than 3. To avoid lengthening too much of unit test run time, we should consider running sleep jobs, or processing less bytes in that specific case. - There is another problem I missed which is that JobQueueTaskScheduler does not really understand the notion of queues while we may not be able to have the unit test depending on capacity scheduler. This issue is probably outside the scope of this jira, though. > Gridmix should provide an option to submit jobs to the same queues as > specified in the trace. > - > > Key: MAPREDUCE-1711 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1711 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix >Reporter: Hong Tang > Attachments: MR-1711-yhadoop-20-1xx-2.patch, > MR-1711-yhadoop-20-1xx.patch, mr-1711-yhadoop-20.1xx-20100416.patch > > > Gridmix should provide an option to submit jobs to the same queues as > specified in the trace. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1723) Capacity Scheduler should allow configuration of Map & Reduce task slots independently per queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860147#action_12860147 ] Hemanth Yamijala commented on MAPREDUCE-1723: - Hmm. In HADOOP-3445 (God, I am surprised I still remember the number, *smile*) which introduced the capacity scheduler, Vivek had argued to have separate percentages for map and reduce capacities. At the time though, consensus drove towards having a single number. I think a big factor driving that decision was the absence of limits and presence of pre-emption. At that time, queues could not impose limits and hence spare capacity could be always used elsewhere; and pre-emption was meant to ensure that queues could get their 'guaranteed' capacity when required. With time, limits have come in and pre-emption has gone out. There is this valid use case that has come up. To me it seems like there are two ways to approach this problem. One is to do the enhancement proposed in the JIRA. Two is to re-introduce pre-emption. Clearly the first option is simple and easy to understand; I can think of ways we can keep the spec and implementation simple for the default case and still support this special requirement. The only thing bothering me is that it seems to be handling a specific type of cluster setup (i.e. the kind of queue and job profile that is described). The second option is clearly quite complicated. But we've had repeated cases from people asking for pre-emption in the scheduler, and I think it is a topic that's going to die only when it gets implemented. *smile*. As a side note while we are still discussing this, Subramaniam, what is the proportion of map and reduce slots in your cluster ? Are they the same ? > Capacity Scheduler should allow configuration of Map & Reduce task slots > independently per queue > > > Key: MAPREDUCE-1723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1723 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/capacity-sched >Affects Versions: 0.20.1 > Environment: all >Reporter: Subramaniam Krishnan > Fix For: 0.20.3 > > > The Capacity Scheduler allows configuration of percentage of task slots per > queue. We have a scenario in which our biggest queue (50% quota) has Jobs > with mainly Map tasks & we need to enforce strict capacity limits per queue > due to SLA requirements. So other smaller queues which require Reduce tasks > gets starved even though the Reduce slots are idle. The Grid can be more > efficiently utilized if Capacity Scheduler allows configuration of Map & > Reduce task slots capacity independently per queue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
