[jira] Updated: (MAPREDUCE-1895) MapEventFetcherThread should not iterate over jobs that are not localized
[ https://issues.apache.org/jira/browse/MAPREDUCE-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1895: --- Description: We have seen a scenario of lost trackers on our clusters because of the following: TaskLauncher has locked a TaskTracker$RunningJob and doing localizeJob, which involves DFS operations. Map-event fetcher has locked TaskTracker.runningJobs map and is waiting to lock the RunningJob object. TaskTracker offerService is waiting to lock TaskTracker.runningJobs map, thus failing to send heartbeats in 10 minutes. So, I think map-event fetcher should skip jobs that are not localized. was: We have seen a scenario of lost trackers on our clusters because of the following: TaskLauncher has locked a TaskTracker$RunningJob and doing localizeJob, which involves DFS operations. Map-event fetcher has locked TaskTracker.runningJobs map and is waiting to lock the RunningJob object. TaskTracker offerService is waiting to lock TaskTracker.runningJobs map, thus failing to send heartbeats in 10 minutes. So, I think map-event fetcher should circuit jobs that are not localized. MapEventFetcherThread should not iterate over jobs that are not localized - Key: MAPREDUCE-1895 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1895 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Amareshwari Sriramadasu We have seen a scenario of lost trackers on our clusters because of the following: TaskLauncher has locked a TaskTracker$RunningJob and doing localizeJob, which involves DFS operations. Map-event fetcher has locked TaskTracker.runningJobs map and is waiting to lock the RunningJob object. TaskTracker offerService is waiting to lock TaskTracker.runningJobs map, thus failing to send heartbeats in 10 minutes. So, I think map-event fetcher should skip jobs that are not localized. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1850) Include job submit host information (name and ip) in jobconf and jobdetails display
[ https://issues.apache.org/jira/browse/MAPREDUCE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1850: Attachment: mapred-1850-5.patch ah! another inadvertent omission. Sorry. Think I got them all. Use of String constant has not been consistent though I have done my part. Include job submit host information (name and ip) in jobconf and jobdetails display --- Key: MAPREDUCE-1850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1850 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Krishna Ramachandran Assignee: Krishna Ramachandran Attachments: mapred-1850-1.patch, mapred-1850-2.patch, mapred-1850-3.patch, mapred-1850-4.patch, mapred-1850-5.patch, mapred-1850.patch, mapred-1850.patch Enhancement to identify the source (submit host and ip) of a job request. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1744) DistributedCache creates its own FileSytem instance when adding a file/archive to the path
[
https://issues.apache.org/jira/browse/MAPREDUCE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Ramachandran updated MAPREDUCE-1744:
Status: Resolved (was: Patch Available)
Resolution: Invalid
Marking it as invalid
should be addressed/fixed by
https://issues.apache.org/jira/browse/MAPREDUCE-950
DistributedCache creates its own FileSytem instance when adding a
file/archive to the path
--
Key: MAPREDUCE-1744
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1744
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Dick King
Assignee: Krishna Ramachandran
Attachments: BZ-3503564--2010-05-06.patch, h1744.patch,
mapred-1744-1.patch, mapred-1744-2.patch, mapred-1744-3.patch,
mapred-1744.patch, MAPREDUCE-1744.patch
According to the contract of {{UserGroupInformation.doAs()}} the only
required operations within the {{doAs()}} block are the
creation of a {{JobClient}} or getting a {{FileSystem}} .
The {{DistributedCache.add(File/Archive)ToClasspath()}} methods create a
{{FileSystem}} instance outside of the {{doAs()}} block,
this {{FileSystem}} instance is not in the scope of the proxy user but of the
superuser and permissions may make the method
fail.
One option is to overload the methods above to receive a filesystem.
Another option is to do obtain the {{FileSystem}} within a {{doAs()}} block,
for this it would be required to have the proxy
user set in the passed configuration.
The second option seems nicer, but I don't know if the proxy user is as a
property in the jobconf.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1850) Include job submit host information (name and ip) in jobconf and jobdetails display
[ https://issues.apache.org/jira/browse/MAPREDUCE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1850: --- Status: Patch Available (was: Open) Latest patch looks fine Include job submit host information (name and ip) in jobconf and jobdetails display --- Key: MAPREDUCE-1850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1850 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Krishna Ramachandran Assignee: Krishna Ramachandran Attachments: mapred-1850-1.patch, mapred-1850-2.patch, mapred-1850-3.patch, mapred-1850-4.patch, mapred-1850-5.patch, mapred-1850.patch, mapred-1850.patch Enhancement to identify the source (submit host and ip) of a job request. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882469#action_12882469 ] Hadoop QA commented on MAPREDUCE-1893: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448005/SliveMultiR.patch against trunk revision 957772. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/267/console This message is automatically generated. Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: SliveMultiR.patch Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882482#action_12882482
]
Amar Kamat commented on MAPREDUCE-1876:
---
Failures are mostly either timeouts or MiniDFS class not found or
DelegationToken errors. Doesnt seem related to this patch. Testing again on my
local box.
TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
type for reduce tasks
Key: MAPREDUCE-1876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Fix For: 0.22.0
Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
{{TaskAttemptStartedEvent}} is used to log the start time of both the map and
reduce task attempts to {{JobHistory}}. Following is the implementation of
_getEventType()_ method of {{TaskAttemptStartedEvent}}
{code}
/** Get the event type */
public EventType getEventType() {
return EventType.MAP_ATTEMPT_STARTED;
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1882) Use Jsch instead of Shell.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882487#action_12882487 ] Balaji Rajagopalan commented on MAPREDUCE-1882: --- Looks good, one other helper that we can add is scp, since Jsch supports that also. Use Jsch instead of Shell.java --- Key: MAPREDUCE-1882 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1882 Project: Hadoop Map/Reduce Issue Type: New Feature Components: test Environment: herriot framework Reporter: Balaji Rajagopalan Attachments: RemoteExecution.patch Original Estimate: 24h Remaining Estimate: 24h In herriot ( hadoop system test case dev) we often find that we are resorted to habit of ssh to remote node execute a shell command, and come out. It is wise to use Jsch instead of doing this through Shell.java ( hadoop code), since Jsch provides nice Java abstraction, the JIRA will only close after we import Jsch input hadoop build system and also fix all the existing test cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1888) Streaming overrides user given output key and value types.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-1888:
Attachment: 1888.patch
Attaching patch fixing the issue. Added testcase that validates the user set
config properties mapreduce.map.output.key.class and mapreduce.output.key.class
are honored in streaming jobs with java classes as mapper and reducer.
Streaming overrides user given output key and value types.
--
Key: MAPREDUCE-1888
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1888
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Assignee: Ravi Gummadi
Fix For: 0.22.0
Attachments: 1888.patch
The following code in StreamJob.java overrides user given output key and
value types.
{code}
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());
idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1890) Create automated test scenarios for decommissioning of task trackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882492#action_12882492 ] Balaji Rajagopalan commented on MAPREDUCE-1890: --- +//String confFile = mapred-site.xml; +//HashtableString,Object prop = new HashtableString,Object(); +//prop.put(mapred.hosts.exclude, /tmp/mapred.exclude); +//cluster.restartClusterWithNewConfig(prop, confFile); There is bunch of commented out code, it looks like you might need the above code to make is work correctly. Also the code in tear down is commented out. +ListTTClient ttClients = cluster.getTTClients(); + +//One slave is got +TTClient ttClient = (TTClient)ttClients.get(0); I have a similar helper method, that I will check in AbstractDaemonCluster with health script, it is better to use that helper, I will check in this code shortly. //The client shich needs tobe decommissioned is put in the exclude path. Fix the typo in the comment. One question is you were telling me that hadoopqa user is executing mradmin refresh which requires mapred priviledge, so how is this achieved wiith this code. If this is going to be deployment specific changes to mapred-site.xml we should call out that dependency, so it is actionable. Create automated test scenarios for decommissioning of task trackers Key: MAPREDUCE-1890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1890 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Iyappan Srinivasan Attachments: TestDecomissioning.patch Test scenarios : 1) Put a healthy slave task tracker in the dfs.exclude file. 2) As a valid user, decommission a node in the cluster by issuing the command hadoop mradmin -refreshNodes 3) Make sure that the node is decommissioned. 4) Now take the task tracker out of the file. 5) As a valid user, again issue the command hadoop mradmin -refreshNodes 6) Make sure that the node is not in the decommiossion list. 7) Bring back that node. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1896) [Herriot] New property for multi user list.
[Herriot] New property for multi user list. --- Key: MAPREDUCE-1896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1896 Project: Hadoop Map/Reduce Issue Type: Task Components: test Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Adding new property for multi user list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1896) [Herriot] New property for multi user list.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1896: - Attachment: MAPREDUCE-1896.patch Patch for trunk. [Herriot] New property for multi user list. --- Key: MAPREDUCE-1896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1896 Project: Hadoop Map/Reduce Issue Type: Task Components: test Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-1896.patch Adding new property for multi user list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882519#action_12882519
]
Amar Kamat commented on MAPREDUCE-1876:
---
All tests passed except TestSimulatorDeterministicReplay, which is a known
issue. Not sure why so many tests failed with hudson.
TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
type for reduce tasks
Key: MAPREDUCE-1876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Fix For: 0.22.0
Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
{{TaskAttemptStartedEvent}} is used to log the start time of both the map and
reduce task attempts to {{JobHistory}}. Following is the implementation of
_getEventType()_ method of {{TaskAttemptStartedEvent}}
{code}
/** Get the event type */
public EventType getEventType() {
return EventType.MAP_ATTEMPT_STARTED;
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1863) [Rumen] Null failedMapAttemptCDFs in job traces generated by Rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882537#action_12882537 ] Hadoop QA commented on MAPREDUCE-1863: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448019/rumen-npe-v1.1-bin.patch against trunk revision 957772. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/268/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/268/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/268/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/268/console This message is automatically generated. [Rumen] Null failedMapAttemptCDFs in job traces generated by Rumen -- Key: MAPREDUCE-1863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1863 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Amar Kamat Assignee: Amar Kamat Fix For: 0.22.0 Attachments: counters-test-trace.json.gz, dispatch-trace-output.json.gz, rumen-npe-v1.1-bin.patch, rumen-npe-v1.1.patch All the traces generated by Rumen for jobs having failed task attempts has null value for failedMapAttemptCDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1876) TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event type for reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated MAPREDUCE-1876:
---
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
I just committed this. Thanks Amar!
TaskAttemptStartedEvent.java incorrectly logs MAP_ATTEMPT_STARTED as event
type for reduce tasks
Key: MAPREDUCE-1876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1876
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Fix For: 0.22.0
Attachments: mapreduce-1876-v1.1.patch, mapreduce-1876-v1.2.patch
{{TaskAttemptStartedEvent}} is used to log the start time of both the map and
reduce task attempts to {{JobHistory}}. Following is the implementation of
_getEventType()_ method of {{TaskAttemptStartedEvent}}
{code}
/** Get the event type */
public EventType getEventType() {
return EventType.MAP_ATTEMPT_STARTED;
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1844) Tests failing with java.lang.NoClassDefFoundError
[ https://issues.apache.org/jira/browse/MAPREDUCE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved MAPREDUCE-1844. -- Resolution: Fixed Fixed by HDFS-1193 Tests failing with java.lang.NoClassDefFoundError - Key: MAPREDUCE-1844 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1844 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Amar Kamat Priority: Blocker Tests are failing with java.lang.NoClassDefFoundError (see http://pastebin.com/Y3E8iDw0). Steps to reproduce on trunk 1) Delete ~/.ivy2 2) checkout trunk 3) ant -Dtestcase=TestMRCLI run-test-mapred -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1840) [Gridmix] Exploit/Add security features in GridMix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882714#action_12882714 ] Chris Douglas commented on MAPREDUCE-1840: -- +1 on these changes. Could this link to the other, open JIRAs incorporated in this patch? [Gridmix] Exploit/Add security features in GridMix -- Key: MAPREDUCE-1840 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1840 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Affects Versions: 0.22.0 Reporter: Amar Kamat Assignee: Amar Kamat Fix For: 0.22.0 Attachments: mapreduce-gridmix-fp-v1.3.3.patch, mapreduce-gridmix-fp-v1.3.9.patch Use security information while replaying jobs in Gridmix. This includes - Support for multiple users - Submitting jobs as different users - Allowing usage of secure cluster (hdfs + mapreduce) - Support for multiple queues Other features include : - Support for sleep job - Support for load job + testcases for verifying all of the above changes -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1893) Multiple reducers for Slive
[ https://issues.apache.org/jira/browse/MAPREDUCE-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-1893: --- Attachment: SliveMultiR.patch Fixed the audit warning. I am going to commit it if there are no objections. Please feel free to review this. Multiple reducers for Slive --- Key: MAPREDUCE-1893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1893 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks, test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: SliveMultiR.patch, SliveMultiR.patch Slive currently uses single reducer. It could use multiple ones. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1850) Include job submit host information (name and ip) in jobconf and jobdetails display
[ https://issues.apache.org/jira/browse/MAPREDUCE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882716#action_12882716 ] Hadoop QA commented on MAPREDUCE-1850: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12448021/mapred-1850-5.patch against trunk revision 957897. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/269/console This message is automatically generated. Include job submit host information (name and ip) in jobconf and jobdetails display --- Key: MAPREDUCE-1850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1850 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Krishna Ramachandran Assignee: Krishna Ramachandran Attachments: mapred-1850-1.patch, mapred-1850-2.patch, mapred-1850-3.patch, mapred-1850-4.patch, mapred-1850-5.patch, mapred-1850.patch, mapred-1850.patch Enhancement to identify the source (submit host and ip) of a job request. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1845) FairScheduler.tasksToPeempt() can return negative number
[
https://issues.apache.org/jira/browse/MAPREDUCE-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882755#action_12882755
]
Scott Chen commented on MAPREDUCE-1845:
---
I have carefully read the code. I think this bug is not that easy to trigger.
Because when update tasksDueToMinShare and tasksDueToFairShare, tasksToPreempt
will check
{code}
curTime - sched.getLastTimeAtMinShare() minShareTimeout
curTime - sched.getLastTimeAtHalfFairShare() fairShareTimeout
{code}
So they may turn negative only when they got the timeout and over-scheduled at
the same time.
And this two thing must happen between update() and preemptIfNecessary():
{code}
update()
// spent lots of time here so we get timeout
// get over-scheduled tasks
preemptIfNecessary()
{code}
It is not likely to happen unless people set starving timeout to be very short.
But it is still good to fix it.
I will write a unit test to simulate this situation.
FairScheduler.tasksToPeempt() can return negative number
Key: MAPREDUCE-1845
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1845
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/fair-share
Affects Versions: 0.22.0
Reporter: Scott Chen
Assignee: Scott Chen
Fix For: 0.22.0
Attachments: MAPREDUCE-1845.20100717.txt
This method can return negative number. This will cause the preemption to
under-preempt.
The bug was discovered by Joydeep.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1897) trunk build broken on compile-mapred-test
trunk build broken on compile-mapred-test
-
Key: MAPREDUCE-1897
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1897
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Environment: RHEL4 Linux, Java 1.6.0_15-b03
Reporter: Greg Roelofs
...apparently. Fresh checkout of trunk (all three hadoop-*), build.properties
project.version fix, ant veryclean mvn-install of common, hdfs, and then
mapreduce:
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:52:
cannot access org.apache.hadoop.test.system.DaemonProtocol
[javac] class file for org.apache.hadoop.test.system.DaemonProtocol not
found
[javac] static class FakeJobTracker extends JobTracker {
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
non-static variable this cannot be referenced from a static context
[javac] this.trackers = tts;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
cannot find symbol
[javac] symbol : variable trackers
[javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
[javac] this.trackers = tts;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
cannot find symbol
[javac] symbol : method taskTrackers()
[javac] location: class
org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
[javac] taskTrackers().size() - getBlacklistedTrackerCount(),
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
cannot find symbol
[javac] symbol : method getBlacklistedTrackerCount()
[javac] location: class
org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
[javac] taskTrackers().size() - getBlacklistedTrackerCount(),
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:68:
cannot find symbol
[javac] symbol : method getBlacklistedTrackerCount()
[javac] location: class
org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
[javac] getBlacklistedTrackerCount(), 0, 0, 0, totalSlots/2,
totalSlots/2,
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:64:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
non-static variable this cannot be referenced from a static context
[javac] this.totalSlots = totalSlots;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
cannot find symbol
[javac] symbol : variable totalSlots
[javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
[javac] this.totalSlots = totalSlots;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:91:
establishFirstContact(org.apache.hadoop.mapred.JobTracker,java.lang.String) in
org.apache.hadoop.mapred.FakeObjectUtilities cannot be applied to
(org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker,java.lang.String)
[javac] FakeObjectUtilities.establishFirstContact(jobTracker, s);
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:170:
cannot find symbol
[javac] symbol : constructor
MyFakeJobInProgress(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker)
[javac] location: class
org.apache.hadoop.mapred.TestJobInProgress.MyFakeJobInProgress
[javac] MyFakeJobInProgress job1 = new MyFakeJobInProgress(conf,
jobTracker);
[javac]^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:185:
cannot find symbol
[javac] symbol : constructor
MyFakeJobInProgress(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker)
[javac] location: class
org.apache.hadoop.mapred.TestJobInProgress.MyFakeJobInProgress
[javac] MyFakeJobInProgress job2 = new MyFakeJobInProgress(conf,
[jira] Commented: (MAPREDUCE-1845) FairScheduler.tasksToPeempt() can return negative number
[ https://issues.apache.org/jira/browse/MAPREDUCE-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882768#action_12882768 ] Scott Chen commented on MAPREDUCE-1845: --- Update. Change the patch according to Matei's suggestions. FairScheduler.tasksToPeempt() can return negative number Key: MAPREDUCE-1845 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1845 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.22.0 Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1845-v2.txt, MAPREDUCE-1845.20100717.txt This method can return negative number. This will cause the preemption to under-preempt. The bug was discovered by Joydeep. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1897) trunk build broken on compile-mapred-test
[
https://issues.apache.org/jira/browse/MAPREDUCE-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882797#action_12882797
]
Greg Roelofs commented on MAPREDUCE-1897:
-
Still no joy after multiple rounds of clean(ish) builds (e.g., removal of all
of .ivy2; removal of build and build-fi in all three subdirs; removal of
.ivy2/cache/*org.apache.hadoop*). However, ant binary works, so I'm going to
ignore ant mvn-install for now.
trunk build broken on compile-mapred-test
-
Key: MAPREDUCE-1897
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1897
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Environment: RHEL4 Linux, Java 1.6.0_15-b03
Reporter: Greg Roelofs
...apparently. Fresh checkout of trunk (all three hadoop-*),
build.properties project.version fix, ant veryclean mvn-install of common,
hdfs, and then mapreduce:
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:52:
cannot access org.apache.hadoop.test.system.DaemonProtocol
[javac] class file for org.apache.hadoop.test.system.DaemonProtocol not
found
[javac] static class FakeJobTracker extends JobTracker {
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
non-static variable this cannot be referenced from a static context
[javac] this.trackers = tts;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
cannot find symbol
[javac] symbol : variable trackers
[javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
[javac] this.trackers = tts;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
cannot find symbol
[javac] symbol : method taskTrackers()
[javac] location: class
org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
[javac] taskTrackers().size() - getBlacklistedTrackerCount(),
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
cannot find symbol
[javac] symbol : method getBlacklistedTrackerCount()
[javac] location: class
org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
[javac] taskTrackers().size() - getBlacklistedTrackerCount(),
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:68:
cannot find symbol
[javac] symbol : method getBlacklistedTrackerCount()
[javac] location: class
org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
[javac] getBlacklistedTrackerCount(), 0, 0, 0, totalSlots/2,
totalSlots/2,
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:64:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
non-static variable this cannot be referenced from a static context
[javac] this.totalSlots = totalSlots;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
cannot find symbol
[javac] symbol : variable totalSlots
[javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
[javac] this.totalSlots = totalSlots;
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:91:
establishFirstContact(org.apache.hadoop.mapred.JobTracker,java.lang.String)
in org.apache.hadoop.mapred.FakeObjectUtilities cannot be applied to
(org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker,java.lang.String)
[javac] FakeObjectUtilities.establishFirstContact(jobTracker,
s);
[javac] ^
[javac]
/home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:170:
cannot find symbol
[javac] symbol : constructor
MyFakeJobInProgress(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker)
[javac] location: class
[jira] Commented: (MAPREDUCE-1791) Remote cluster control functionality needs JavaDocs improvement
[ https://issues.apache.org/jira/browse/MAPREDUCE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882808#action_12882808 ] Konstantin Boudnik commented on MAPREDUCE-1791: --- The contrib test is failing for 34 days and is clearly unrelated to this patch. Remote cluster control functionality needs JavaDocs improvement --- Key: MAPREDUCE-1791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1791 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: test Affects Versions: 0.21.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Attachments: mapreduce-1791.patch, mapreduce-1791.patch This is MR part of HADOOP-6752. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
