[jira] Commented: (MAPREDUCE-1208) TestKillSubProcesses fails with NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796577#action_12796577
]
Amar Kamat commented on MAPREDUCE-1208:
---
Any updates on this? Recently one of my runs failed with this NPE.
TestKillSubProcesses fails with NPE
---
Key: MAPREDUCE-1208
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1208
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Amar Kamat
Assignee: Ravi Gummadi
Fix For: 0.21.0
Here is the stack trace
{code}
junit.framework.AssertionFailedError: null
at
org.apache.hadoop.mapred.TestKillSubProcesses.validateKillingSubprocesses(TestKillSubProcesses.java:246)
at
org.apache.hadoop.mapred.TestKillSubProcesses.runKillingJobAndValidate(TestKillSubProcesses.java:98)
at
org.apache.hadoop.mapred.TestKillSubProcesses.runTests(TestKillSubProcesses.java:336)
at
org.apache.hadoop.mapred.TestKillSubProcesses.testJobKillFailAndSucceed(TestKillSubProcesses.java:320)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796590#action_12796590 ] Kay Kay commented on MAPREDUCE-1352: It seems like the artifacts published on https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ are *not* from branch-0.21 . TODO: * Checkin the patch to branch-0.21 * Republish hadoop-mapred and hadoop-mapred-test from the same. That should resolve the dynamic link issue faced by HBASE-1433 . Thanks for taking this on a priority since it is blocking the ivy configuration of hbase. 0.21.0 - snapshot incorrect dependency published in .pom files --- Key: MAPREDUCE-1352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: 0.21.0 Reporter: Kay Kay Assignee: Giridharan Kesavan Priority: Critical Fix For: 0.21.0 Attachments: mapreduce-1352.patch The snapshot available here at - https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ , has an incorrect dependency specified for hadoop-core ( in the .pom file ). The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in ivy/libaries.properties but pom.xml published in the repository refers to 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1316) JobTracker holds stale references to retired jobs via unreported tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1316: -- Attachment: mapreduce-1316-v1.7.patch Attaching a patch for trunk. The only change with this fix is that JobTracker.removeJobTasks() iterates over task-ids that the tip has scheduled so far instead of iterating over task statuses. This will take care of the corner case where the task is scheduled but hasn't returned with its status before the job retires. Added a testcase to check the same. test-patch and ant tests passed on my box. JobTracker holds stale references to retired jobs via unreported tasks --- Key: MAPREDUCE-1316 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1316 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Amar Kamat Assignee: Amar Kamat Priority: Blocker Attachments: mapreduce-1316-v1.7.patch JobTracker fails to remove _unreported_ tasks' mapping from _taskToTIPMap_ if the job finishes and retires. _Unreported tasks_ refers to tasks that were scheduled but the tasktracker did not report back with the task status. In such cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the job is gone leading to memory leak. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796599#action_12796599 ] Hadoop QA commented on MAPREDUCE-1186: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429424/patch-1186-4.txt against trunk revision 895914. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/355/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/355/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/355/console This message is automatically generated. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796606#action_12796606 ] Amareshwari Sriramadasu commented on MAPREDUCE-1186: The same test passed on my machine. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[
https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796605#action_12796605
]
Amareshwari Sriramadasu commented on MAPREDUCE-1186:
Test timeout for TestEmptyJob is not related to the patch.
Looking at the console log, I could see jetty problem. It timed out because
tracker could not come up (see HADOOP-4744). The corresponding console log:
{noformat}
[exec] [junit] 2010-01-05 07:00:33,128 INFO http.HttpServer
(HttpServer.java:start(435)) - Port returned by
webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the
listener on 0
[exec] [junit] 2010-01-05 07:00:33,128 INFO http.HttpServer
(HttpServer.java:start(440)) - listener.getLocalPort() returned 60202
webServer.getConnectors()[0].getLocalPort() returned 60202
[exec] [junit] 2010-01-05 07:00:33,129 INFO http.HttpServer
(HttpServer.java:start(473)) - Jetty bound to port 60202
[exec] [junit] 82779 [main] INFO org.mortbay.log - jetty-6.1.14
[exec] [junit] 2010-01-05 07:00:33,146 INFO net.NetworkTopology
(NetworkTopology.java:add(327)) - Adding a new node: /default-rack/host0.foo.com
[exec] [junit] 2010-01-05 07:00:33,148 INFO mapred.JobTracker
(JobTracker.java:addNewTracker(2180)) - Adding tracker
tracker_host0.foo.com:localhost/127.0.0.1:36038 to host host0.foo.com
[exec] [junit] 2010-01-05 07:00:33,150 ERROR mapred.TaskTracker
(TaskTracker.java:offerService(1360)) - Caught exception: java.io.IOException:
Jetty problem. Jetty didn't bind to a valid port
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.checkJettyPort(TaskTracker.java:1180)
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1338)
[exec] [junit] at
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2037)
[exec] [junit] at
org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:207)
[exec] [junit] at java.lang.Thread.run(Thread.java:619)
[exec] [junit]
[exec] [junit] 2010-01-05 07:00:33,152 INFO mapred.TaskTracker
(TaskTracker.java:run(755)) - Shutting down: Map-events fetcher for all reduce
tasks on tracker_host0.foo.com:localhost/127.0.0.1:36038
{noformat}
While localizing a DistributedCache file, TT sets permissions recursively on
the whole base-dir
---
Key: MAPREDUCE-1186
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.21.0
Reporter: Vinod K V
Assignee: Amareshwari Sriramadasu
Fix For: 0.22.0
Attachments: patch-1186-1.txt, patch-1186-2.txt,
patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt,
patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt
This is a performance problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Attachment: patch-899-1.txt Patch updated to trunk. Earlier patch had bug in LinuxTaskController tests, fixed it. Ran all LinuxTaskController tests and all of them passed (used available patches for MAPREDUCE-1322 and MAPREDUCE-1186 for failing tests) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured. --- Key: MAPREDUCE-899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Vinod K V Assignee: Vinod K V Fix For: 0.22.0 Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, patch-899.txt To enforce the accessibility of job files to only the job-owner and the TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid linux TaskController binary is group owned by a _special group_ to which only TaskTracker belongs and not just any group to which TT belongs. If the trust is broken, possibly due to misconfiguration by admins, the local files become accessible to unintended users, yet giving false sense of security to the admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Fix Version/s: 0.22.0 Assignee: Amareshwari Sriramadasu (was: Vinod K V) Status: Patch Available (was: Open) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured. --- Key: MAPREDUCE-899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, patch-899.txt To enforce the accessibility of job files to only the job-owner and the TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid linux TaskController binary is group owned by a _special group_ to which only TaskTracker belongs and not just any group to which TT belongs. If the trust is broken, possibly due to misconfiguration by admins, the local files become accessible to unintended users, yet giving false sense of security to the admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796660#action_12796660 ] Hadoop QA commented on MAPREDUCE-899: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429436/patch-899-1.txt against trunk revision 895914. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/356/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/356/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/356/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/356/console This message is automatically generated. When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured. --- Key: MAPREDUCE-899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: MAPREDUCE-899-20090828.txt, patch-899-1.txt, patch-899.txt To enforce the accessibility of job files to only the job-owner and the TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid linux TaskController binary is group owned by a _special group_ to which only TaskTracker belongs and not just any group to which TT belongs. If the trust is broken, possibly due to misconfiguration by admins, the local files become accessible to unintended users, yet giving false sense of security to the admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1224) Calling SELECT t.* from table AS t to get meta information is too expensive for big tables
[ https://issues.apache.org/jira/browse/MAPREDUCE-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796710#action_12796710 ] Hudson commented on MAPREDUCE-1224: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Calling SELECT t.* from table AS t to get meta information is too expensive for big tables -- Key: MAPREDUCE-1224 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1224 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Affects Versions: 0.20.1 Environment: all platforms, generic jdbc driver Reporter: Spencer Ho Assignee: Spencer Ho Fix For: 0.22.0 Attachments: MAPREDUCE-1224.patch, SqlManager.java The SqlManager uses the query, SELECT t.* from table AS t to get table spec is too expensive for big tables, and it was called twice to generate column names and types. For tables that are big enough to be map-reduced, this is too expensive to make sqoop useful. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-967) TaskTracker does not need to fully unjar job jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796712#action_12796712 ] Hudson commented on MAPREDUCE-967: -- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) TaskTracker does not need to fully unjar job jars - Key: MAPREDUCE-967 URL: https://issues.apache.org/jira/browse/MAPREDUCE-967 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: mapreduce-967-branch-0.20.txt, mapreduce-967.txt, mapreduce-967.txt, mapreduce-967.txt, mapreduce-967.txt, mapreduce-967.txt, mapreduce-967.txt In practice we have seen some users submitting job jars that consist of 10,000+ classes. Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily avoided -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1009) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796711#action_12796711 ] Hudson commented on MAPREDUCE-1009: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Forrest documentation needs to be updated to describes features provided for supporting hierarchical queues --- Key: MAPREDUCE-1009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1009 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 0.21.0 Reporter: Hemanth Yamijala Assignee: Vinod K V Priority: Blocker Fix For: 0.21.0 Attachments: MAPREDUCE-1009-20091008.txt, MAPREDUCE-1009-20091116.txt, MAPREDUCE-1009-20091124.txt, MAPREDUCE-1009-20091211.txt, MAPREDUCE-1009-20091217.txt, MAPREDUCE-1009-20091222.txt Forrest documentation must be updated for describing how to set up and use hierarchical queues in the framework and the capacity scheduler. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1222) [Mumak] We should not include nodes with numeric ips in cluster topology.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796713#action_12796713 ] Hudson commented on MAPREDUCE-1222: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) [Mumak] We should not include nodes with numeric ips in cluster topology. - Key: MAPREDUCE-1222 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1222 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/mumak Affects Versions: 0.21.0, 0.22.0 Reporter: Hong Tang Assignee: Hong Tang Fix For: 0.21.0 Attachments: IPv6-predicate.patch, mapreduce-1222-20091119.patch, mapreduce-1222-20091121.patch, mapreduce-1222-20091210.patch Rumen infers cluster topology by parsing input split locations from job history logs. Due to HDFS-778, a cluster node may appear both as a numeric ip or as a host name in job history logs. We should exclude nodes appeared as numeric ips in cluster toplogy when we run mumak until a solution is found so that numeric ips would never appear in input split locations. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1201) Make ProcfsBasedProcessTree collect CPU usage information
[ https://issues.apache.org/jira/browse/MAPREDUCE-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796715#action_12796715 ] Hudson commented on MAPREDUCE-1201: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Make ProcfsBasedProcessTree collect CPU usage information - Key: MAPREDUCE-1201 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1201 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.22.0 Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1201-v2.patch, MAPREDUCE-1201-v3-modified.patch, MAPREDUCE-1201-v3.patch, MAPREDUCE-1201.patch This information can be reported back to jobtracker to help profiling jobs and scheduling tasks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1241) JobTracker should not crash when mapred-queues.xml does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796717#action_12796717 ] Hudson commented on MAPREDUCE-1241: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) JobTracker should not crash when mapred-queues.xml does not exist - Key: MAPREDUCE-1241 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1241 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Owen O'Malley Assignee: Todd Lipcon Priority: Blocker Fix For: 0.21.0 Attachments: mapreduce-1241.txt, mapreduce-1241.txt Currently, if you bring up the JobTracker on an old configuration directory, it gets a NullPointerException looking for the mapred-queues.xml file. It should just assume a default queue and continue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1174) Sqoop improperly handles table/column names which are reserved sql words
[
https://issues.apache.org/jira/browse/MAPREDUCE-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796716#action_12796716
]
Hudson commented on MAPREDUCE-1174:
---
Integrated in Hadoop-Mapreduce-trunk #196 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
Sqoop improperly handles table/column names which are reserved sql words
Key: MAPREDUCE-1174
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1174
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Fix For: 0.22.0
Attachments: MAPREDUCE-1174.2.patch, MAPREDUCE-1174.3.patch,
MAPREDUCE-1174.4.patch, MAPREDUCE-1174.patch
In some databases it is legal to name tables and columns with terms that
overlap SQL reserved keywords (e.g., {{CREATE}}, {{table}}, etc.). In such
cases, the database allows you to escape the table and column names. We
should always escape table and column names when possible.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-181) Secure job submission
[
https://issues.apache.org/jira/browse/MAPREDUCE-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796718#action_12796718
]
Hudson commented on MAPREDUCE-181:
--
Integrated in Hadoop-Mapreduce-trunk #196 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
Secure job submission
--
Key: MAPREDUCE-181
URL: https://issues.apache.org/jira/browse/MAPREDUCE-181
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Amar Kamat
Assignee: Devaraj Das
Fix For: 0.22.0
Attachments: 181-1.patch, 181-2.patch, 181-3.patch, 181-3.patch,
181-4.patch, 181-5.1.patch, 181-5.1.patch, 181-6.patch, 181-8.patch,
hadoop-3578-branch-20-example-2.patch, hadoop-3578-branch-20-example.patch,
HADOOP-3578-v2.6.patch, HADOOP-3578-v2.7.patch, MAPRED-181-v3.32.patch,
MAPRED-181-v3.8.patch
Currently the jobclient accesses the {{mapred.system.dir}} to add job
details. Hence the {{mapred.system.dir}} has the permissions of
{{rwx-wx-wx}}. This could be a security loophole where the job files might
get overwritten/tampered after the job submission.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1155) Streaming tests swallow exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796720#action_12796720 ] Hudson commented on MAPREDUCE-1155: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) . Streaming tests swallow exceptions. Contributed by Todd Lipcon. Streaming tests swallow exceptions -- Key: MAPREDUCE-1155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1155 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.22.0 Attachments: mapreduce-1155.patch, mapreduce-1155.patch, mapreduce-1155.txt, mapreduce-1155.txt Many of the streaming tests (including TestMultipleArchiveFiles) catch exceptions and print their stack trace rather than failing the job. This means that tests do not fail even when the job fails. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1265) Include tasktracker name in the task attempt error log
[ https://issues.apache.org/jira/browse/MAPREDUCE-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796719#action_12796719 ] Hudson commented on MAPREDUCE-1265: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Include tasktracker name in the task attempt error log -- Key: MAPREDUCE-1265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1265 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.22.0 Attachments: MAPREDUCE-1265-v2.patch, MAPREDUCE-1265.patch When task attempt receive an error, TaskInProgress will log the task attempt id and diagnosis string in the JobTracker log. Ex: 2009-xx-xx 23:50:45,994 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_2009__r_09_1: Error: java.lang.OutOfMemoryError: Java heap space 2009-xx-xx 22:53:53,146 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_2009__m_000478_0: Task attempt_2009__m_000478_0 failed to report status for 601 seconds. Killing! When we want to debug a machine for example, a node has been blacklisted in the past few days. We have to use the task attempt id to find the TT. This is not very convenient. It will be nice if we can also log the tasktracker which causes this error. This way we can just grep the hostname to quickly find all the relevant error message. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796721#action_12796721 ] Hudson commented on MAPREDUCE-698: -- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Per-pool task limits for the fair scheduler --- Key: MAPREDUCE-698 URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/fair-share Reporter: Matei Zaharia Assignee: Kevin Peterson Fix For: 0.22.0 Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, mapreduce-698-trunk-4.patch, mapreduce-698-trunk-5.patch, mapreduce-698-trunk.patch, mapreduce-698-trunk.patch The fair scheduler could use a way to cap the share of a given pool similar to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1326) fi tests don't use fi-site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796723#action_12796723 ] Hudson commented on MAPREDUCE-1326: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) fi tests don't use fi-site.xml --- Key: MAPREDUCE-1326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1326 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.22.0 Attachments: MAPREDUCE-1326.patch When fault injection framework was ported to the Mapreduce fi-site.xml is missed from the testing process. E.g. when the tests run they won't use FI configuration. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1146) Sqoop dependencies break Ecpilse build on Linux
[
https://issues.apache.org/jira/browse/MAPREDUCE-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796724#action_12796724
]
Hudson commented on MAPREDUCE-1146:
---
Integrated in Hadoop-Mapreduce-trunk #196 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
Sqoop dependencies break Ecpilse build on Linux
---
Key: MAPREDUCE-1146
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1146
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/sqoop
Environment: Linux, Sun JDK6
Reporter: Konstantin Boudnik
Assignee: Aaron Kimball
Fix For: 0.22.0
Attachments: MAPREDUCE-1146.2.patch, MAPREDUCE-1146.3.patch,
MAPREDUCE-1146.4.patch, MAPREDUCE-1146.patch
Under Linux there's the error in the Eclipse Problems view:
{noformat}
- com.sun.tools cannot be resolved at line 166 of
org.apache.hadoop.sqoop.orm.CompilationManager
{noformat}
The problem doesn't appear on MacOS though
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1148) SQL identifiers are a superset of Java identifiers
[
https://issues.apache.org/jira/browse/MAPREDUCE-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796722#action_12796722
]
Hudson commented on MAPREDUCE-1148:
---
Integrated in Hadoop-Mapreduce-trunk #196 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
SQL identifiers are a superset of Java identifiers
--
Key: MAPREDUCE-1148
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1148
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/sqoop
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Fix For: 0.22.0
Attachments: MAPREDUCE-1148.2.patch, MAPREDUCE-1148.3.patch,
MAPREDUCE-1148.patch
SQL identifiers can contain arbitrary characters, can start with numbers, can
be words like {{class}} which are reserved in Java, etc. If Sqoop uses these
names literally for class and field names then compilation errors can occur
in auto-generated classes. SQL identifiers need to be cleansed to map onto
Java identifiers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1301) TestDebugScriptWithLinuxTaskController fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796725#action_12796725 ] Hudson commented on MAPREDUCE-1301: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) TestDebugScriptWithLinuxTaskController fails - Key: MAPREDUCE-1301 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1301 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1301-1.txt, patch-1301.txt After MAPREDUCE:879, TestDebugScriptWithLinuxTaskController fails with following exception : java.lang.NullPointerException at org.apache.hadoop.mapred.TestTaskTrackerLocalization.getFilePermissionAttrs(TestTaskTrackerLocalization.java:274) at org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:294) at org.apache.hadoop.mapred.TestDebugScript.verifyDebugScriptOutput(TestDebugScript.java:162) at org.apache.hadoop.mapred.TestDebugScriptWithLinuxTaskController.testDebugScriptExecutionAsDifferentUser(TestDebugScriptWithLinuxTaskController.java:50) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1293) AutoInputFormat doesn't work with non-default FileSystems
[ https://issues.apache.org/jira/browse/MAPREDUCE-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796728#action_12796728 ] Hudson commented on MAPREDUCE-1293: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) AutoInputFormat doesn't work with non-default FileSystems - Key: MAPREDUCE-1293 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1293 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Fix For: 0.22.0 Attachments: MAPREDUCE-1293.txt AutoInputFormat uses the wrong FileSystem.get() method when getting a reference to a FileSystem object. AutoInputFormat gets the default FileSystem, so this method breaks if the InputSplit's path is pointing to a different FileSystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1213) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796732#action_12796732 ] Hudson commented on MAPREDUCE-1213: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) TaskTrackers restart is very slow because it deletes distributed cache directory synchronously -- Key: MAPREDUCE-1213 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Zheng Shao Fix For: 0.22.0 Attachments: MAPREDUCE-1213.1.patch, MAPREDUCE-1213.2.patch, MAPREDUCE-1213.3.patch, MAPREDUCE-1213.4.patch We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1143) runningMapTasks counter is not properly decremented in case of failed Tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796726#action_12796726 ] Hudson commented on MAPREDUCE-1143: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) runningMapTasks counter is not properly decremented in case of failed Tasks. Key: MAPREDUCE-1143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1143 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: rahul k singh Assignee: rahul k singh Priority: Blocker Fix For: 0.21.0 Attachments: MAPRED-1143-1.patch, MAPRED-1143-2.patch, MAPRED-1143-2.patch, MAPRED-1143-3.patch, MAPRED-1143-4.patch, MAPRED-1143-5.patch.txt, MAPRED-1143-6.patch, MAPRED-1143-7.patch, MAPRED-1143-v21.patch, MAPRED-1143-ydist-1.patch, MAPRED-1143-ydist-2.patch, MAPRED-1143-ydist-3.patch, MAPRED-1143-ydist-4.patch, MAPRED-1143-ydist-5.patch, MAPRED-1143-ydist-6.patch, MAPRED-1143-ydist-7.patch, MAPRED-1143-ydist-8.patch.txt, MAPRED-1143-ydist-9.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1124) TestGridmixSubmission fails sometimes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796727#action_12796727 ] Hudson commented on MAPREDUCE-1124: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) TestGridmixSubmission fails sometimes - Key: MAPREDUCE-1124 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1124 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.21.0 Reporter: Amareshwari Sriramadasu Assignee: Chris Douglas Fix For: 0.21.0 Attachments: M1124-0.patch, M1124-1.patch, M1124-2.patch, M1124-3.patch, M1124-4.patch, M1124-5.patch, M1124-y20-1.patch, M1124-y20.patch, TEST-org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.txt TestGridmixSubmission fails sometimes with following error : Mismatched output bytes 4547848/4561267 junit.framework.AssertionFailedError: Mismatched output bytes 4547848/4561267 at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263) at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1294) Build fails to pull latest hadoop-core-* artifacts
[ https://issues.apache.org/jira/browse/MAPREDUCE-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796729#action_12796729 ] Hudson commented on MAPREDUCE-1294: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Build fails to pull latest hadoop-core-* artifacts -- Key: MAPREDUCE-1294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1294 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Critical Fix For: 0.22.0 Attachments: MAPREDUCE-1294.patch This is the same as HDFS-825 for mapreduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1250) Refactor job token to use a common token interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796733#action_12796733 ] Hudson commented on MAPREDUCE-1250: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Refactor job token to use a common token interface -- Key: MAPREDUCE-1250 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1250 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security Reporter: Kan Zhang Assignee: Kan Zhang Fix For: 0.22.0 Attachments: m1250-09.patch, m1250-12.patch, m1250-14.patch The idea is to use a common token interface for both job token and delegation token (HADOOP-6373) so that the RPC layer that uses them don't have to differentiate them. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1209) Move common specific part of the test TestReflectionUtils out of mapred into common
[ https://issues.apache.org/jira/browse/MAPREDUCE-1209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796730#action_12796730 ] Hudson commented on MAPREDUCE-1209: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Move common specific part of the test TestReflectionUtils out of mapred into common --- Key: MAPREDUCE-1209 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1209 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: test Affects Versions: 0.21.0, 0.22.0 Reporter: Vinod K V Assignee: Todd Lipcon Priority: Blocker Fix For: 0.21.0, 0.22.0 Attachments: mapreduce-1209.txt, mapreduce-1209.txt As commented by Tom here (https://issues.apache.org/jira/browse/HADOOP-6230?focusedCommentId=12751058page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12751058), TestReflectionUtils has a single test testSetConf() to test backward compatibility of ReflectionUtils for JobConfigurable objects. TestReflectionUtils can be spilt into two tests - one on common and one in mapred - this single test may reside in mapred till the mapred package is removed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1083) Use the user-to-groups mapping service in the JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796731#action_12796731 ] Hudson commented on MAPREDUCE-1083: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Use the user-to-groups mapping service in the JobTracker - Key: MAPREDUCE-1083 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1083 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Reporter: Arun C Murthy Assignee: Boris Shkolnik Fix For: 0.22.0 Attachments: HADOOP-4656_mr.patch, MAPREDUCE-1083-2.patch, MAPREDUCE-1083-3.patch HADOOP-4656 introduces a user-to-groups mapping service on the server-side. The JobTracker should use this to map users to their groups rather than relying on the information passed by the client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1258) Fair scheduler event log not logging job info
[ https://issues.apache.org/jira/browse/MAPREDUCE-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796736#action_12796736 ] Hudson commented on MAPREDUCE-1258: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Fair scheduler event log not logging job info - Key: MAPREDUCE-1258 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1258 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.21.0 Reporter: Matei Zaharia Assignee: Matei Zaharia Priority: Minor Fix For: 0.21.0, 0.22.0 Attachments: mapreduce-1258-1.patch The MAPREDUCE-706 patch seems to have left an unfinished TODO in the Fair Scheduler - namely, in the dump() function for periodically dumping scheduler state to the event log, the part that dumps information about jobs is commented out. This makes the event log less useful than it was before. It should be fairly easy to update this part to use the new scheduler data structures (Schedulable etc) and print the data. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-896) Users can set non-writable permissions on temporary files for TT and can abuse disk usage.
[ https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796735#action_12796735 ] Hudson commented on MAPREDUCE-896: -- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Users can set non-writable permissions on temporary files for TT and can abuse disk usage. -- Key: MAPREDUCE-896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-896 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Ravi Gummadi Fix For: 0.22.0 Attachments: MR-896.patch, MR-896.v1.patch, MR-896.v2.patch, MR-896.v3.patch, MR-896.v4.patch, MR-896.v5.patch, MR-896.v6.patch, MR-896.v7.patch, MR-896.v8.patch, y896.v1.patch, y896.v2.1.fix.patch, y896.v2.1.fix.v1.patch, y896.v2.1.fix.v2.patch, y896.v2.1.patch, y896.v2.patch As of now, irrespective of the TaskController in use, TT itself does a full delete on local files created by itself or job tasks. This step, depending upon TT's umask and the permissions set by files by the user, for e.g in job-work/task-work or child.tmp directories, may or may not go through successful completion fully. Thus is left an opportunity for abusing disk space usage either accidentally or intentionally by TT/users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1084) Implementing aspects development and fault injeciton framework for MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796734#action_12796734 ] Hudson commented on MAPREDUCE-1084: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Implementing aspects development and fault injeciton framework for MapReduce Key: MAPREDUCE-1084 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1084 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, test Reporter: Konstantin Boudnik Assignee: Sreekanth Ramakrishnan Fix For: 0.22.0 Attachments: mapreduce-1084-1-withoutsvnexternals.patch, mapreduce-1084-1.patch, mapreduce-1084-2.patch, mapreduce-1084-3.patch, mapreduce-1084-5.patch, mapreduce-1084-6-withRandom.patch, mapreduce-1084-6.patch, mapreduce-1084-final.patch Similar to HDFS-435 and HADOOP-6204 this JIRA will track the introduction of injection framework for MapReduce. After HADOOP-6204 is in place this particular modification should be very trivial and would take importing (via svn:external) of src/test/build and some tweaking of the build.xml file -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1251) c++ utils doesn't compile
[ https://issues.apache.org/jira/browse/MAPREDUCE-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796737#action_12796737 ] Hudson commented on MAPREDUCE-1251: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) c++ utils doesn't compile - Key: MAPREDUCE-1251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1251 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1, 0.20.2, 0.21.0, 0.22.0 Environment: ubuntu karmic 64-bit Reporter: Eli Collins Assignee: Eli Collins Attachments: HDFS-790-1.patch, HDFS-790.patch, MR-1251.patch c++ utils doesn't compile on ubuntu karmic 64-bit. The latest patch for HADOOP-5611 needs to be applied first. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1165) SerialUtils.hh: __PRETTY_FUNCTION__ is a GNU extension and not portable
[ https://issues.apache.org/jira/browse/MAPREDUCE-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796738#action_12796738 ] Hudson commented on MAPREDUCE-1165: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) SerialUtils.hh: __PRETTY_FUNCTION__ is a GNU extension and not portable --- Key: MAPREDUCE-1165 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1165 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Assignee: Allen Wittenauer Fix For: 0.22.0 Attachments: MAPREREDUCE-1165.patch SerialUtils.hh uses __PRETTY_FUNCTION__ to print the name of the function during an assertion. That is a GNU extension and is not portable across compilers. [C99 defines __func__, which should probably be used instead.] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1331) TestMiniMRWithDFS.testWithDFSWithDefaultPort inadvertently mistyped
[ https://issues.apache.org/jira/browse/MAPREDUCE-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796740#action_12796740 ] Hudson commented on MAPREDUCE-1331: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) TestMiniMRWithDFS.testWithDFSWithDefaultPort inadvertently mistyped --- Key: MAPREDUCE-1331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1331 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Hemanth Yamijala Assignee: Devaraj Das Attachments: 1331.patch One of the patches, possibly MAPREDUCE-181, inadvertently mistyped TestMiniMRWithDFS.testWithDFSWithDefaultPort to tesWithDFSWithDefaultPort. As a result it wouldn't run as a JUnit test. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1284) TestLocalizationWithLinuxTaskController fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796742#action_12796742 ] Hudson commented on MAPREDUCE-1284: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) TestLocalizationWithLinuxTaskController fails - Key: MAPREDUCE-1284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1284 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker, test Affects Versions: 0.22.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.21.0 Attachments: MR-1284.patch With current trunk, the testcase TestLocalizationWithLinuxTaskController fails with an exit code of 139 from task-controller when doing INITIALIZE_USER -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-744) Support in DistributedCache to share cache files with other users after HADOOP-4493
[ https://issues.apache.org/jira/browse/MAPREDUCE-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796743#action_12796743 ] Hudson commented on MAPREDUCE-744: -- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Support in DistributedCache to share cache files with other users after HADOOP-4493 --- Key: MAPREDUCE-744 URL: https://issues.apache.org/jira/browse/MAPREDUCE-744 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Reporter: Vinod K V Assignee: Devaraj Das Fix For: 0.22.0 Attachments: 744-1.patch, 744-2.patch, 744-3.patch, 744-4.patch, 744-5.patch, 744-6.patch, 744-early.patch HADOOP-4493 aims to completely privatize the files distributed to TT via DistributedCache. This jira issues focuses on sharing some/all of these files with all other users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof can cause JobClient to report job failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796741#action_12796741
]
Hudson commented on MAPREDUCE-1131:
---
Integrated in Hadoop-Mapreduce-trunk #196 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
Using profilers other than hprof can cause JobClient to report job failure
--
Key: MAPREDUCE-1131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Fix For: 0.22.0
Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch,
MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch,
MAPREDUCE-1131.patch
If task profiling is enabled, the JobClient will download the {{profile.out}}
file created by the tasks under profile. If this causes an IOException, the
job is reported as a failure to the client, even though all the tasks
themselves may complete successfully. The expected result files are assumed
to be generated by hprof. Using the profiling system with other profilers
will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1286) Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796744#action_12796744 ] Hudson commented on MAPREDUCE-1286: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else Key: MAPREDUCE-1286 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.20.2 Environment: Linux F12; streaming Reporter: Yuri Pradkin Assignee: Yuri Pradkin Attachments: mr-1286 I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y -Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s... Since I don't see any reason to have these double quotes in the original code; I propose they're removed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1059) distcp can generate uneven map task assignments
[ https://issues.apache.org/jira/browse/MAPREDUCE-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796714#action_12796714 ] Hudson commented on MAPREDUCE-1059: --- Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/]) distcp can generate uneven map task assignments --- Key: MAPREDUCE-1059 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1059 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-1059.2.patch, MAPREDUCE-1059.3.patch, MAPREDUCE-1059.patch distcp writes out a SequenceFile containing the source files to transfer, and their sizes. Map tasks are created over spans of this file, representing files which each mapper should transfer. In practice, some transfer loads yield many empty map tasks and a few tasks perform the bulk of the work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1155) Streaming tests swallow exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796765#action_12796765 ] Tom White commented on MAPREDUCE-1155: -- Hudson's build is now passing: http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-trunk/196/ Streaming tests swallow exceptions -- Key: MAPREDUCE-1155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1155 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.22.0 Attachments: mapreduce-1155.patch, mapreduce-1155.patch, mapreduce-1155.txt, mapreduce-1155.txt Many of the streaming tests (including TestMultipleArchiveFiles) catch exceptions and print their stack trace rather than failing the job. This means that tests do not fail even when the job fails. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796788#action_12796788 ] Hemanth Yamijala commented on MAPREDUCE-1186: - Two minor nits before this is good to go: - I would recommend we pull out the new code that sets up permission in TrackerDistributedCacheManager into a separate method. localizeCache seems big enough to split - I also think a test case that verifies permissions are only set for newly localized *public* files is required for completeness of testing. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1355) contrib/index - fails to build with error - Overriding index.ivy.settings' is not allowed when using override='notallowed'
contrib/index - fails to build with error - Overriding index.ivy.settings' is not allowed when using override='notallowed' Key: MAPREDUCE-1355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1355 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/index Reporter: Kay Kay Priority: Critical Attachments: MAPREDUCE-1355.patch Checked out the trunk of mapreduce - tried to build contrib/index $ cd contrib/index $ ant clean compile test Failed with the error - mapreduce/src/contrib/build-contrib.xml:311: Overriding a previous definition of ivy:settings with the id 'index.ivy.settings' is not allowed when using override='notallowed' , (when in fact the override has been specified to be 'false' ). Bumping the ivy.version to 2.1.0 seems to be fix the issue. Marking this as critical since it is not possible to build in the first place. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1356) Allow user-specified hive table name in sqoop
Allow user-specified hive table name in sqoop - Key: MAPREDUCE-1356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1356 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-1356.patch The table name used in a hive-destination import is currently pegged to the input table name. This should be user-configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1166) SerialUtils.cc: dynamic allocation of arrays based on runtime variable is not portable
[
https://issues.apache.org/jira/browse/MAPREDUCE-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-1166:
Status: Open (was: Patch Available)
This patch isn't quite complete.
SerialUtils.cc: dynamic allocation of arrays based on runtime variable is not
portable
--
Key: MAPREDUCE-1166
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1166
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: MAPREREDUCE-1166.patch
In SerialUtils.cc, the following code appears:
int len;
if (b -120) {
negative = true;
len = -120 - b;
} else {
negative = false;
len = -112 - b;
}
uint8_t barr[len];
as far as I'm aware, this is not legal in ANSI C and will be rejected by ANSI
compliant compilers. Instead, this should be malloc()'d based upon the size
of len and free()'d later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1356) Allow user-specified hive table name in sqoop
[ https://issues.apache.org/jira/browse/MAPREDUCE-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-1356: - Status: Patch Available (was: Open) Allow user-specified hive table name in sqoop - Key: MAPREDUCE-1356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1356 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-1356.patch The table name used in a hive-destination import is currently pegged to the input table name. This should be user-configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1357) contrib/index - As part of the hudson build
contrib/index - As part of the hudson build Key: MAPREDUCE-1357 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1357 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Kay Kay As of now - it seems like contrib/index test cases do not come under the hudson build (contrib/index not being actively maintained ? ) test suite. Given that it is in the tree - might be useful to bring that in the first place. At its current state - most probably it might fail and may need to apply - MAPREDUCE-1328 and MAPREDUCE-1334 . But as the first step - we may need to bring in contrib/index ( If it is dormant - try to understand a bit of background ). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1358) Utils.OutputLogFilter incorrectly filters for _logs
Utils.OutputLogFilter incorrectly filters for _logs --- Key: MAPREDUCE-1358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1358 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon OutputLogFilter checks if the path contains _logs. This would incorrectly filter out all contents of a directory called server_logs, for example. Instead it should check for a path component exactly equal to _logs -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1218: -- Attachment: MAPREDUCE-1218-v4.patch I have moved the CPU usage calculation and added the corresponding unit tests. Collecting cpu and memory usage for TaskTrackers Key: MAPREDUCE-1218 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.22.0 Environment: linux Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch, MAPREDUCE-1218-v3.patch, MAPREDUCE-1218-v4.patch, MAPREDUCE-1218.patch The information can be used for resource aware scheduling. Note that this is related to MAPREDUCE-220. There the per task resource information is collected. This one collects the per machine information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1218: -- Status: Open (was: Patch Available) Collecting cpu and memory usage for TaskTrackers Key: MAPREDUCE-1218 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.22.0 Environment: linux Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch, MAPREDUCE-1218-v3.patch, MAPREDUCE-1218-v4.patch, MAPREDUCE-1218.patch The information can be used for resource aware scheduling. Note that this is related to MAPREDUCE-220. There the per task resource information is collected. This one collects the per machine information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1218: -- Status: Patch Available (was: Open) Collecting cpu and memory usage for TaskTrackers Key: MAPREDUCE-1218 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.22.0 Environment: linux Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch, MAPREDUCE-1218-v3.patch, MAPREDUCE-1218-v4.patch, MAPREDUCE-1218.patch The information can be used for resource aware scheduling. Note that this is related to MAPREDUCE-220. There the per task resource information is collected. This one collects the per machine information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1327) Oracle database import via sqoop fails when a table contains the column types such as TIMESTAMP(6) WITH LOCAL TIME ZONE and TIMESTAMP(6) WITH TIME ZONE
[ https://issues.apache.org/jira/browse/MAPREDUCE-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonid Furman updated MAPREDUCE-1327: - Affects Version/s: 0.22.0 Release Note: I generated the patch based on the build from the revision 830358. Therefore, the changes in this patch may not be compatible with the latest build (which contains compilation errors - see revision 896302. Once the build has no compiling errors, I can resubmit this patch again. Otherwise, feel free to manually apply the code changes I made to the main trunk after it is reviewed and approved. Hadoop Flags: [Incompatible change] Status: Patch Available (was: Open) The motivation for this fix is the following. Oracle releases 10.2+ introduced new data types TIMESTAMP WITH TIME ZONE and TIMESTAMP WITH LOCAL TIME ZONE, and Sqoop did not handle these types properly. To be specific, it did not find the proper Java data types mapping for those SQL columns, which caused the exception described earlier. This error occurred in two situations: 1. Sqoop was attempting to map the table metadata to Java data types while running table import. 2. Sqoop was attempting to map the table metadata to Java data types while loading data to hive. Hence, I added two methods toJavaType and toHiveType method to SqlManager interface, and implemented these methods in OracleManager class. From now on, any new data types that are very specific to database vendor and not yet supported can be handled properly. Please feel free to provide feedback. Thanks, Leonid. Oracle database import via sqoop fails when a table contains the column types such as TIMESTAMP(6) WITH LOCAL TIME ZONE and TIMESTAMP(6) WITH TIME ZONE --- Key: MAPREDUCE-1327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1327 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/sqoop Affects Versions: 0.22.0 Reporter: Leonid Furman Fix For: 0.22.0 Original Estimate: 96h Remaining Estimate: 96h When Oracle table contains the columns TIMESTAMP(6) WITH LOCAL TIME ZONE and TIMESTAMP(6) WITH TIME ZONE, Sqoop fails to map values for those columns to valid Java data types, resulting in the following exception: ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:253) at org.apache.hadoop.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:701) at org.apache.hadoop.sqoop.orm.ClassWriter.generate(ClassWriter.java:597) at org.apache.hadoop.sqoop.Sqoop.generateORM(Sqoop.java:75) at org.apache.hadoop.sqoop.Sqoop.importTable(Sqoop.java:87) at org.apache.hadoop.sqoop.Sqoop.run(Sqoop.java:175) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.sqoop.Sqoop.main(Sqoop.java:201) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) I have modified the code for Hadoop and Sqoop so this bug is fixed on my machine. Please let me know if you would like me to generate the patch and upload it to this ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1358) Utils.OutputLogFilter incorrectly filters for _logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-1358: --- Attachment: mapreduce-1358.txt Simple patch plus test case Utils.OutputLogFilter incorrectly filters for _logs --- Key: MAPREDUCE-1358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1358 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mapreduce-1358.txt OutputLogFilter checks if the path contains _logs. This would incorrectly filter out all contents of a directory called server_logs, for example. Instead it should check for a path component exactly equal to _logs -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1358) Utils.OutputLogFilter incorrectly filters for _logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-1358: --- Status: Patch Available (was: Open) Utils.OutputLogFilter incorrectly filters for _logs --- Key: MAPREDUCE-1358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1358 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mapreduce-1358.txt OutputLogFilter checks if the path contains _logs. This would incorrectly filter out all contents of a directory called server_logs, for example. Instead it should check for a path component exactly equal to _logs -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1302) TrackerDistributedCacheManager can delete file asynchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1302: -- Status: Patch Available (was: Open) TrackerDistributedCacheManager can delete file asynchronously - Key: MAPREDUCE-1302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1302 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.2, 0.21.0, 0.22.0 Reporter: Zheng Shao Assignee: Zheng Shao Attachments: MAPREDUCE-1302.0.patch, MAPREDUCE-1302.1.patch, MAPREDUCE-1302.2.patch, MAPREDUCE-1302.3.patch With the help of AsyncDiskService from MAPREDUCE-1213, we should be able to delete files from distributed cache asynchronously. That will help make task initialization faster, because task initialization calls the code that localizes files into the cache and may delete some other files. The deletion can slow down the task initialization speed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1302) TrackerDistributedCacheManager can delete file asynchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1302: -- Status: Open (was: Patch Available) TrackerDistributedCacheManager can delete file asynchronously - Key: MAPREDUCE-1302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1302 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.2, 0.21.0, 0.22.0 Reporter: Zheng Shao Assignee: Zheng Shao Attachments: MAPREDUCE-1302.0.patch, MAPREDUCE-1302.1.patch, MAPREDUCE-1302.2.patch, MAPREDUCE-1302.3.patch With the help of AsyncDiskService from MAPREDUCE-1213, we should be able to delete files from distributed cache asynchronously. That will help make task initialization faster, because task initialization calls the code that localizes files into the cache and may delete some other files. The deletion can slow down the task initialization speed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1302) TrackerDistributedCacheManager can delete file asynchronously
[ https://issues.apache.org/jira/browse/MAPREDUCE-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-1302: -- Attachment: MAPREDUCE-1302.3.patch Renamed SUBDIR to TOBEDELETED to avoid confusion. TrackerDistributedCacheManager can delete file asynchronously - Key: MAPREDUCE-1302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1302 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.2, 0.21.0, 0.22.0 Reporter: Zheng Shao Assignee: Zheng Shao Attachments: MAPREDUCE-1302.0.patch, MAPREDUCE-1302.1.patch, MAPREDUCE-1302.2.patch, MAPREDUCE-1302.3.patch With the help of AsyncDiskService from MAPREDUCE-1213, we should be able to delete files from distributed cache asynchronously. That will help make task initialization faster, because task initialization calls the code that localizes files into the cache and may delete some other files. The deletion can slow down the task initialization speed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1212) Mapreduce contrib project ivy dependencies are not included in binary target
[
https://issues.apache.org/jira/browse/MAPREDUCE-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-1212:
-
Attachment: MAPREDUCE-1212.2.patch
New patch based on discussion in HADOOP-6370. All the common deps are copied
into contrib/(projname)/lib except for those which are already in the global
lib directory.
Adds test ivy configurations to some contrib projects so that they do not
pull in hadoop-hdfs-test and hadoop-core-test in common mode.
Removed a few other spurious dependencies (e.g., slf-log4j) from some projects.
They should be fine using the global log4j. (Since that's all we were shipping,
anyway)
Mapreduce contrib project ivy dependencies are not included in binary target
Key: MAPREDUCE-1212
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1212
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Priority: Critical
Attachments: MAPREDUCE-1212.2.patch, MAPREDUCE-1212.patch
As in HADOOP-6370, only Hadoop's own library dependencies are promoted to
${build.dir}/lib; any libraries required by contribs are not redistributed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1212) Mapreduce contrib project ivy dependencies are not included in binary target
[
https://issues.apache.org/jira/browse/MAPREDUCE-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-1212:
-
Status: Patch Available (was: Open)
Mapreduce contrib project ivy dependencies are not included in binary target
Key: MAPREDUCE-1212
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1212
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Priority: Critical
Attachments: MAPREDUCE-1212.2.patch, MAPREDUCE-1212.patch
As in HADOOP-6370, only Hadoop's own library dependencies are promoted to
${build.dir}/lib; any libraries required by contribs are not redistributed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796986#action_12796986 ] Hadoop QA commented on MAPREDUCE-1218: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429493/MAPREDUCE-1218-v4.patch against trunk revision 896265. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/246/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/246/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/246/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/246/console This message is automatically generated. Collecting cpu and memory usage for TaskTrackers Key: MAPREDUCE-1218 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.22.0 Environment: linux Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.22.0 Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch, MAPREDUCE-1218-v3.patch, MAPREDUCE-1218-v4.patch, MAPREDUCE-1218.patch The information can be used for resource aware scheduling. Note that this is related to MAPREDUCE-220. There the per task resource information is collected. This one collects the per machine information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1327) Oracle database import via sqoop fails when a table contains the column types such as TIMESTAMP(6) WITH LOCAL TIME ZONE and TIMESTAMP(6) WITH TIME ZONE
[ https://issues.apache.org/jira/browse/MAPREDUCE-1327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796987#action_12796987 ] Hadoop QA commented on MAPREDUCE-1327: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429497/MAPREDUCE-1327.patch.gz against trunk revision 896265. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/247/console This message is automatically generated. Oracle database import via sqoop fails when a table contains the column types such as TIMESTAMP(6) WITH LOCAL TIME ZONE and TIMESTAMP(6) WITH TIME ZONE --- Key: MAPREDUCE-1327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1327 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/sqoop Affects Versions: 0.22.0 Reporter: Leonid Furman Fix For: 0.22.0 Attachments: MAPREDUCE-1327.patch.gz Original Estimate: 96h Remaining Estimate: 96h When Oracle table contains the columns TIMESTAMP(6) WITH LOCAL TIME ZONE and TIMESTAMP(6) WITH TIME ZONE, Sqoop fails to map values for those columns to valid Java data types, resulting in the following exception: ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.sqoop.orm.ClassWriter.generateFields(ClassWriter.java:253) at org.apache.hadoop.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:701) at org.apache.hadoop.sqoop.orm.ClassWriter.generate(ClassWriter.java:597) at org.apache.hadoop.sqoop.Sqoop.generateORM(Sqoop.java:75) at org.apache.hadoop.sqoop.Sqoop.importTable(Sqoop.java:87) at org.apache.hadoop.sqoop.Sqoop.run(Sqoop.java:175) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.sqoop.Sqoop.main(Sqoop.java:201) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) I have modified the code for Hadoop and Sqoop so this bug is fixed on my machine. Please let me know if you would like me to generate the patch and upload it to this ticket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1356) Allow user-specified hive table name in sqoop
[ https://issues.apache.org/jira/browse/MAPREDUCE-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796988#action_12796988 ] Hadoop QA commented on MAPREDUCE-1356: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429492/MAPREDUCE-1356.patch against trunk revision 896265. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/357/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/357/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/357/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/357/console This message is automatically generated. Allow user-specified hive table name in sqoop - Key: MAPREDUCE-1356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1356 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Attachments: MAPREDUCE-1356.patch The table name used in a hive-destination import is currently pegged to the input table name. This should be user-configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1342) Potential JT deadlock in faulty TT tracking
[
https://issues.apache.org/jira/browse/MAPREDUCE-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12796996#action_12796996
]
Amar Kamat commented on MAPREDUCE-1342:
---
What if we move the code from JobTracker.blacklistedTaskTrackers() to
FaultyTrackersInfo. Something like
{code}
FaultyTrackersInfo {
blacklistedTaskTrackers {
synchronized (potentiallyFaultyTrackers) {
synchronized (taskTrackers) {
// code that we have today JobTracker.blacklistedTaskTrackers()
for (TaskTracker tt : taskTrackers.values()) {
// ...
}
}
}
}
}
blacklistedTaskTrackers() {
return FaultyTrackersInfo.blacklistedTaskTrackers()
}
{code}
This kindof solves the lock reversal issue we are facing now and also makes
more sense because JobTracker.FaultyTrackersInfo is the right module to answer
the blacklistedTaskTrackers() query. Thoughts?
Potential JT deadlock in faulty TT tracking
---
Key: MAPREDUCE-1342
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1342
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Attachments: cycle0.png, mapreduce-1342-1.patch,
mapreduce-1342-2.patch
JT$FaultyTrackersInfo.incrementFaults first locks potentiallyFaultyTrackers,
and then calls blackListTracker, which calls removeHostCapacity, which locks
JT.taskTrackers
On the other hand, JT.blacklistedTaskTrackers() locks taskTrackers, then
calls faultyTrackers.isBlacklisted() which goes on to lock
potentiallyFaultyTrackers.
I haven't produced such a deadlock, but the lock ordering here is inverted
and therefore could deadlock.
Not sure if this goes back to 0.21 or just in trunk.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
[
https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12797001#action_12797001
]
Vinod K V commented on MAPREDUCE-1218:
--
Looked at that latest patch. Coming out good. Some comments, most of them are
finishing touches.
- LinuxResourceCalculatorPlugin.java
-- The method {{#getCpuUsage()}} in itself doesn't update the cpu-usage.
So, if one makes two calls to this method separated by a time interval, the
result won't reflect the updated cpu-usage unless calls to
{{#getCumulativeCpuTime()}} are explicitly made in between. This should be
fixed by calling {{#readProcStatFile()}} in this method also. The main method
should be modified to call this method twice and the tests should also verify
this.
-- You've changed {{CPU_TIME_FORMAT}} from _^cpu[0-9]*[ \t]*([0-9]*)[
\t]*([0-9]*)[ \t]*([0-9]*)[ \t].*_ to _^cpu[ \t]*([0-9]*)[ \t]*([0-9]*)[
\t]*([0-9]*)[ \t].*_. I guess the earlier is correct with cpu names not having
any space/tab?
-- Why even have MINIMUM_UPDATE_INTERVAL in updateCpuUsage()? If this is
mainly used for making sure sampleTime is not equal to lastSampleTime, then we
can do so directly and remove MINIMUM_UPDATE_INTERVAL altogether.
- TaskTracker.java
-- To handle the deprecation of
{{mapreduce.tasktracker.memorycalculatorplugin}} in TaskTracker, for memory
calculations we should first try to use the class denoted by this configuration
if present, otherwise only we should fall back to the new resource-calculator.
To facilitate this we will also retain a deprecated
{{TTConfig.TT_MEMORY_CALCULATOR_PLUGIN}} constant.
-- Nit: +661 LOG.info( Using MemoryCalculatorPlugin : +
resourceCalculatorPlugin); should instead be LOG.info( Using
ResourceCalculatorPlugin : + resourceCalculatorPlugin);
- Please annotate Dummy{Resource|Memory}CalculatorPlugin classes as
@InterfaceAudience.Private because they both are only test specific.
- We should document {{mapreduce.tasktracker.resourcecalculatorplugin}} in
mapred-default.xml and remove the documentation for
{{mapreduce.tasktracker.memorycalculatorplugin}} from the same.
- Please convert TestLinuxResourceCalculatorPlugin and TestTTResourceReporting
into Junit 4 testcases.
Collecting cpu and memory usage for TaskTrackers
Key: MAPREDUCE-1218
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Affects Versions: 0.22.0
Environment: linux
Reporter: Scott Chen
Assignee: Scott Chen
Fix For: 0.22.0
Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch,
MAPREDUCE-1218-v3.patch, MAPREDUCE-1218-v4.patch, MAPREDUCE-1218.patch
The information can be used for resource aware scheduling.
Note that this is related to MAPREDUCE-220. There the per task resource
information is collected.
This one collects the per machine information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12797006#action_12797006 ] Kay Kay commented on MAPREDUCE-1352: @Giri / @stack - can you help with the publishing this patch to the branch / republishing artifacts to validate hbase-1433 before that patch could be taken up for consideration with hbase community. 0.21.0 - snapshot incorrect dependency published in .pom files --- Key: MAPREDUCE-1352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: 0.21.0 Reporter: Kay Kay Assignee: Giridharan Kesavan Priority: Critical Fix For: 0.21.0 Attachments: mapreduce-1352.patch The snapshot available here at - https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ , has an incorrect dependency specified for hadoop-core ( in the .pom file ). The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in ivy/libaries.properties but pom.xml published in the repository refers to 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
