[jira] [Updated] (MAPREDUCE-5014) Extending DistCp through a custom CopyListing is not possible
[ https://issues.apache.org/jira/browse/MAPREDUCE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-5014: -- Target Version/s: 0.23.5, 0.23.4, 0.23.3, 0.23.1, 0.23.0, trunk (was: 0.23.0, 0.23.1, 0.23.3, trunk, 0.23.4, 0.23.5) Fix Version/s: 2.5.0 I've just merged this back to branch-2 as well. Extending DistCp through a custom CopyListing is not possible - Key: MAPREDUCE-5014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5014 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.23.0, 0.23.1, 0.23.3, trunk, 0.23.4, 0.23.5 Reporter: Srikanth Sundarrajan Assignee: Srikanth Sundarrajan Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5014.patch, MAPREDUCE-5014.patch, m5014_20130322.patch, m5014_20130322_b-1.patch, m5014_20130322b.patch, m5014_20130322b_b-1.patch Original Estimate: 24h Remaining Estimate: 24h * While it is possible to implement a custom CopyListing in DistCp, DistCp driver class doesn't allow for using this custom CopyListing. * Allow SimpleCopyListing to provide an option to exclude files (For instance it is useful to exclude FileOutputCommiter.SUCCEEDED_FILE_NAME during copy as premature copy can indicate that the entire data is available at the destination) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (MAPREDUCE-5579) Improve JobTracker web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers moved HADOOP-10038 to MAPREDUCE-5579: Affects Version/s: (was: 1.2.2) 1.2.2 Key: MAPREDUCE-5579 (was: HADOOP-10038) Project: Hadoop Map/Reduce (was: Hadoop Common) Improve JobTracker web UI - Key: MAPREDUCE-5579 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5579 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 1.2.2 Reporter: David Chen Attachments: jobdetails.png, jobtasks.png, jobtracker.png Users will often need to use the JobTracker web UI to debug or tune their jobs in addition to checking the status of their jobs. The current web UI is cumbersome to navigate. The goal is to make the JobTracker web UI easier to navigate and present the data in a cleaner and more intuitive format. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Moved] (MAPREDUCE-5571) allow access to the DFS job submission + staging directory by members of the job submitters group
[ https://issues.apache.org/jira/browse/MAPREDUCE-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers moved HADOOP- to MAPREDUCE-5571: --- Affects Version/s: (was: 2.0.5-alpha) (was: 1.2.1) 1.2.1 2.0.5-alpha Key: MAPREDUCE-5571 (was: HADOOP-) Project: Hadoop Map/Reduce (was: Hadoop Common) allow access to the DFS job submission + staging directory by members of the job submitters group - Key: MAPREDUCE-5571 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5571 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.5-alpha, 1.2.1 Environment: linux Reporter: bradley childs Attachments: HADOOP-1.2-PERM.patch, hadoop-2.0.5-perm.patch The job submission and staging directories are explicitly given 0700 permissions restricting access of job submission files only to the submitter UID. this prevents hadoop daemon services running under different UIDs from reading the job submitters files. it is common unix practice to run daemon services under their own UIDs for security purposes. This bug can be demonstrated by creating a single node configuration, which runs LocalFileSystem and not HDFS. Create two users and add them to a 'hadoop' group. Start the hadoop services with one of the users, then submit a map/reduce job with the other user (or run one of the examples). Job submission ultimately fails and the M/R job doesn't execute. The fix is simple enough and secure-- change the staging directory permissions to 2750. i have demonstrated the patch against 2.0.5 (along with another fix for an incorrect decimal-octal conversion) and will attach the patch. this bug is present since very early versions. i would like to fix it at the lowest level as it's a simple file mode change in all versions, and localized to one file. is this possible? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (MAPREDUCE-5524) java.io.IOException: Task process exit with nonzero status of 255. how to fix it?
[ https://issues.apache.org/jira/browse/MAPREDUCE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5524. --- Resolution: Invalid Please email u...@hadoop.apache.org with this question. Apache JIRA is for reporting bugs and tracking features/improvements. It's not intended for user-level help. java.io.IOException: Task process exit with nonzero status of 255. how to fix it? --- Key: MAPREDUCE-5524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5524 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: hawkswood Task ..FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 255. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Moved] (MAPREDUCE-5512) TaskTracker hung after failed reconnect to the JobTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers moved HADOOP-9970 to MAPREDUCE-5512:
---
Affects Version/s: (was: 1.3.0)
1.3.0
Key: MAPREDUCE-5512 (was: HADOOP-9970)
Project: Hadoop Map/Reduce (was: Hadoop Common)
TaskTracker hung after failed reconnect to the JobTracker
-
Key: MAPREDUCE-5512
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5512
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: hadoop-tasktracker-RD00155DD09100.log, tt_Hung.txt
TaskTracker hung after failed reconnect to the JobTracker.
This is the problematic piece of code:
{code}
this.distributedCacheManager = new TrackerDistributedCacheManager(
this.fConf, taskController);
this.distributedCacheManager.startCleanupThread();
this.jobClient = (InterTrackerProtocol)
UserGroupInformation.getLoginUser().doAs(
new PrivilegedExceptionActionObject() {
public Object run() throws IOException {
return RPC.waitForProxy(InterTrackerProtocol.class,
InterTrackerProtocol.versionID,
jobTrackAddr, fConf);
}
});
{code}
In case RPC.waitForProxy() throws, TrackerDistributedCacheManager cleanup
thread will never be stopped, and given that it is a non daemon thread it
will keep TT up forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5435) Nodemanager stops working automatically
[ https://issues.apache.org/jira/browse/MAPREDUCE-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724209#comment-13724209 ] Aaron T. Myers commented on MAPREDUCE-5435: --- Vishket, in the future, please email u...@hadoop.apache.org when asking questions like this. Apache JIRA is for reporting bugs or enhancements. Nodemanager stops working automatically --- Key: MAPREDUCE-5435 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5435 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vishket Hi Everyone, I have been trying to setup a 10 node Hadoop Cluster(Hadoop 2.0.5 alpha). I've completed editing all the configuration files and am now trying to run the daemons. All the processes work fine apart from the nodemanager. The nodemanager runs fine on the slave however, on the master, it will only run for 10-15 sec and then stops. Same thing happens if I run the start command again. Any suggestions? Thanks in advance! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned MAPREDUCE-5193: - Assignee: Andrew Wang (was: Aaron T. Myers) A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Andrew Wang Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13647186#comment-13647186 ] Aaron T. Myers commented on MAPREDUCE-5193: --- Thanks a lot for taking up this issue in my stead, Andrew. +1, the patch looks good to me. I'm going to commit this momentarily. A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Andrew Wang Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-5193: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrew. A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Andrew Wang Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645329#comment-13645329 ] Aaron T. Myers commented on MAPREDUCE-5193: --- Somewhat curiously, I'm having trouble reproducing this locally. I would expect these tests to fail pretty consistently. Zhijie, can you reproduce these test failures? A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Aaron T. Myers HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645726#comment-13645726 ] Aaron T. Myers commented on MAPREDUCE-5193: --- [~ste...@apache.org] Thanks for the pointer, but I don't think that's what the issue was here. Zhijie referenced these two builds, both of which failed with the expected error message regarding the block size being less than the configured minimum: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3557//testReport/ https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3558//testReport/ Just to see if this still reproduces, I'm going to upload the same patch that was posted to MAPREDUCE-5156 to this JIRA for Jenkins to run. A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: MAPREDUCE-5156.1.patch HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-5193: -- Attachment: MAPREDUCE-5156.1.patch A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: MAPREDUCE-5156.1.patch HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-5193: -- Status: Patch Available (was: Open) A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: MAPREDUCE-5156.1.patch HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
Aaron T. Myers created MAPREDUCE-5193: - Summary: A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Aaron T. Myers HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5033) mapred shell script should respect usage flags (--help -help -h)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13587931#comment-13587931 ] Aaron T. Myers commented on MAPREDUCE-5033: --- +1, patch looks good to me. I'm going to commit this momentarily. mapred shell script should respect usage flags (--help -help -h) Key: MAPREDUCE-5033 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5033 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Attachments: mapreduce-5033-1.patch Like in HADOOP-9267, the mapred shell script should respect the normal Unix-y help flags. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5033) mapred shell script should respect usage flags (--help -help -h)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-5033: -- Resolution: Fixed Fix Version/s: 2.0.4-beta Target Version/s: 2.0.4-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrew. mapred shell script should respect usage flags (--help -help -h) Key: MAPREDUCE-5033 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5033 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Fix For: 2.0.4-beta Attachments: mapreduce-5033-1.patch Like in HADOOP-9267, the mapred shell script should respect the normal Unix-y help flags. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5004) Somebody working on Genetic Algorithm library on Map Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5004. --- Resolution: Invalid Hi Abhishek, I'm not quite sure what you were trying to get at with this JIRA, but I recommend emailing either u...@hadoop.apache.org or common-...@hadoop.apache.org with your question. Somebody working on Genetic Algorithm library on Map Reduce --- Key: MAPREDUCE-5004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5004 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Abhishek Bajpai -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572729#comment-13572729 ] Aaron T. Myers commented on MAPREDUCE-4977: --- +1, the patch looks good to me. Documentation for pluggable shuffle and pluggable sort -- Key: MAPREDUCE-4977 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4877.patch, MAPREDUCE-4877.patch Add documentation with basic information on pluggable shuffle and sort. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4953) HadoopPipes misuses fprintf
[
https://issues.apache.org/jira/browse/MAPREDUCE-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571045#comment-13571045
]
Aaron T. Myers commented on MAPREDUCE-4953:
---
+1, the patch looks good to me. I confirmed that this gets rid of the compiler
warning.
I'm going to commit this momentarily.
HadoopPipes misuses fprintf
---
Key: MAPREDUCE-4953
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4953
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 3.0.0, 2.0.3-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Attachments: mapreduce-4953.txt
{code}
[exec]
/mnt/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:130:58:
warning: format not a string literal and no format arguments
[-Wformat-security]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4953) HadoopPipes misuses fprintf
[
https://issues.apache.org/jira/browse/MAPREDUCE-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-4953:
--
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this to trunk and branch-2.
Thanks a lot for the contribution, Andy.
HadoopPipes misuses fprintf
---
Key: MAPREDUCE-4953
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4953
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 3.0.0, 2.0.3-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Fix For: 2.0.3-alpha
Attachments: mapreduce-4953.txt
{code}
[exec]
/mnt/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:130:58:
warning: format not a string literal and no format arguments
[-Wformat-security]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4820) MRApps distributed-cache duplicate checks are incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-4820: -- Target Version/s: 2.0.3-alpha Fix Version/s: (was: 2.0.3-alpha) MRApps distributed-cache duplicate checks are incorrect --- Key: MAPREDUCE-4820 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4820 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.0.2-alpha Reporter: Alejandro Abdelnur Priority: Blocker This seems a combination of issues that are being exposed in 2.0.2-alpha by MAPREDUCE-4549. MAPREDUCE-4549 introduces a check to to ensure there are not duplicate JARs in the distributed-cache (using the JAR name as identity). In Hadoop 2 (different from Hadoop 1), all JARs in the distributed-cache are symlink-ed to the current directory of the task. MRApps, when setting up the DistributedCache (MRApps#setupDistributedCache-parseDistributedCacheArtifacts) assumes that the local resources (this includes files in the CURRENT_DIR/, CURRENT_DIR/classes/ and files in CURRENT_DIR/lib/) are part of the distributed-cache already. For systems, like Oozie, which use a launcher job to submit the real job this poses a problem because MRApps is run from the launcher job to submit the real job. The configuration of the real job has the correct distributed-cache entries (no duplicates), but because the current dir has the same files, the submission fails. It seems that MRApps should not be checking dups in the distributed-cached against JARs in the CURRENT_DIR/ or CURRENT_DIR/lib/. The dup check should be done among distributed-cached entries only. It seems YARNRunner is symlink-ing all files in the distributed cached in the current directory. In Hadoop 1 this was done only for files added to the distributed-cache using a fragment (ie #FOO) to trigger a symlink creation. Marking as a blocker because without a fix for this, Oozie cannot submit jobs to Hadoop 2 (i've debugged Oozie in a live cluster being used by BigTop -thanks Roman- to test their release work, and I've verified that Oozie 3.3 does not create duplicated entries in the distributed-cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-4822:
--
Summary: Unnecessary conversions in History Events (was: Unnessisary
conversions in History Events)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Priority: Trivial
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4760) Make a version of Counters that is composite for the job and stores the counter values in arrays
[ https://issues.apache.org/jira/browse/MAPREDUCE-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-4760: -- Summary: Make a version of Counters that is composite for the job and stores the counter values in arrays (was: Make a version of Counters that is composit for the job and stores the counter values in arrays.) Make a version of Counters that is composite for the job and stores the counter values in arrays Key: MAPREDUCE-4760 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4760 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Robert Joseph Evans Priority: Minor String interning reduced the size of counters a lot. After that and the fix for a memory leak in the IPC server a job with 2 map tasks and 3000 reducers takes about 200MB to store the state of all of the tasks. Looking at a memory dump of the AM each task attempt has a pointer to a Counters object that is about 2kb to 3kb in size. That means Counters account for about 56MB of the 200MB of state. This job only had about 40 task counters in it. Each counter stores a long value so if we stored them in a long[] instead we should only be taking up 7MB. Also assuming that some of the counters only appear in a map task or a reduce task we should be able to have one CompositCounters for map tasks and one for reduce tasks so it would reduce the size even further. NOTE: without this change I would expect to be able to run a 100,000 task job in the default 1024MB AM heap (875MB/200MB * 2300) I reserved 150MB for IPC buffers and event data. With this change we could expect to run about 130,000 tasks (875MB/150MB * 2300). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4616) Improvement to MultipleOutputs javadocs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned MAPREDUCE-4616: - Assignee: Tony Burton Improvement to MultipleOutputs javadocs --- Key: MAPREDUCE-4616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4616 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 1.0.3 Reporter: Tony Burton Assignee: Tony Burton Priority: Minor Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4616.patch, MAPREDUCE-4616.patch In the new API, and using MultipleOutputs it is possible to segment output into directories by using MultipleOutputs.write(KEYOUT key, VALUEOUT value, String baseOutputPath) in the Reducer to determine the output directory, and by using LazyOutputFormat at the job-level config to suppress normal output [eg use LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class); instead of job.setOutputFormatClass(TextOutputFormat.class);] This recreates the functionality previously provided in the old API by using MultipleTextOutputFormat (etc) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4717) Mapreduce job fails to run after configuring multiple namespaces [HDFS Federation]
[ https://issues.apache.org/jira/browse/MAPREDUCE-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-4717. --- Resolution: Not A Problem Hi Sagar, this looks to me to be most likely a configuration error, in which case you should try emailing a user mailing list. I'm guessing from the facts that you say you configured federation and you're using MR1 that you're using CDH, in which case you should email cdh-u...@cloudera.org. If I'm wrong about that and you're somehow using a straight Apache release, then you should email u...@hadoop.apache.org. Mapreduce job fails to run after configuring multiple namespaces [HDFS Federation] -- Key: MAPREDUCE-4717 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4717 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.2 Environment: 2 Standalone Desktop with 360Gb HDD and 4GB RAM - Acting as two Namenodes 2 Virtual Machine with 60GB HDD and 1GB RAM - Acting as Job tracker and zookeeper Reporter: Sagar Shimpi I am having setup of 4 nodes with following details - Standalone Desktop-1 - NameNode1,Tasktracker,Zookeeper,Jobtracker,datanode,HMaster Standalone Desktop-2 - NameNode2,Tasktracker,datanode.RegionServer Virtual Machine-1 - Namenode3,Datanode,Tasktracker Virtual Machine-2 - Namenode4,Datanode,Tasktracker I have configured HDFS Federation with following name service - a) nameservice1 b) oss-hadoop-nameservice While executing Mapreduce job I am getting following error - -bash-4.1$ id uid=496(hdfs) gid=496(hdfs) groups=496(hdfs),497(hadoop) -bash-4.1$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /hbase/install.log.syslog /hbase/testing 12/10/10 12:30:21 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: viewfs://cluster6/ java.io.IOException: viewfs://cluster6/ at org.apache.hadoop.fs.viewfs.InodeTree.init(InodeTree.java:338) at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.init(ViewFileSystem.java:178) at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:178) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:844) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:844) at org.apache.hadoop.mapreduce.Job.submit(Job.java:481) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:511) at org.apache.hadoop.examples.WordCount.main(WordCount.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) -bash-4.1$ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Assigned] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[
https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers reassigned MAPREDUCE-4678:
-
Assignee: Chris McConnell
Running the Pentomino example with defaults throws
java.lang.NegativeArraySizeException
---
Key: MAPREDUCE-4678
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 2.0.0-alpha
Reporter: Chris McConnell
Assignee: Chris McConnell
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt
HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the
following error message:
{code}
INFO util.NativeCodeLoader: Loaded the native-hadoop library
INFO mapred.FileInputFormat: Total input paths to process : 1
INFO mapred.JobClient: Running job: job_x
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_, Status : FAILED
java.lang.NegativeArraySizeException
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:95)
at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di
stributedPentomino.java:51)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat
ion.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4653) TestRandomAlgorithm has an unused import statement
[
https://issues.apache.org/jira/browse/MAPREDUCE-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455096#comment-13455096
]
Aaron T. Myers commented on MAPREDUCE-4653:
---
Hi Amir, when posting a patch you should create the diff from the root
directory of the repository. Mind posting an updated patch?
TestRandomAlgorithm has an unused import statement
-
Key: MAPREDUCE-4653
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4653
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 1.0.3
Environment: hadoop 1.0.3 on any platform
Reporter: Amir Sanjar
Priority: Trivial
Fix For: 1.0.3
Attachments: MAPREDUCE-4653.patch
Original Estimate: 24h
Remaining Estimate: 24h
need to remove the import statement usinf below patch. will attach a patch
shortly .
Index: TestRandomAlgorithm.java
===
--- TestRandomAlgorithm.java (revision 1380737)
+++ TestRandomAlgorithm.java (working copy)
@@ -30,8 +30,6 @@
import org.junit.Test;
-import com.sun.tools.javac.code.Attribute.Array;
-
public class TestRandomAlgorithm {
private static final int[][] parameters = new int[][] {
{5, 1, 1},
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4653) TestRandomAlgorithm has an unused import statement
[
https://issues.apache.org/jira/browse/MAPREDUCE-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers reassigned MAPREDUCE-4653:
-
Assignee: Amir Sanjar
TestRandomAlgorithm has an unused import statement
-
Key: MAPREDUCE-4653
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4653
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 1.0.3
Environment: hadoop 1.0.3 on any platform
Reporter: Amir Sanjar
Assignee: Amir Sanjar
Priority: Trivial
Fix For: 1.0.3
Attachments: MAPREDUCE-4653.patch
Original Estimate: 24h
Remaining Estimate: 24h
need to remove the import statement usinf below patch. will attach a patch
shortly .
Index: TestRandomAlgorithm.java
===
--- TestRandomAlgorithm.java (revision 1380737)
+++ TestRandomAlgorithm.java (working copy)
@@ -30,8 +30,6 @@
import org.junit.Test;
-import com.sun.tools.javac.code.Attribute.Array;
-
public class TestRandomAlgorithm {
private static final int[][] parameters = new int[][] {
{5, 1, 1},
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4653) TestRandomAlgorithm has an unused import statement
[
https://issues.apache.org/jira/browse/MAPREDUCE-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13455190#comment-13455190
]
Aaron T. Myers commented on MAPREDUCE-4653:
---
Hi Amir, it looks like you've now posted a patch for the trunk branch, which is
good, but you should still be creating a patch based in the root of the
repository directory structure, i.e. the diff you make should include the full
path to TestRandomAlgorithm.java, not just the file name.
TestRandomAlgorithm has an unused import statement
-
Key: MAPREDUCE-4653
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4653
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 1.0.3
Environment: hadoop 1.0.3 on any platform
Reporter: Amir Sanjar
Assignee: Amir Sanjar
Priority: Trivial
Fix For: 1.0.3
Attachments: MAPREDUCE-4653.patch, MAPREDUCE-4653.patch
Original Estimate: 24h
Remaining Estimate: 24h
need to remove the import statement usinf below patch. will attach a patch
shortly .
Index: TestRandomAlgorithm.java
===
--- TestRandomAlgorithm.java (revision 1380737)
+++ TestRandomAlgorithm.java (working copy)
@@ -30,8 +30,6 @@
import org.junit.Test;
-import com.sun.tools.javac.code.Attribute.Array;
-
public class TestRandomAlgorithm {
private static final int[][] parameters = new int[][] {
{5, 1, 1},
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4653) TestRandomAlgorithm has an unused import statement
[
https://issues.apache.org/jira/browse/MAPREDUCE-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-4653:
--
Target Version/s: 3.0.0
Status: Patch Available (was: Open)
That looks better. Marking this patch available so that test-patch runs.
TestRandomAlgorithm has an unused import statement
-
Key: MAPREDUCE-4653
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4653
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 1.0.3
Environment: hadoop 1.0.3 on any platform
Reporter: Amir Sanjar
Assignee: Amir Sanjar
Priority: Trivial
Fix For: 1.0.3
Attachments: MAPREDUCE-4653.patch
Original Estimate: 24h
Remaining Estimate: 24h
need to remove the import statement usinf below patch. will attach a patch
shortly .
Index: TestRandomAlgorithm.java
===
--- TestRandomAlgorithm.java (revision 1380737)
+++ TestRandomAlgorithm.java (working copy)
@@ -30,8 +30,6 @@
import org.junit.Test;
-import com.sun.tools.javac.code.Attribute.Array;
-
public class TestRandomAlgorithm {
private static final int[][] parameters = new int[][] {
{5, 1, 1},
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4491) Encryption and Key Protection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449260#comment-13449260 ] Aaron T. Myers commented on MAPREDUCE-4491: --- bq. This is an important point as we do not want Tasktracker to decrypt the blob of keys and blindly hand over to Tasks. The JobClient stores JobId along with keys as part of the encrypted blob. The taskTracker decrypts the encrypted blob, verifies that the JobId in the encrypted blob matches JobId of the task. The keys are handed over to Tasks only if the JobId verification is successful. This ensures that keys are handed over to the correct tasks. Unless I'm missing something, this seems to be insecure unless secure authentication (i.e. Kerberos) is enabled, since someone could connect to the TT from a different task and simply report a different JobId. Or do I misunderstand somehow? Encryption and Key Protection - Key: MAPREDUCE-4491 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4491 Project: Hadoop Map/Reduce Issue Type: New Feature Components: documentation, security, task-controller, tasktracker Reporter: Benoy Antony Assignee: Benoy Antony Attachments: Hadoop_Encryption.pdf, Hadoop_Encryption.pdf When dealing with sensitive data, it is required to keep the data encrypted wherever it is stored. Common use case is to pull encrypted data out of a datasource and store in HDFS for analysis. The keys are stored in an external keystore. The feature adds a customizable framework to integrate different types of keystores, support for Java KeyStore, read keys from keystores, and transport keys from JobClient to Tasks. The feature adds PGP encryption as a codec and additional utilities to perform encryption related steps. The design document is attached. It explains the requirement, design and use cases. Kindly review and comment. Collaboration is very much welcome. I have a tested patch for this for 1.1 and will upload it soon as an initial work for further refinement. Update: The patches are uploaded to subtasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2374) Text File Busy errors launching MR tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444189#comment-13444189
]
Aaron T. Myers commented on MAPREDUCE-2374:
---
bq. ATM - In future, please open a separate YARN jira. It's very odd to have
MAPREDUCE jira numbers in hadoop-yarn-project/CHANGES.txt.
Sorry, Arun. I didn't quite know what to do with this since this JIRA was
opened long before YARN was split off from MR. When committing, I also noticed
that there is some precedent for having a MAPREDUCE JIRA entry in the YARN
CHANGES.txt (MAPREDUCE-4323) and so thought it was OK. I agree that putting a
MAPREDUCE jira in the YARN CHANGES.txt is odd, but opening a separate JIRA just
so we have a new number for putting in CHANGES.txt also seemed odd to me. Doing
so would also serve to artificially separate the issues from the branch-1 and
branch-0.23 fixes, which seemed less than ideal as well. Anyway, my point is
just that the right thing to do was not obvious.
Text File Busy errors launching MR tasks
--
Key: MAPREDUCE-2374
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2374
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Andy Isaacson
Fix For: 1.2.0, 0.23.3, 2.2.0-alpha
Attachments: failed_taskjvmsh.strace, mapreduce-2374-2.txt,
mapreduce-2374-branch-1.patch, mapreduce-2374-on-20sec.txt,
mapreduce-2374.txt, mapreduce-2374.txt, mapreduce-2374.txt,
successfull_taskjvmsh.strace
Some very small percentage of tasks fail with a Text file busy error.
The following was the original diagnosis:
{quote}
Our use of PrintWriter in TaskController.writeCommand is unsafe, since that
class swallows all IO exceptions. We're not currently checking for errors,
which I'm seeing result in occasional task failures with the message Text
file busy - assumedly because the close() call is failing silently for some
reason.
{quote}
.. but turned out to be another issue as well (see below)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2374) Text File Busy errors launching MR tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-2374:
--
Resolution: Fixed
Fix Version/s: (was: 0.22.1)
2.2.0-alpha
1.2.0
Assignee: Andy Isaacson (was: Todd Lipcon)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed Andy's latest patches to to branch-1, branch-2, and trunk.
Thanks a lot for the contribution, Andy, and thanks a lot to everyone for the
discussion to get to the bottom of this issue.
Text File Busy errors launching MR tasks
--
Key: MAPREDUCE-2374
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2374
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Andy Isaacson
Fix For: 1.2.0, 2.2.0-alpha
Attachments: failed_taskjvmsh.strace, mapreduce-2374-2.txt,
mapreduce-2374-branch-1.patch, mapreduce-2374-on-20sec.txt,
mapreduce-2374.txt, mapreduce-2374.txt, mapreduce-2374.txt,
successfull_taskjvmsh.strace
Some very small percentage of tasks fail with a Text file busy error.
The following was the original diagnosis:
{quote}
Our use of PrintWriter in TaskController.writeCommand is unsafe, since that
class swallows all IO exceptions. We're not currently checking for errors,
which I'm seeing result in occasional task failures with the message Text
file busy - assumedly because the close() call is failing silently for some
reason.
{quote}
.. but turned out to be another issue as well (see below)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4328) Add the option to quiesce the JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439865#comment-13439865 ] Aaron T. Myers commented on MAPREDUCE-4328: --- Thanks a lot for the explanation, Arun. Makes sense. Add the option to quiesce the JobTracker Key: MAPREDUCE-4328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4328 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.0.3 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.2.0 Attachments: MAPREDUCE-4328.patch, MAPREDUCE-4328.patch, TestJobTrackerQuiescence.java In several failure scenarios it would be very handy to have an option to quiesce the JobTracker. Recently, we saw a case where the NameNode had to be rebooted at a customer due to a random hardware failure - in such a case it would have been nice to not lose jobs by quiescing the JobTracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-4577) HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
[
https://issues.apache.org/jira/browse/MAPREDUCE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers moved HDFS-3842 to MAPREDUCE-4577:
-
Component/s: (was: hdfs client)
test
Target Version/s: 2.2.0-alpha (was: 2.2.0-alpha)
Affects Version/s: (was: 2.2.0-alpha)
2.2.0-alpha
Key: MAPREDUCE-4577 (was: HDFS-3842)
Project: Hadoop Map/Reduce (was: Hadoop HDFS)
HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
---
Key: MAPREDUCE-4577
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4577
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.2.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Aaron T. Myers
Priority: Critical
Attachments: HDFS-3842.patch
Before HDFS-3672, locally applying MAPREDUCE-4470 made
TestCombineFileInputFormat to pass all it tests.
After HDFS-3672, TestCombineFileInputFormat.testMissingBlocks() fails:
{code}
$ mvn clean test -Dtest=TestCombineFileInputFormat
Running org.apache.hadoop.mapred.TestCombineFileInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 sec
Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.526 sec
FAILURE!
Results :
Tests in error:
testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat):
org.apache.hadoop.fs.BlockLocation
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4577) HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
[
https://issues.apache.org/jira/browse/MAPREDUCE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439884#comment-13439884
]
Aaron T. Myers commented on MAPREDUCE-4577:
---
Thanks for the reviews, guys. I've moved this JIRA to the MR project since it's
really an MR test bug, not in HDFS.
HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
---
Key: MAPREDUCE-4577
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4577
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.2.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Aaron T. Myers
Priority: Critical
Attachments: HDFS-3842.patch
Before HDFS-3672, locally applying MAPREDUCE-4470 made
TestCombineFileInputFormat to pass all it tests.
After HDFS-3672, TestCombineFileInputFormat.testMissingBlocks() fails:
{code}
$ mvn clean test -Dtest=TestCombineFileInputFormat
Running org.apache.hadoop.mapred.TestCombineFileInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 sec
Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.526 sec
FAILURE!
Results :
Tests in error:
testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat):
org.apache.hadoop.fs.BlockLocation
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4577) HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
[
https://issues.apache.org/jira/browse/MAPREDUCE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-4577:
--
Priority: Minor (was: Critical)
HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
---
Key: MAPREDUCE-4577
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4577
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.2.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Aaron T. Myers
Priority: Minor
Attachments: HDFS-3842.patch
Before HDFS-3672, locally applying MAPREDUCE-4470 made
TestCombineFileInputFormat to pass all it tests.
After HDFS-3672, TestCombineFileInputFormat.testMissingBlocks() fails:
{code}
$ mvn clean test -Dtest=TestCombineFileInputFormat
Running org.apache.hadoop.mapred.TestCombineFileInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 sec
Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.526 sec
FAILURE!
Results :
Tests in error:
testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat):
org.apache.hadoop.fs.BlockLocation
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4577) HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
[
https://issues.apache.org/jira/browse/MAPREDUCE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439917#comment-13439917
]
Aaron T. Myers commented on MAPREDUCE-4577:
---
The test is still failing because of MAPREDUCE-4470, but the test is no longer
failing with the ArrayStoreException, as it was without this patch.
I'm going to commit this momentarily.
HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
---
Key: MAPREDUCE-4577
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4577
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.2.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Aaron T. Myers
Priority: Minor
Attachments: HDFS-3842.patch
Before HDFS-3672, locally applying MAPREDUCE-4470 made
TestCombineFileInputFormat to pass all it tests.
After HDFS-3672, TestCombineFileInputFormat.testMissingBlocks() fails:
{code}
$ mvn clean test -Dtest=TestCombineFileInputFormat
Running org.apache.hadoop.mapred.TestCombineFileInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 sec
Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.526 sec
FAILURE!
Results :
Tests in error:
testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat):
org.apache.hadoop.fs.BlockLocation
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4577) HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
[
https://issues.apache.org/jira/browse/MAPREDUCE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-4577:
--
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this to trunk and branch-2. Thanks a lot for the quick
reviews, guys.
HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
---
Key: MAPREDUCE-4577
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4577
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.2.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Aaron T. Myers
Priority: Minor
Fix For: 2.2.0-alpha
Attachments: HDFS-3842.patch
Before HDFS-3672, locally applying MAPREDUCE-4470 made
TestCombineFileInputFormat to pass all it tests.
After HDFS-3672, TestCombineFileInputFormat.testMissingBlocks() fails:
{code}
$ mvn clean test -Dtest=TestCombineFileInputFormat
Running org.apache.hadoop.mapred.TestCombineFileInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 sec
Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.526 sec
FAILURE!
Results :
Tests in error:
testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat):
org.apache.hadoop.fs.BlockLocation
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4470) Fix TestCombineFileInputFormat.testForEmptyFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439922#comment-13439922 ] Aaron T. Myers commented on MAPREDUCE-4470: --- I've just committed the fix for the ArrayStoreException in TestCombineFileInputFormat as described in MAPREDUCE-4577. We should be good to go on this JIRA. Fix TestCombineFileInputFormat.testForEmptyFile --- Key: MAPREDUCE-4470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4470 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.0-alpha Reporter: Kihwal Lee Fix For: 2.1.0-alpha, 3.0.0 Attachments: MAPREDUCE-4470-2.patch, MAPREDUCE-4470.patch, TestFileInputFormat.java, TestFileInputFormat.java TestCombineFileInputFormat.testForEmptyFile started failing after HADOOP-8599. It expects one split on an empty input file, but with HADOOP-8599 it gets zero. The new behavior seems correct, but is it breaking anything else? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4417) add support for encrypted shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13411703#comment-13411703 ] Aaron T. Myers commented on MAPREDUCE-4417: --- bq. Do you already have an HDFS solution in place? This only covers a fraction of the data traffic. Just filed: HDFS-3637 add support for encrypted shuffle - Key: MAPREDUCE-4417 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4417 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2, security Affects Versions: 2.0.0-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.1-alpha Currently Shuffle fetches go on the clear. While Kerberos provides comprehensive authentication for the cluster, it does not provide confidentiality. When processing sensitive data confidentiality may be desired (at the expense of job performance and resources utilization for doing encryption). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4390) java.io.IOException: File /user/XXXX/QuasiMonteCarlo_TMP_3_141592654/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datano
[ https://issues.apache.org/jira/browse/MAPREDUCE-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-4390. --- Resolution: Invalid Hi Srikanth, the Apache JIRA is for tracking established bugs or improvements. This looks like an operational issue. Specifically, it looks like you don't have any DNs running. I recommend you email mapreduce-u...@hadoop.apache.org to get more help with this. java.io.IOException: File /user//QuasiMonteCarlo_TMP_3_141592654/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. - Key: MAPREDUCE-4390 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4390 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples, job submission, mrv2 Affects Versions: 0.23.0 Environment: Ubuntu Server 11.04 Reporter: srikanth ayalasomayajulu Labels: hadoop Fix For: 0.23.0 Original Estimate: 2h Remaining Estimate: 2h Tried to run an example program on hadoop0.23.0 and getting the following error. error: java.io.IOException: File /user/X/QuasiMonteCarlo_TMP_3_141592654/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1181) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1486) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:390) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4391) datanode.DataNode (DataNode.java:handshake(820)) - Problem connecting to server: master/192.168.100.140:9000
[ https://issues.apache.org/jira/browse/MAPREDUCE-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-4391. --- Resolution: Invalid Hi Srikanth, Apache JIRA is for tracking confirmed bugs or improvements. This issue looks like a misconfiguration. You'll probably get more help by emailing hdfs-u...@hadoop.apache.org with a description of your issue. datanode.DataNode (DataNode.java:handshake(820)) - Problem connecting to server: master/192.168.100.140:9000 Key: MAPREDUCE-4391 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4391 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Environment: Ubuntu Server 11.04, Hadoop 0.23.0 Reporter: srikanth ayalasomayajulu Labels: datanode, hadoop Fix For: 0.23.0 Original Estimate: 2h Remaining Estimate: 2h datanode cannot able to connect to namenode, and in turn resulting in future errors during running examples. 2012-07-04 15:25:09,636 WARN datanode.DataNode (DataNode.java:handshake(820)) - Problem connecting to server: master/192.168.100.140:9000 2012-07-04 15:25:15,638 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 0 time(s). 2012-07-04 15:25:16,640 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 1 time(s). 2012-07-04 15:25:17,642 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 2 time(s). 2012-07-04 15:25:18,643 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 3 time(s). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4328) Add the option to quiesce the JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13295192#comment-13295192 ] Aaron T. Myers commented on MAPREDUCE-4328: --- Seems like we should also implement an analogous feature in trunk/2.0, so as not to have a feature regression from branch-1. Add the option to quiesce the JobTracker Key: MAPREDUCE-4328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4328 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.0.3 Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: MAPREDUCE-4328.patch In several failure scenarios it would be very handy to have an option to quiesce the JobTracker. Recently, we saw a case where the NameNode had to be rebooted at a customer due to a random hardware failure - in such a case it would have been nice to not lose jobs by quiescing the JobTracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4170) Move sleep and fail jobs from tests module to examples module
Aaron T. Myers created MAPREDUCE-4170: - Summary: Move sleep and fail jobs from tests module to examples module Key: MAPREDUCE-4170 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4170 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples, test Affects Versions: 2.0.0 Reporter: Aaron T. Myers Priority: Minor The sleep job used to be in the examples jar in MR1. I'm not quite sure when, but the sleep job has been moved to the tests module/jar in MR2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4170) Move sleep and fail jobs from tests module to examples module
[
https://issues.apache.org/jira/browse/MAPREDUCE-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258644#comment-13258644
]
Aaron T. Myers commented on MAPREDUCE-4170:
---
The sleep and fail jobs seem like they belong more with examples than with
tests, IMO.
Here are the programs in the tests jar:
{code}
DFSCIOTest: Distributed i/o benchmark of libhdfs.
DistributedFSCheck: Distributed checkup of the file system consistency.
JHLogAnalyzer: Job History Log analyzer.
MRReliabilityTest: A program that tests the reliability of the MR framework
by injecting faults/failures
SliveTest: HDFS Stress Test and Live Data Verification.
TestDFSIO: Distributed i/o benchmark.
fail: a job that always fails
filebench: Benchmark SequenceFile(Input|Output)Format (block,record
compressed and uncompressed), Text(Input|Output)Format (compressed and
uncompressed)
loadgen: Generic map/reduce load generator
mapredtest: A map/reduce test check.
mrbench: A map/reduce benchmark that can create many small jobs
nnbench: A benchmark that stresses the namenode.
sleep: A job that sleeps at each map and reduce task.
testbigmapoutput: A map/reduce program that works on a very big
non-splittable file and does identity map/reduce
testfilesystem: A test for FileSystem read/write.
testmapredsort: A map/reduce program that validates the map-reduce
framework's sort.
testsequencefile: A test for flat files of binary key value pairs.
testsequencefileinputformat: A test for sequence file input format.
testtextinputformat: A test for text input format.
threadedmapbench: A map/reduce benchmark that compares the performance of
maps with multiple spills over maps with 1 spill
{code}
and here are the programs in the examples jar:
{code}
aggregatewordcount: An Aggregate based map/reduce program that counts the
words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the
histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact
digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact
bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino
problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual
data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
{code}
Move sleep and fail jobs from tests module to examples module
-
Key: MAPREDUCE-4170
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4170
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples, test
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Priority: Minor
The sleep job used to be in the examples jar in MR1. I'm not quite sure
when, but the sleep job has been moved to the tests module/jar in MR2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4170) Move sleep and fail jobs from tests module to examples module
[ https://issues.apache.org/jira/browse/MAPREDUCE-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258645#comment-13258645 ] Aaron T. Myers commented on MAPREDUCE-4170: --- Come to think of it, maybe JHLogAnalyzer should be moved as well, but that I don't feel as strongly about that as I do about sleep and fail. Move sleep and fail jobs from tests module to examples module - Key: MAPREDUCE-4170 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4170 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples, test Affects Versions: 2.0.0 Reporter: Aaron T. Myers Priority: Minor The sleep job used to be in the examples jar in MR1. I'm not quite sure when, but the sleep job has been moved to the tests module/jar in MR2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2934) MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
[
https://issues.apache.org/jira/browse/MAPREDUCE-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-2934:
--
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks a lot for the review, Todd.
MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
--
Key: MAPREDUCE-2934
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2934
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: mapreduce-2934.0.patch
Once HADOOP-7607 goes in, {{ProtoOverHadoopRpcEngine.stopProxy}} will need to
be removed or at least have its {{@Override}} annotation removed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2936) Contrib Raid compilation broken after HDFS-1620
[ https://issues.apache.org/jira/browse/MAPREDUCE-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098178#comment-13098178 ] Aaron T. Myers commented on MAPREDUCE-2936: --- Thanks a lot for taking care of this, Vinod. I'm sorry I missed this when committing it. In the future, this should be noticed earlier once HADOOP-7612 goes in. Contrib Raid compilation broken after HDFS-1620 --- Key: MAPREDUCE-2936 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2936 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: MAPREDUCE-2936-20110906.txt After working around MAPREDUCE-2935 by removing TestServiceLevelAuthorization and runing the following: At the trunk level: mvn clean install package -Dtar -Pdist -Dmaven.test.skip.exec=true In hadoop-mapreduce-project: ant compile-contrib -Dresolvers=internal yields 14 errors. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2934) MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
[
https://issues.apache.org/jira/browse/MAPREDUCE-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-2934:
--
Status: Patch Available (was: Open)
MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
--
Key: MAPREDUCE-2934
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2934
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: mapreduce-2934.0.patch
Once HADOOP-7607 goes in, {{ProtoOverHadoopRpcEngine.stopProxy}} will need to
be removed or at least have its {{@Override}} annotation removed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2934) MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
--
Key: MAPREDUCE-2934
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2934
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Once HADOOP-7607 goes in, {{ProtoOverHadoopRpcEngine.stopProxy}} will need to
be removed or at least have its {{@Override}} annotation removed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2934) MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
[
https://issues.apache.org/jira/browse/MAPREDUCE-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-2934:
--
Attachment: mapreduce-2934.0.patch
Patch which addresses the issue.
MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
--
Key: MAPREDUCE-2934
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2934
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: mapreduce-2934.0.patch
Once HADOOP-7607 goes in, {{ProtoOverHadoopRpcEngine.stopProxy}} will need to
be removed or at least have its {{@Override}} annotation removed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2926) 500 Error in ResourceManager UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096533#comment-13096533
]
Aaron T. Myers commented on MAPREDUCE-2926:
---
I just committed HADOOP-7606. Is this now fixed?
500 Error in ResourceManager UI
---
Key: MAPREDUCE-2926
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2926
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Fix For: 0.23.0, 0.24.0
Attachments: MR-2926-v1.patch, MR-2926-v2.patch
When accessing the resource manager UI the following is returned
{noformat}
Problem accessing /. Reason:
org.codehaus.jackson.type.JavaType.init(Ljava/lang/Class;)V
Caused by:
java.lang.NoSuchMethodError:
org.codehaus.jackson.type.JavaType.init(Ljava/lang/Class;)V
at org.codehaus.jackson.map.type.TypeBase.init(TypeBase.java:15)
at org.codehaus.jackson.map.type.SimpleType.init(SimpleType.java:45)
at org.codehaus.jackson.map.type.SimpleType.init(SimpleType.java:40)
at
org.codehaus.jackson.map.type.TypeBindings.clinit(TypeBindings.java:20)
at
org.codehaus.jackson.map.type.TypeFactory._fromType(TypeFactory.java:530)
at org.codehaus.jackson.map.type.TypeFactory.type(TypeFactory.java:63)
at org.codehaus.jackson.map.ObjectMapper.clinit(ObjectMapper.java:179)
at org.apache.hadoop.yarn.webapp.Controller.clinit(Controller.java:43)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
com.google.inject.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:81)
at
com.google.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at
com.google.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:111)
at com.google.inject.InjectorImpl$4$1.call(InjectorImpl.java:758)
at com.google.inject.InjectorImpl.callInContext(InjectorImpl.java:804)
at com.google.inject.InjectorImpl$4.get(InjectorImpl.java:754)
at com.google.inject.InjectorImpl.getInstance(InjectorImpl.java:793)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:136)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:892)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Powered by Jetty://
{noformat}
--
This
[jira] [Commented] (MAPREDUCE-2853) Add teraread example
[ https://issues.apache.org/jira/browse/MAPREDUCE-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086735#comment-13086735 ] Aaron T. Myers commented on MAPREDUCE-2853: --- Patch looks pretty good, Todd. The only thing I notice is that it looks like there are a few unused imports in the file. +1 pending removal of those. Add teraread example -- Key: MAPREDUCE-2853 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2853 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks, examples Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Attachments: mapreduce-2853.txt Teragen is a good benchmark of raw DFS write throughput. Terasort is a good benchmark of the whole MR system (input, shuffle, output). I've added a simple teraread example which reads through the terasort input data without performing any processing: this acts as a good benchmark of a read-only workload (similar to real-life find a needle in a haystack MR jobs) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-109) Setting up ctr-A as custom delimiter for mapred.textoutputformat.separator
[ https://issues.apache.org/jira/browse/MAPREDUCE-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned MAPREDUCE-109: Assignee: Michael Katzenellenbogen Setting up ctr-A as custom delimiter for mapred.textoutputformat.separator Key: MAPREDUCE-109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-109 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.2, 0.23.0 Reporter: Suhas Gogate Assignee: Michael Katzenellenbogen Attachments: MAPREDUCE-109-v2.patch, MAPREDUCE-109.patch Feature added by this Jira has a problem while setting up some of the invalid xml characters e.g. ctrl-A e.g. mapred.textoutputformat.separator = \u0001 e,g, String delim = \u0001; Conf.set(mapred.textoutputformat.separator, delim); Job client serializes the jobconf with mapred.textoutputformat.separator set to \u0001 (ctrl-A) and problem happens when it is de-serialized (read back) by job tracker, where it encounters invalid xml character. The test for this feature public : testFormatWithCustomSeparator() does not serialize the jobconf after adding the separator as ctrl-A and hence does not detect the specific problem. Here is an exception: 08/12/06 01:40:50 INFO mapred.FileInputFormat: Total input paths to process : 1 org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference #1 is an invalid XML character. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:961) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:864) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:832) at org.apache.hadoop.conf.Configuration.get(Configuration.java:291) at org.apache.hadoop.mapred.JobConf.getJobPriority(JobConf.java:1163) at org.apache.hadoop.mapred.JobInProgress.init(JobInProgress.java:179) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) at org.apache.hadoop.ipc.Client.call(Client.java:715) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026) at -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-109) Setting up ctr-A as custom delimiter for mapred.textoutputformat.separator
[
https://issues.apache.org/jira/browse/MAPREDUCE-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082990#comment-13082990
]
Aaron T. Myers commented on MAPREDUCE-109:
--
Hey Michael, patch looks pretty good to me. Two tiny stylistic comments:
# Please put spaces around = in {{out=new BufferedWriter(new
FileWriter(CONFIG));}}
# The indentation is wrong in the change to {{Configuration.java}}. Hadoop uses
2 space indentation, not 4.
Setting up ctr-A as custom delimiter for mapred.textoutputformat.separator
Key: MAPREDUCE-109
URL: https://issues.apache.org/jira/browse/MAPREDUCE-109
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.2, 0.23.0
Reporter: Suhas Gogate
Assignee: Michael Katzenellenbogen
Attachments: MAPREDUCE-109-v2.patch, MAPREDUCE-109.patch
Feature added by this Jira has a problem while setting up some of the invalid
xml characters e.g. ctrl-A e.g. mapred.textoutputformat.separator = \u0001
e,g,
String delim = \u0001;
Conf.set(mapred.textoutputformat.separator, delim);
Job client serializes the jobconf with mapred.textoutputformat.separator set
to \u0001 (ctrl-A) and problem happens when it is de-serialized (read back)
by job tracker, where it encounters invalid xml character.
The test for this feature public : testFormatWithCustomSeparator() does not
serialize the jobconf after adding the separator as ctrl-A and hence does not
detect the specific problem.
Here is an exception:
08/12/06 01:40:50 INFO mapred.FileInputFormat: Total input paths to process :
1
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.RuntimeException: org.xml.sax.SAXParseException: Character
reference #1 is an invalid XML
character.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:961)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:864)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:832)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:291)
at
org.apache.hadoop.mapred.JobConf.getJobPriority(JobConf.java:1163)
at
org.apache.hadoop.mapred.JobInProgress.init(JobInProgress.java:179)
at
org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.ipc.Client.call(Client.java:715)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
at
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2797) Some java files cannot be compiled
[ https://issues.apache.org/jira/browse/MAPREDUCE-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2797: -- Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) +1, the patch looks good to me. Some java files cannot be compiled -- Key: MAPREDUCE-2797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2797 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/raid, test Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: m2797_20110810.patch Due to the changes in HDFS-2239, the following files cannot be compiled (Thanks Amar for pointing them out.) 1. src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java 2. src/test/mapred/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java 3. src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCacheOldApi.java 4. src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRaid.java -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2767) Remove Linux task-controller from 0.22 branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080229#comment-13080229 ] Aaron T. Myers commented on MAPREDUCE-2767: --- Hey Milind, I'm pretty sure you have to actually attach a new patch. Simply going Patch Available - Open - Patch Available isn't sufficient, I believe. Remove Linux task-controller from 0.22 branch - Key: MAPREDUCE-2767 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2767 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Milind Bhandarkar Assignee: Milind Bhandarkar Priority: Blocker Fix For: 0.22.0 Attachments: MR2767.patch There's a potential security hole in the task-controller as it stands. Based on the discussion on general@, removing task-controller from the 0.22 branch will pave way for 0.22.0 release. (This was done for the 0.21.0 release as well: see MAPREDUCE-2014.) We can roll a 0.22.1 release with the task-controller when it is fixed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2732) Some tests using FSNamesystem.LOG cannot be compiled
[ https://issues.apache.org/jira/browse/MAPREDUCE-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071545#comment-13071545 ] Aaron T. Myers commented on MAPREDUCE-2732: --- I'm not an M/R committer, but this patch looks good to me. Some tests using FSNamesystem.LOG cannot be compiled Key: MAPREDUCE-2732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2732 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: m2723_20110726.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data from the distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070334#comment-13070334 ] Aaron T. Myers commented on MAPREDUCE-2572: --- Hey Robert, even if no algorithmic changes are necessary to address this issue, perhaps a good change to make would be to change the default value of this config to a higher value? Throttle the deletion of data from the distributed cache Key: MAPREDUCE-2572 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Affects Versions: 0.20.205.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Attachments: THROTTLING-security-v1.patch When deleting entries from the distributed cache we do so in a background thread. Once the size limit of the distributed cache is reached all unused entries are deleted. MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold. In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive. It would be better to be able to throttle this deletion so that it is spread out over a longer period of time. This jira is to add in this throttling. On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2705) tasks localized and launched serially by TaskLauncher - causing other tasks to be delayed
[ https://issues.apache.org/jira/browse/MAPREDUCE-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069099#comment-13069099 ] Aaron T. Myers commented on MAPREDUCE-2705: --- Hey Tom, simply changing the JIRA status to be PA - Open - PA won't retrigger Hudson. You either need to upload another attachment (could be the same contents) or get someone with Hudson access to retrigger the build for you. I just did the latter. You can watch the build here: https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/489/ tasks localized and launched serially by TaskLauncher - causing other tasks to be delayed - Key: MAPREDUCE-2705 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2705 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.20.205.0, 0.23.0 Attachments: MAPREDUCE-2705-branch20.patch, MAPREDUCE-2705-trunk.patch The current TaskLauncher serially launches new tasks one at a time. During the launch it does the localization and then starts the map/reduce task. This can cause any other tasks to be blocked waiting for the current task to be localized and started. In some instances we have seen a task that has a large file to localize (1.2MB) block another task for about 40 minutes. This particular task being blocked was a cleanup task which caused the job to be delayed finishing for the 40 minutes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2684) Job Tracker can starve reduces with very large input.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064923#comment-13064923 ] Aaron T. Myers commented on MAPREDUCE-2684: --- Hey Robert, is this not a duplicate of https://issues.apache.org/jira/browse/MAPREDUCE-2324 ? Job Tracker can starve reduces with very large input. - Key: MAPREDUCE-2684 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2684 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.204.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans If mapreduce.reduce.input.limit is mis-configured or if a cluster is just running low on disk space in general then reduces with large a input may never get scheduled causing the Job to never fail and never succeed, just starve until the job is killed. The JobInProgess tries to guess at the size of the input to all reducers in a job. If the size is over mapreduce.reduce.input.limit then the job is killed. If it is not then findNewReduceTask() checks to see if the estimated size is too big to fit on the node currently looking for work. If it is not then it will let some other task have a chance at the slot. The idea is to keep track of how often it happens that a Reduce Slot is rejected because of the lack of space vs how often it succeeds and then guess if the reduce tasks will ever be scheduled. So I would like some feedback on this. 1) How should we guess. Someone who found the bug here suggested P1 + (P2 * S), where S is the number of successful assignments. Possibly P1 = 20 and P2 = 2.0. I am not really sure. 2) What should we do when we guess that it will never get a slot? Should we fail the job or do we say, even though it might fail, well lets just schedule the it and see if it really will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2654) Missing events in Job History
[ https://issues.apache.org/jira/browse/MAPREDUCE-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064220#comment-13064220 ] Aaron T. Myers commented on MAPREDUCE-2654: --- Hey Keren, it would probably be good to attach the logs to this JIRA. Are you capable of reliably reproducing this? If so, how? Missing events in Job History - Key: MAPREDUCE-2654 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2654 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.21.0 Reporter: Keren Ouaknine Labels: hadoop -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2373) When tasks exit with a nonzero exit status, task runner should log the stderr as well as stdout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13060228#comment-13060228 ] Aaron T. Myers commented on MAPREDUCE-2373: --- The amendment patch looks good, Todd. Thanks for doing that. When tasks exit with a nonzero exit status, task runner should log the stderr as well as stdout --- Key: MAPREDUCE-2373 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2373 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: mapreduce-2373-on-20sec.txt, mr-2373-amendment.txt Currently, if the taskjvm.sh script fails to exec java for some reason, it prints its error message to stderr. This doesn't make it to the logs anywhere. Logging the stderr is very useful to understand why taskjvm.sh failed to start the Child jvm. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13060233#comment-13060233
]
Aaron T. Myers commented on MAPREDUCE-2644:
---
Hey Josh, the usual thing done in the rest of the Hadoop code when providing a
default value for a config is to use the two-argument version of
{{Configuration.get(...)}} whose first argument is the config key and whose
second argument will be returned as the value if no value for that key is set
in the conf object.
NodeManager fails to create containers when NM_LOG_DIR is not explicitly set
in the Configuration
-
Key: MAPREDUCE-2644
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Josh Wills
Assignee: Josh Wills
Attachments: MAPREDUCE-2644.patch
If the yarn configuration does not explicitly specify a value for the
yarn.server.nodemanager.log.dir property, container allocation will fail on
the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp
directory. In most of the code, we handle this by defaulting to /tmp/logs,
but we cannot do this in the LocalDirAllocator context, so we need to set the
default value explicitly in the Configuration.
Marking this as major b/c it's annoying to bump into it when you're getting
your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2644) NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration
[
https://issues.apache.org/jira/browse/MAPREDUCE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13060237#comment-13060237
]
Aaron T. Myers commented on MAPREDUCE-2644:
---
One option would be to change {{LocalDirAllocator}} to accept a default value
for the config key you pass in. Another (probably better) option would be to
just add a default value to the {{mapred-default.xml}} file, which gets loaded
any time an MR config object is created.
NodeManager fails to create containers when NM_LOG_DIR is not explicitly set
in the Configuration
-
Key: MAPREDUCE-2644
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2644
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Josh Wills
Assignee: Josh Wills
Attachments: MAPREDUCE-2644.patch
If the yarn configuration does not explicitly specify a value for the
yarn.server.nodemanager.log.dir property, container allocation will fail on
the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp
directory. In most of the code, we handle this by defaulting to /tmp/logs,
but we cannot do this in the LocalDirAllocator context, so we need to set the
default value explicitly in the Configuration.
Marking this as major b/c it's annoying to bump into it when you're getting
your first MRv2 cluster up and running. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2639) MR-279: Fixup the exponentially smoothed runtime estimator, fix a couple of bugs in DataStatistics, and do a little bit of cleanup.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13059644#comment-13059644 ] Aaron T. Myers commented on MAPREDUCE-2639: --- Hey Josh, I'd recommend breaking this out into separate JIRAs as subtasks of this one, so that those who want to give feedback can localize the review/discussion to smaller areas. MR-279: Fixup the exponentially smoothed runtime estimator, fix a couple of bugs in DataStatistics, and do a little bit of cleanup. --- Key: MAPREDUCE-2639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2639 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Environment: All Reporter: Josh Wills Assignee: Josh Wills Priority: Minor Attachments: MAPREDUCE-2639-1.patch, MAPREDUCE-2639-2.patch, MAPREDUCE-2639-3.patch A catch-all JIRA for a pass I took through the v2.app.speculate package. 1) Fixed the ExponentiallySmoothedTaskRuntimeEstimator so that it can run and pass the test defined in TestRuntimeEstimators. 2) Fixed two bugs in DataStatistics: 1) a divide by zero in the variance calculation in the case that count == 0 and 2) a synchronization issue in how the updateStatistics method was implemented, 3) A bunch of typo corrections, formatting fixes, and adding some consistency around the null value checking. I probably need to do a couple more passes through this code to get it into better shape, but this seemed like a good start. Will attach my patch momentarily. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2560) Support specification of codecs by name
[ https://issues.apache.org/jira/browse/MAPREDUCE-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned MAPREDUCE-2560: - Assignee: Arun Ramakrishnan (was: Anthony Urso) Support specification of codecs by name --- Key: MAPREDUCE-2560 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2560 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Assignee: Arun Ramakrishnan Labels: newbie By changing the code to take advantage of HADOOP-7323, it will be possible to specify compression codecs in configuration by name (e.g. 'gzip'), not only by classname, although that will still be supported, of course (e.g. 'org.apache.hadoop.io.compress.GzipCodec'). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050148#comment-13050148 ] Aaron T. Myers commented on MAPREDUCE-2550: --- Todd, are you satisfied with Eric's response to your question? If so, I'll go ahead and commit this. bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050149#comment-13050149 ] Aaron T. Myers commented on MAPREDUCE-2550: --- Never mind - I can't commit this since it's M/R. :P Todd, are you satisfied with Eric's response to your question? If so, would you mind committing this? bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049310#comment-13049310 ] Aaron T. Myers commented on MAPREDUCE-2589: --- Hi Sherry, does this issue not also affect trunk? If so, would you mind preparing a trunk patch as well? TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Attachments: MAPREDUCE-2589.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2550) bin/mapred no longer works from a source checkout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13046168#comment-13046168 ] Aaron T. Myers commented on MAPREDUCE-2550: --- +1. I applied the patch to trunk and can confirm that `bin/mapred' did not work without this patch, and does work with it. bin/mapred no longer works from a source checkout - Key: MAPREDUCE-2550 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2550 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.20.3 Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Blocker Fix For: 0.20.3 Attachments: MAPREDUCE-2550-1.patch, MAPREDUCE-2550.patch Developer may want to run hadoop without extracting tarball. It would be nice if existing method to run mapred scripts from source code is preserved for developers. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-2109. --- Resolution: Won't Fix Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt, mapreduce-2109.1.txt, mapreduce-2109.2.txt, mapreduce-2109.3.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2473: -- Attachment: mapreduce-2473.3.patch Rebased patch against trunk. MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 Attachments: mapreduce-2473.0.patch, mapreduce-2473.1.patch, mapreduce-2473.2.patch, mapreduce-2473.3.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2473: -- Attachment: mapreduce-2473.1.patch Updated patch addressing Todd's comments. MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 Attachments: mapreduce-2473.0.patch, mapreduce-2473.1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2473: -- Attachment: hadoop-7214.10.patch D'oh! I missed a license header. Fixed in this patch. MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 Attachments: hadoop-7214.10.patch, mapreduce-2473.0.patch, mapreduce-2473.1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2473: -- Attachment: (was: hadoop-7214.10.patch) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 Attachments: mapreduce-2473.0.patch, mapreduce-2473.1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2473: -- Attachment: mapreduce-2473.2.patch Thanks for the comments, Todd. Updated patch attached. MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 Attachments: mapreduce-2473.0.patch, mapreduce-2473.1.patch, mapreduce-2473.2.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2472) Extra whitespace in mapred.child.java.opts breaks JVM initialization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned MAPREDUCE-2472: - Assignee: Aaron T. Myers Extra whitespace in mapred.child.java.opts breaks JVM initialization Key: MAPREDUCE-2472 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2472 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.2, 0.22.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: mapreduce-2472.0.patch When creating taskjvm.sh, we split mapred.child.java.opts on and then create a quoted argument for each of those results. So, if you have an extra space anywhere in this configuration, you get an argument '' in the child command line, which the JVM interprets as an empty class name. This results in a ClassNotFoundException and the task cannot run. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2472) Extra whitespace in mapred.child.java.opts breaks JVM initialization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2472: -- Attachment: mapreduce-2472.0.patch Patch which changes the split from being on spaces to any amount of whitespace. Add a few tests for this as well. Extra whitespace in mapred.child.java.opts breaks JVM initialization Key: MAPREDUCE-2472 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2472 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.2, 0.22.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: mapreduce-2472.0.patch When creating taskjvm.sh, we split mapred.child.java.opts on and then create a quoted argument for each of those results. So, if you have an extra space anywhere in this configuration, you get an argument '' in the child command line, which the JVM interprets as an empty class name. This results in a ClassNotFoundException and the task cannot run. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2472) Extra whitespace in mapred.child.java.opts breaks JVM initialization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2472: -- Status: Patch Available (was: Open) Extra whitespace in mapred.child.java.opts breaks JVM initialization Key: MAPREDUCE-2472 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2472 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.2, 0.22.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: mapreduce-2472.0.patch When creating taskjvm.sh, we split mapred.child.java.opts on and then create a quoted argument for each of those results. So, if you have an extra space anywhere in this configuration, you get an argument '' in the child command line, which the JVM interprets as an empty class name. This results in a ClassNotFoundException and the task cannot run. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2472) Extra whitespace in mapred.child.java.opts breaks JVM initialization
[ https://issues.apache.org/jira/browse/MAPREDUCE-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2472: -- Attachment: mapreduce-2472.1.patch Thanks for the comments, Todd. Updated patch to address Todd's comments. Also refactored the tests a little bit and added tests to check for proper handling of leading or trailing whitespace. Extra whitespace in mapred.child.java.opts breaks JVM initialization Key: MAPREDUCE-2472 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2472 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.2, 0.22.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: mapreduce-2472.0.patch, mapreduce-2472.1.patch When creating taskjvm.sh, we split mapred.child.java.opts on and then create a quoted argument for each of those results. So, if you have an extra space anywhere in this configuration, you get an argument '' in the child command line, which the JVM interprets as an empty class name. This results in a ClassNotFoundException and the task cannot run. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[
https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated MAPREDUCE-2473:
--
Attachment: mapreduce-2473.0.patch
Patch which uses the {{RefreshUserMappingsProtocol}}. This patch is dependent
upon the latest patch posted to HADOOP-7214.
MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
-
Key: MAPREDUCE-2473
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.23.0
Attachments: mapreduce-2473.0.patch
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2473: -- Status: Patch Available (was: Open) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 Attachments: mapreduce-2473.0.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[
https://issues.apache.org/jira/browse/MAPREDUCE-2473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029129#comment-13029129
]
Aaron T. Myers commented on MAPREDUCE-2473:
---
{quote}
It seems unfortunate that we're lumping in the getGroups API (an action
intended for non-admin usage) with the refresh methods (admin actions).
Unfortunately the ACLs for this whole Protocol are determined by
security.refresh.user.mappings.protocol.acl which is a misnomer when it also
affects the group command.
Any thoughts on a way around this?
{quote}
Certainly. We could have a separate protocol interface for refreshing versus
fetching user group mappings. It seemed logical to me to lump the two together,
since they both cover user - group mapping, but I don't feel very strongly
about this.
MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
-
Key: MAPREDUCE-2473
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.23.0
Attachments: mapreduce-2473.0.patch
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2457) job submission should inject group.name (on the JT side)
[
https://issues.apache.org/jira/browse/MAPREDUCE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026891#comment-13026891
]
Aaron T. Myers commented on MAPREDUCE-2457:
---
Since we're now adding the concept of a primary group, I wonder if we
shouldn't make this explicit, rather than just pick the first from the list.
i.e. change {{GroupMappingServiceProvider}} to add a {{public String
getPrimaryGroup(String user)}} method, and add a method to
{{UserGroupInformation}} to get the primary group of the user.
Thoughts?
job submission should inject group.name (on the JT side)
Key: MAPREDUCE-2457
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2457
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.21.0, 0.22.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
Fix For: 0.22.0
Attachments: MAPREDUCE-2457-1.patch, MAPREDUCE-2457.patch
Until Hadoop 0.20, the JobClient was injecting the property 'group.name' on
the JobConf submitted to the JobTracker.
Since Hadoop 0.21, due to security related changes, this is not done anymore.
This breaks backwards compatibility for jobs/components that expect the
'group.name' to be automatically set at submission time.
An example of a component being affected by this change is the FairScheduler
where it is common to use the group.name as pool name. Different from other
properties, a special characteristic of the group.name is that its value
cannot be tampered by a user.
For security reasons this should not be done (as it was done before) in the
JobClient side. Instead, it should be done in the JobTracker when the JobConf
is received.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2340) optimize JobInProgress.initTasks()
[
https://issues.apache.org/jira/browse/MAPREDUCE-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12997292#comment-12997292
]
Aaron T. Myers commented on MAPREDUCE-2340:
---
Thanks a lot for doing this performance analysis, Kang. Your results seem
promising.
Quick comment on the patch: it seems to me that if you find {{node}} to be
{{null}} you should then be assigning the result of
{{jobtracker.resolveAndAddToTopology(host)}} to the {{node}} variable. As it
stands {{node}} will still be null entering the loop.
optimize JobInProgress.initTasks()
--
Key: MAPREDUCE-2340
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2340
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Affects Versions: 0.20.1, 0.21.0
Reporter: Kang Xiao
Attachments: MAPREDUCE-2340.patch
JobTracker's hostnameToNodeMap cache can speed up JobInProgress.initTasks()
and JobInProgress.createCache() significantly. A test for 1 job with 10
maps on a 2400 cluster shows nearly 10 and 50 times speed up for initTasks()
and createCache().
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (MAPREDUCE-2276) Fix build failure introduced by HDFS-1547
[ https://issues.apache.org/jira/browse/MAPREDUCE-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-2276. --- Resolution: Duplicate Resolving as duplicate of https://issues.apache.org/jira/browse/HDFS-1585 Fix build failure introduced by HDFS-1547 - Key: MAPREDUCE-2276 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2276 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.23.0 MiniDFSCluster#startDataNodes() method signature changes introduced by HDFS-1547 breaks the mapreduce build -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2158) Remove need for taskcontroller.cfg
Remove need for taskcontroller.cfg -- Key: MAPREDUCE-2158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2158 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task-controller Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers The taskcontroller.cfg configuration file seems unnecessary. It's parsed by the task-controller binary, and the path to it is baked into that binary at compile-time. It contains only 3 values, none of which are secret, all of which are also specified in either core-site.xml or mapred-site.xml. It seems like getting rid of taskcontroller.cfg will simplify configuration, and have no impact on the security of the system I suggest we either: # Pass these values as arguments to task-controller when we execute it. # Use xerces to enable the task-controller to parse *-site.xml. Of these two options, I think option 1 seems the cleanest/easiest. Either would be backward-compatible with existing configurations. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2109: -- Attachment: mapreduce-2109.3.txt Thanks for the review, Eli. I've updated the patch to address your comments. Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt, mapreduce-2109.1.txt, mapreduce-2109.2.txt, mapreduce-2109.3.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12919077#action_12919077 ] Aaron T. Myers commented on MAPREDUCE-2109: --- Hi Arun, Several possible use cases are described in HADOOP-6988, but to summarize: It's presently possible to fetch delegation tokens to files, and to pass these to bin/hadoop when submitting a job. I'd like to be able to specify both MR and HDFS delegation token files upfront so that job submission (eg done by Hue or Hive) doesn't require using kinit. This change could also be useful to pass delegation token files obtained from multiple distinct NNs. Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt, mapreduce-2109.1.txt, mapreduce-2109.2.txt, mapreduce-2109.3.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2109: -- Attachment: mapreduce-2109.0.txt Adding support for HADOOP_TOKEN_FILE_LOCATION being interpreted as a comma-separated list of paths to delegation token files. Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2109: -- Attachment: mapreduce-2109.1.txt Same patch, this time with -p0. Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt, mapreduce-2109.1.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-2109: -- Attachment: mapreduce-2109.2.txt Removed some log output that shouldn't have been included. Apologies for the noise. Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt, mapreduce-2109.1.txt, mapreduce-2109.2.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2101) compile-mapred-test broken in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12916792#action_12916792
]
Aaron T. Myers commented on MAPREDUCE-2101:
---
I think you just need to update your checkout of HDFS trunk. This method was
added in HDFS-1399.
compile-mapred-test broken in trunk
---
Key: MAPREDUCE-2101
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2101
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ramkumar Vadali
r1002905 seems to have caused it.
{code}
compile-mapred-test:
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/classes
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/testjar
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/testshell
[javac] /data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:587:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 320 source files to
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/classes
[javac]
/data/users/rvadali/apache/hadoop-mapred-trunk-2/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java:62:
cannot find symbol
[javac] symbol : method
getRpcServer(org.apache.hadoop.hdfs.server.namenode.NameNode)
[javac] location: class
org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter
[javac] SetClass? protocolsWithAcls =
NameNodeAdapter.getRpcServer(dfs.getNameNode())
[javac]^
[javac]
/data/users/rvadali/apache/hadoop-mapred-trunk-2/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java:79:
cannot find symbol
[javac] symbol : method
getRpcServer(org.apache.hadoop.hdfs.server.namenode.NameNode)
[javac] location: class
org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter
[javac] protocolsWithAcls =
NameNodeAdapter.getRpcServer(dfs.getNameNode())
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:518: The following
error occurred while executing this line:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:587: Compile
failed; see the compiler error output for details.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2101) compile-mapred-test broken in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12916809#action_12916809
]
Aaron T. Myers commented on MAPREDUCE-2101:
---
It just worked for me. I ran the following, and it compiled just fine:
{code}ant veryclean compile-mapred-test{code}
compile-mapred-test broken in trunk
---
Key: MAPREDUCE-2101
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2101
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ramkumar Vadali
r1002905 seems to have caused it.
{code}
compile-mapred-test:
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/classes
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/testjar
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/testshell
[javac] /data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:587:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 320 source files to
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/classes
[javac]
/data/users/rvadali/apache/hadoop-mapred-trunk-2/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java:62:
cannot find symbol
[javac] symbol : method
getRpcServer(org.apache.hadoop.hdfs.server.namenode.NameNode)
[javac] location: class
org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter
[javac] SetClass? protocolsWithAcls =
NameNodeAdapter.getRpcServer(dfs.getNameNode())
[javac]^
[javac]
/data/users/rvadali/apache/hadoop-mapred-trunk-2/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java:79:
cannot find symbol
[javac] symbol : method
getRpcServer(org.apache.hadoop.hdfs.server.namenode.NameNode)
[javac] location: class
org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter
[javac] protocolsWithAcls =
NameNodeAdapter.getRpcServer(dfs.getNameNode())
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:518: The following
error occurred while executing this line:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:587: Compile
failed; see the compiler error output for details.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-2101) compile-mapred-test broken in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers reassigned MAPREDUCE-2101:
-
Assignee: Aaron T. Myers
compile-mapred-test broken in trunk
---
Key: MAPREDUCE-2101
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2101
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ramkumar Vadali
Assignee: Aaron T. Myers
r1002905 seems to have caused it.
{code}
compile-mapred-test:
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/classes
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/testjar
[mkdir] Created dir:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/testshell
[javac] /data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:587:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 320 source files to
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build/test/mapred/classes
[javac]
/data/users/rvadali/apache/hadoop-mapred-trunk-2/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java:62:
cannot find symbol
[javac] symbol : method
getRpcServer(org.apache.hadoop.hdfs.server.namenode.NameNode)
[javac] location: class
org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter
[javac] SetClass? protocolsWithAcls =
NameNodeAdapter.getRpcServer(dfs.getNameNode())
[javac]^
[javac]
/data/users/rvadali/apache/hadoop-mapred-trunk-2/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java:79:
cannot find symbol
[javac] symbol : method
getRpcServer(org.apache.hadoop.hdfs.server.namenode.NameNode)
[javac] location: class
org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter
[javac] protocolsWithAcls =
NameNodeAdapter.getRpcServer(dfs.getNameNode())
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:518: The following
error occurred while executing this line:
/data/users/rvadali/apache/hadoop-mapred-trunk-2/build.xml:587: Compile
failed; see the compiler error output for details.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2067) Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies
[ https://issues.apache.org/jira/browse/MAPREDUCE-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12915863#action_12915863 ] Aaron T. Myers commented on MAPREDUCE-2067: --- I ran test-core and all the tests passed. Here's the results of test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile. Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies -- Key: MAPREDUCE-2067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2067 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: mapreduce-2067.1.txt, mapreduce-2067.txt.0 MR portion of HADOOP-6951. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
