[jira] [Created] (MAPREDUCE-4054) Executing start-all.sh and stop-all.sh scripts not displaying proper message
Executing start-all.sh and stop-all.sh scripts not displaying proper message Key: MAPREDUCE-4054 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4054 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Nishan Shetty Priority: Trivial Execute start-all.sh and stop-all.sh scripts it displays message like This script is Deprecated. Instead use start-dfs.sh/stop-dfs.sh and start-mapred.sh/stop-mapred.sh. Instead of start-mapred.sh/stop-mapred.sh it should be start-yarn.sh/stop-yarn.sh. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4054) Fix issues in start-all.sh and stop-all.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4054: - Component/s: (was: mrv2) Description: 1. Warning message Execute start-all.sh and stop-all.sh scripts it displays message like This script is Deprecated. Instead use start-dfs.sh/stop-dfs.sh and start-mapred.sh/stop-mapred.sh. Instead of start-mapred.sh/stop-mapred.sh it should be start-yarn.sh/stop-yarn.sh. 2. start/stop hdfs 3. start/stop yarn was:Execute start-all.sh and stop-all.sh scripts it displays message like This script is Deprecated. Instead use start-dfs.sh/stop-dfs.sh and start-mapred.sh/stop-mapred.sh. Instead of start-mapred.sh/stop-mapred.sh it should be start-yarn.sh/stop-yarn.sh. Priority: Major (was: Trivial) Assignee: Devaraj K Summary: Fix issues in start-all.sh and stop-all.sh (was: Executing start-all.sh and stop-all.sh scripts not displaying proper message) Fix issues in start-all.sh and stop-all.sh -- Key: MAPREDUCE-4054 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4054 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.24.0 Reporter: Nishan Shetty Assignee: Devaraj K 1. Warning message Execute start-all.sh and stop-all.sh scripts it displays message like This script is Deprecated. Instead use start-dfs.sh/stop-dfs.sh and start-mapred.sh/stop-mapred.sh. Instead of start-mapred.sh/stop-mapred.sh it should be start-yarn.sh/stop-yarn.sh. 2. start/stop hdfs 3. start/stop yarn -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3315) Master-Worker Application on YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235534#comment-13235534 ] Sharad Agarwal commented on MAPREDUCE-3315: --- bq. Should I use Hadoop IPC or RMI Hadoop IPC bq. Should the Master be in the ApplicationManager or be run as a Container? For Master, you need to write a yarn ApplicationMaster (AM) see - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html AM runs in one of the container on the cluster. Also it would be good to completely separate the user API (writing master-worker apps) and the runtime implementation clearly. Master-Worker Application on YARN - Key: MAPREDUCE-3315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3315 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Sharad Agarwal Assignee: Sharad Agarwal Fix For: 0.24.0 Currently master worker scenarios are forced fit into Map-Reduce. Now with YARN, these can be first class and would benefit real/near realtime workloads and be more effective in using the cluster resources. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4050) Invalid node link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4050: -- Status: Patch Available (was: Open) Invalid node link - Key: MAPREDUCE-4050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4050 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4050.patch, MAPREDUCE-4050.png When a task is in *UNASSIGNED* state, node link is displayed as +null+. But I think it is better to display the link as *N/A* rather than +null+. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4050) Invalid node link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4050: -- Attachment: MAPREDUCE-4050.patch Please review the patch Invalid node link - Key: MAPREDUCE-4050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4050 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4050.patch, MAPREDUCE-4050.png When a task is in *UNASSIGNED* state, node link is displayed as +null+. But I think it is better to display the link as *N/A* rather than +null+. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4055) Job history files are not getting copied from intermediate done directory to done directory
Job history files are not getting copied from intermediate done directory to done directory Key: MAPREDUCE-4055 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4055 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Nishan Shetty 1.Submit job 2.After successful execution of job before the Job history files are copied from intermediate done directory to done directory,NameNode got killed. 3.Restart the NameNode after mapreduce.jobhistory.move.interval-ms time is elapsed(default is 3 min). Observe that Job history files are not copied from intermediate done directory to done directory and also logs are not updated with any message Now submit another job observe that Job history files are not copied from intermediate done directory to done directory and also nothing is logged into historyserver logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-120) Reducer sort failed due to wrong key class
[ https://issues.apache.org/jira/browse/MAPREDUCE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235576#comment-13235576 ] Paritosh Ranjan commented on MAPREDUCE-120: --- I am getting a similar/same problem : wrong key class: org.apache.hadoop.io.Text is not class org.apache.hadoop.io.LongWritable. But I want to write Text and that is what I have specified as output key class. Is there any solution to this problem? Reducer sort failed due to wrong key class -- Key: MAPREDUCE-120 URL: https://issues.apache.org/jira/browse/MAPREDUCE-120 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Runping Qi One of my job's reducers failed due to the following exception: java.io.IOException: wrong key class: class org.apache.hadoop.io.LongWritable is not class org.apache.hadoop.io.Text at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:2733) at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2542) at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2326) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:291) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760) I checked the segment files to be merged on the node running the reducer, and founded that one of the map output file did record org.apache.hadoop.io.LongWritable as its key class! The question is how that happened? My job conf specified Text as the map output key class, and other map output files had the correct key class! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4055) Job history files are not getting copied from intermediate done directory to done directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235586#comment-13235586 ] Robert Joseph Evans commented on MAPREDUCE-4055: I believe, and could very well be wrong, that the problem here is that some of the the threads that do cleanup etc in the history server when they catch an exception (like when the NameNode is down) they simply log the exception and exit the thread. Once the thread has exited we will not try any more to clean up or move data to done. Job history files are not getting copied from intermediate done directory to done directory Key: MAPREDUCE-4055 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4055 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Nishan Shetty 1.Submit job 2.After successful execution of job before the Job history files are copied from intermediate done directory to done directory,NameNode got killed. 3.Restart the NameNode after mapreduce.jobhistory.move.interval-ms time is elapsed(default is 3 min). Observe that Job history files are not copied from intermediate done directory to done directory and also logs are not updated with any message Now submit another job observe that Job history files are not copied from intermediate done directory to done directory and also nothing is logged into historyserver logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4050) Invalid node link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235595#comment-13235595 ] Hadoop QA commented on MAPREDUCE-4050: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519439/MAPREDUCE-4050.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2082//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2082//console This message is automatically generated. Invalid node link - Key: MAPREDUCE-4050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4050 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4050.patch, MAPREDUCE-4050.png When a task is in *UNASSIGNED* state, node link is displayed as +null+. But I think it is better to display the link as *N/A* rather than +null+. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Attachment: Hadoop Shuffle Consumer Plugin TLD.rtf Hadoop Shuffle Provider Plugin TLD.rtf Top Level Design for the Shuffle Consumer/Provider Plugin classes plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I will soon attach document with suggested API for the plugin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235637#comment-13235637 ] Avner BenHanoch commented on MAPREDUCE-4049: # I just attached top level design documents for both plugins, based on current 1.0 branch. # I want to have the described plugin-ability (desired with same interface) for all future versions of Hadoop (as mentioned in the Target Version/s field). # On the first phase, I am focusing on the existing 1.0 branch as I know it. In parallel, I'll try to learn what exists in 0.23 or in HDH. Later, I'll be glad to do any adjustments that may be needed for fulfilling the previous point (as much as possible). # I welcome you and other parties, to let me know if there are branch(es) that particularly interest you, so I'll give it priority. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I will soon attach document with suggested API for the plugin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235639#comment-13235639
]
B Anil Kumar commented on MAPREDUCE-4019:
-
I manually tested the correctness of this patch. And the above test failure is
not related to this patch.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.23.1, 0.23.2
Reporter: B Anil Kumar
Priority: Minor
Attachments: MAPREDUCE-4019.patch
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Description: Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) was: Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I will soon attach document with suggested API for the plugin Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4044) YarnClientProtocolProvider does not honor mapred.job.tracker property
[ https://issues.apache.org/jira/browse/MAPREDUCE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235666#comment-13235666 ] Alejandro Abdelnur commented on MAPREDUCE-4044: --- OK, but his does not address the issue of JobClient creation using a custom 'mapred.job.tracker' (and not hadoop site.xml files around) YarnClientProtocolProvider does not honor mapred.job.tracker property - Key: MAPREDUCE-4044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4044 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Alejandro Abdelnur Fix For: 0.23.3 The YarnClientProtocolProvider/YARNRunner/ResourceMgrDelegate bootstrap only looks for 'yarn.resourcemanager.address', they ignore 'mapred.job.tracker' This breaks backward compatibility and creates issues in Oozie. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4044) YarnClientProtocolProvider does not honor mapred.job.tracker property
[ https://issues.apache.org/jira/browse/MAPREDUCE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235672#comment-13235672 ] Mahadev konar commented on MAPREDUCE-4044: -- Actually am not sure if thats true. Have you tried it? YarnClientProtocolProvider does not honor mapred.job.tracker property - Key: MAPREDUCE-4044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4044 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Alejandro Abdelnur Fix For: 0.23.3 The YarnClientProtocolProvider/YARNRunner/ResourceMgrDelegate bootstrap only looks for 'yarn.resourcemanager.address', they ignore 'mapred.job.tracker' This breaks backward compatibility and creates issues in Oozie. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4044) YarnClientProtocolProvider does not honor mapred.job.tracker property
[
https://issues.apache.org/jira/browse/MAPREDUCE-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235734#comment-13235734
]
Alejandro Abdelnur commented on MAPREDUCE-4044:
---
ConfigUtil has:
Configuration.addDeprecation(mapred.job.tracker,
new String[] {JTConfig.RM_ADDRESS});
It seems we should add:
Configuration.addDeprecation(JTConfig.JT_IPC_ADDRESS,
new String[] {YarnConfiguration.RM_ADDRESS});
And also we may need to change Configuration logic to do a transitive closure
of deprecated keys.
This is becoming a mess.
Thoughts?
YarnClientProtocolProvider does not honor mapred.job.tracker property
-
Key: MAPREDUCE-4044
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4044
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Alejandro Abdelnur
Fix For: 0.23.3
The YarnClientProtocolProvider/YARNRunner/ResourceMgrDelegate bootstrap only
looks for 'yarn.resourcemanager.address', they ignore 'mapred.job.tracker'
This breaks backward compatibility and creates issues in Oozie.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4050) Invalid node link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235783#comment-13235783 ] Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4050: --- Verified patch manually, it now displays *N/A* as the value of node link for unassigned task attempts rather than +null+ Invalid node link - Key: MAPREDUCE-4050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4050 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4050.patch, MAPREDUCE-4050.png When a task is in *UNASSIGNED* state, node link is displayed as +null+. But I think it is better to display the link as *N/A* rather than +null+. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated MAPREDUCE-4004: Status: Open (was: Patch Available) DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235790#comment-13235790 ] Mithun Radhakrishnan commented on MAPREDUCE-4004: - Alright, I'm at a loss. I don't see why this patch doesn't apply to trunk/ or branch-0.23.1. (I've verified that it works manually.) DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4043: -- Assignee: Jason Lowe Target Version/s: 0.24.0, 0.23.2, 0.23.3 Status: Patch Available (was: Open) Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4043: -- Attachment: MAPREDUCE-4043.patch I don't believe MAPREDUCE-3727 is relevant as the property being unset in that change needs to be set by the job submitter and that's not happening in this case. I believe the keys are dropped by tasks because the TaskImpl/TaskAttemptImpl constructors take a collection of tokens instead of credentials. Therefore when the map and reduce tasks attempts are constructed by the AM, it ends up dropping the keys out of the credentials since there's only parameters to pass tokens. Here's a patch that changes the collection of tokens argument to credentials, which contain tokens and secret keys. Manual testing with a toy job that sends a secret key works. Needs unit tests. Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4056) Remove MR1 system test
Remove MR1 system test -- Key: MAPREDUCE-4056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4056 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Eli Collins Assignee: Eli Collins Priority: Minor The o.a.h.mapreduce.test package in hadoop-mapreduce-project/src/test/system is MR1 specific (eg built against the JT/TT), is already maintained in branch-1 can be removed from trunk/23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236080#comment-13236080 ] Hadoop QA commented on MAPREDUCE-4020: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519511/MAPREDUCE-4020-branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2084//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2084//console This message is automatically generated. Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4057) Compilation error in RAID
Compilation error in RAID
--
Key: MAPREDUCE-4057
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4057
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/raid
Reporter: Tsz Wo (Nicholas), SZE
{noformat}
[javac] Compiling 33 source files to
/Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/build/contrib/raid/classes
[javac]
/Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java:42:
unreported exception org.apache.hadoop.ipc.StandbyException; must be caught or
declared to be thrown
[javac] return namesystem.getFileInfo(src, resolveLink);
[javac] ^
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Status: Open (was: Patch Available) Forgot to 'svn add' a file Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Status: Patch Available (was: Open) Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Attachment: MAPREDUCE-4020-branch-23.patch Attaching patch with new file and listing in JAXBContext.java Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4057) Compilation error in RAID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236114#comment-13236114
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-4057:
---
HDFS-3089 is going to break RAID since it is changing FSDatasetInterface.
{noformat}
-- ./src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java
444: private final FSDatasetInterface data;
446: private BlockInputStreamFactory(ExtendedBlock block, FSDatasetInt ...
2 occurrence(s)
{noformat}
However, RAID is already broken before HDFS-3089. It looks like that it was
broken by the HA changes.
Compilation error in RAID
--
Key: MAPREDUCE-4057
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4057
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/raid
Reporter: Tsz Wo (Nicholas), SZE
{noformat}
[javac] Compiling 33 source files to
/Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/build/contrib/raid/classes
[javac]
/Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java:42:
unreported exception org.apache.hadoop.ipc.StandbyException; must be caught
or declared to be thrown
[javac] return namesystem.getFileInfo(src, resolveLink);
[javac] ^
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Attachment: MAPREDUCE-4020-branch-23.patch Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236119#comment-13236119 ] Hadoop QA commented on MAPREDUCE-4043: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519508/MAPREDUCE-4043.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2083//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2083//console This message is automatically generated. Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[
https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236129#comment-13236129
]
Thomas Graves commented on MAPREDUCE-4020:
--
can you change the name so that we get: queues : { queue:[] }. That way we
get an array of queue objects it matches the other outputs for apps/jobs/etc..
Also can you please update the documentation -
./hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm and
check the intro one too please.
Web services returns incorrect JSON for deep queue tree
---
Key: MAPREDUCE-4020
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Jason Lowe
Assignee: Anupam Seth
Attachments: MAPREDUCE-4020-branch-23.patch,
MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch
When the capacity scheduler is configured for more than two levels of queues,
the web services API returns incorrect JSON for the subQueues field of some
parent queues. The subQueues field for parent queues should always be an
array, but sometimes the field appears multiple times for a queue and as what
looks like a CapacityQueueInfo object instead of an array. Besides the
sometimes-an-array-sometimes-not problem, parsing the result into a JSON
object causes all but the last subQueues field to be discarded since they
are overwritten by subsequent fields with the same name.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4056) Remove MR1 src/test/system
[ https://issues.apache.org/jira/browse/MAPREDUCE-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-4056: --- Description: hadoop-mapreduce-project/src/test/system is MR1 specific (eg built against the JT/TT), is already maintained in branch-1 can be removed from trunk/23. (was: The o.a.h.mapreduce.test package in hadoop-mapreduce-project/src/test/system is MR1 specific (eg built against the JT/TT), is already maintained in branch-1 can be removed from trunk/23.) Summary: Remove MR1 src/test/system (was: Remove MR1 system test) Remove MR1 src/test/system -- Key: MAPREDUCE-4056 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4056 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Eli Collins Assignee: Eli Collins Priority: Minor hadoop-mapreduce-project/src/test/system is MR1 specific (eg built against the JT/TT), is already maintained in branch-1 can be removed from trunk/23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236170#comment-13236170 ] Hadoop QA commented on MAPREDUCE-4020: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519526/MAPREDUCE-4020-branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2085//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2085//console This message is automatically generated. Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Status: Open (was: Patch Available) Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Status: Patch Available (was: Open) Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4020: --- Attachment: MAPREDUCE-4020-branch-23.patch Thanks for the comments! I have addressed them in the updated patch. Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236176#comment-13236176 ] Hadoop QA commented on MAPREDUCE-4020: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519526/MAPREDUCE-4020-branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2086//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2086//console This message is automatically generated. Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4020) Web services returns incorrect JSON for deep queue tree
[ https://issues.apache.org/jira/browse/MAPREDUCE-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236211#comment-13236211 ] Hadoop QA commented on MAPREDUCE-4020: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519541/MAPREDUCE-4020-branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2087//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2087//console This message is automatically generated. Web services returns incorrect JSON for deep queue tree --- Key: MAPREDUCE-4020 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4020 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.1 Reporter: Jason Lowe Assignee: Anupam Seth Attachments: MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, MAPREDUCE-4020-branch-23.patch, testcase.patch When the capacity scheduler is configured for more than two levels of queues, the web services API returns incorrect JSON for the subQueues field of some parent queues. The subQueues field for parent queues should always be an array, but sometimes the field appears multiple times for a queue and as what looks like a CapacityQueueInfo object instead of an array. Besides the sometimes-an-array-sometimes-not problem, parsing the result into a JSON object causes all but the last subQueues field to be discarded since they are overwritten by subsequent fields with the same name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4058) adjustable task priority
adjustable task priority Key: MAPREDUCE-4058 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4058 Project: Hadoop Map/Reduce Issue Type: New Feature Components: task-controller Affects Versions: 1.0.0 Reporter: Allen Wittenauer For those of us that completely destroy our CPUs, it is beneficial to be able to run user tasks at a different priority than the tasktracker. This would allow for TTs (and by extension, DNs) to get more CPU clock cycles so that things like heartbeats don't disappear. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4058) adjustable task priority
[ https://issues.apache.org/jira/browse/MAPREDUCE-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-4058: Attachment: MAPREDUCE-4058-branch-1.0.patch This patch adds a variable to the task-controller.cfg called 'task.niceness.adjustment' that acts similarly to renice. Note that setpriority() is called while still root, allowing for the tasks to be set to a HIGHER priority than the TT, if someone so desires to. adjustable task priority Key: MAPREDUCE-4058 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4058 Project: Hadoop Map/Reduce Issue Type: New Feature Components: task-controller Affects Versions: 1.0.0 Reporter: Allen Wittenauer Attachments: MAPREDUCE-4058-branch-1.0.patch For those of us that completely destroy our CPUs, it is beneficial to be able to run user tasks at a different priority than the tasktracker. This would allow for TTs (and by extension, DNs) to get more CPU clock cycles so that things like heartbeats don't disappear. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
