[jira] [Assigned] (MAPREDUCE-3540) saveVersion.sh script fails in windows/cygwin (hadoop-yarn-common)
[
https://issues.apache.org/jira/browse/MAPREDUCE-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chackaravarthy reassigned MAPREDUCE-3540:
-
Assignee: (was: chackaravarthy)
saveVersion.sh script fails in windows/cygwin (hadoop-yarn-common)
--
Key: MAPREDUCE-3540
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3540
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Affects Versions: 0.24.0
Reporter: Alejandro Abdelnur
Fix For: 0.24.0
Attachments: MAPREDUCE-3540.patch
{code}
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec
(generate-version) on project hadoop-yarn-common: Comman
d execution failed. Cannot run program scripts\saveVersion.sh (in directory
C:\cygwin\home\tucu\src\hadoop\hadoop-mapreduce-proje
ct\hadoop-yarn\hadoop-yarn-common): CreateProcess error=2, The system cannot
find the file specified - [Help 1]
[ERROR]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237507#comment-13237507 ] Hudson commented on MAPREDUCE-4043: --- Integrated in Hadoop-Hdfs-trunk #994 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/994/]) MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks (Jason Lowe via bobby) (Revision 1304587) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304587 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapTaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/ReduceTaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/MapTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/ReduceTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/krb5.conf Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4043.patch, MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237512#comment-13237512 ] Hudson commented on MAPREDUCE-4043: --- Integrated in Hadoop-Hdfs-0.23-Build #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/207/]) svn merge -c 1304587 from trunk to branch-0.23 FIXES: MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks (Jason Lowe via bobby) (Revision 1304593) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304593 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapTaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/ReduceTaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/MapTaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/ReduceTaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/krb5.conf Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4043.patch, MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237517#comment-13237517 ] Hudson commented on MAPREDUCE-4043: --- Integrated in Hadoop-Mapreduce-0.23-Build #235 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/235/]) svn merge -c 1304587 from trunk to branch-0.23 FIXES: MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks (Jason Lowe via bobby) (Revision 1304593) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304593 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapTaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/ReduceTaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/MapTaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/ReduceTaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/krb5.conf Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4043.patch, MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237534#comment-13237534 ] Hudson commented on MAPREDUCE-4043: --- Integrated in Hadoop-Mapreduce-trunk #1029 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1029/]) MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks (Jason Lowe via bobby) (Revision 1304587) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1304587 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapTaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/ReduceTaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/MapTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/ReduceTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/krb5.conf Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Fix For: 0.23.2 Attachments: MAPREDUCE-4043.patch, MAPREDUCE-4043.patch The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237552#comment-13237552 ] Steve Loughran commented on MAPREDUCE-3992: --- +1 for the patch for branch-1 too. Reduce fetcher doesn't verify HTTP status code of response -- Key: MAPREDUCE-3992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.1, 0.24.0, 1.0.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.24.0, 0.23.3 Attachments: mr-3992-branch-1.txt, mr-3992.txt Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation: - the map output servlet gets an IOException after setting the headers but before the first call to flush() - this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed) - it will still include the response headers indicating it's a valid response In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237557#comment-13237557 ] Alejandro Abdelnur commented on MAPREDUCE-3868: --- Ram, Dhruba, any update on this? Thxs Reenable Raid - Key: MAPREDUCE-3868 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/raid Reporter: Scott Chen Assignee: Ramkumar Vadali Attachments: MAPREDUCE-3868v1.patch, MAPREDUCE-3868v1.sh Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237715#comment-13237715 ] Todd Lipcon commented on MAPREDUCE-3289: Found an issue with this optimization today in the case where some map output partitions are really large (eg on the order of many GBs). Because the fadvise is a blocking call, this can block on the order of minutes before sending the first bytes of response data. This causes the reducer to time out, etc. Instead, we should fadvise only a chunk ahead of the current read position. Make use of fadvise in the NM's shuffle handler --- Key: MAPREDUCE-3289 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, performance Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: mr-3289.txt Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-3992. Resolution: Fixed Fix Version/s: 1.1.0 Target Version/s: 0.24.0, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 0.24.0) Hadoop Flags: Reviewed Committed to branch-1 Reduce fetcher doesn't verify HTTP status code of response -- Key: MAPREDUCE-3992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.1, 0.24.0, 1.0.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.24.0, 1.1.0, 0.23.3 Attachments: mr-3992-branch-1.txt, mr-3992.txt Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation: - the map output servlet gets an IOException after setting the headers but before the first call to flush() - this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed) - it will still include the response headers indicating it's a valid response In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
