[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5156: --- Status: Open (was: Patch Available) MAPREDUCE-5193 is committed, let's see if Jenkins is okay this time. In the while, can you please fix the javadoc like Karthik says. Also, shall we rename driverV2 to something like run() or runDriver()? > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5156.1.patch > > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5193: --- Fix Version/s: 2.0.5-beta > A few MR tests use block sizes which are smaller than the default minimum > block size > > > Key: MAPREDUCE-5193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 2.0.5-beta >Reporter: Aaron T. Myers >Assignee: Andrew Wang > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch > > > HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR > tests deliberately set much smaller block sizes. This JIRA is to update those > tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13648035#comment-13648035 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5205: bq. Did you track down why the original patch works on trunk but not branch-2? I don't understand how it would work in either branch. Not exactly, but like I commented (2nd comment above), it should mostly be because of stale git branch. Don't have conclusive evidence though, except that Omkar also saw exactly what I see. It does fail on trunk now. bq. This suggests that the tokens in the launch context are not actually being added to the UGI? Yes, because in MRAppMaster.initAndStartAppMaster(), we create a new UGI. I thought of changing that too, but wanted to keep this patch minimal. > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5205-20130502.txt > > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5204) Handle YarnRemoteException separately from IOException in MR api
[ https://issues.apache.org/jira/browse/MAPREDUCE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5204: - Attachment: MR-5204.2.patch > Handle YarnRemoteException separately from IOException in MR api > - > > Key: MAPREDUCE-5204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5204 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5204.1.patch, MR-5204.2.patch > > > YarnRemoteException is not rooted as IOException, so in MR api, we need to > handle them separately from IOException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13648013#comment-13648013 ] Omkar Vinit Joshi commented on MAPREDUCE-5205: -- +1 This now lets job execute successfully on single node (unsecured). [~robsparker] Yes I tested this earlier but at that time it passed in trunk (probably earlier patch was not available there when I did git fetch) but failed on branch-2. Today when I tried again it failed for trunk too. The current patch fixed this. > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5205-20130502.txt > > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647999#comment-13647999 ] Daryn Sharp commented on MAPREDUCE-5205: Did you track down why the original patch works on trunk but not branch-2? I don't understand how it would work in either branch. The patch is relying on the token file that is auto-loaded into the login UGI - it's generally intended for passing tokens through the container executor or to new processes. In this case, the AM's remote UGI is devoid of tokens and waiting to be populated. This patch propagates the tokens from the login UGI into this new UGI. This suggests that the tokens in the launch context are not actually being added to the UGI? > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5205-20130502.txt > > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5205: --- Status: Patch Available (was: Open) > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5205-20130502.txt > > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5205: --- Attachment: MAPREDUCE-5205-20130502.txt Here's a simple patch that fixes it. Added a test too. Stole the env-hack from one of the previous patches - MAPREDUCE-3068. Robert, sure that will be great help! I am also pinging [~ojoshi] offline who helped track this down to verify it. > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5205-20130502.txt > > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4366) mapred metrics shows negative count of waiting maps and reduces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4366: -- Status: Patch Available (was: Open) > mapred metrics shows negative count of waiting maps and reduces > --- > > Key: MAPREDUCE-4366 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4366 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 1.0.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4366-branch-1.patch > > > Negative waiting_maps and waiting_reduces count is observed in the mapred > metrics. MAPREDUCE-1238 partially fixed this but it appears there is still > issues as we are seeing it, but not as bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4366) mapred metrics shows negative count of waiting maps and reduces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4366: -- Attachment: MAPREDUCE-4366-branch-1.patch > mapred metrics shows negative count of waiting maps and reduces > --- > > Key: MAPREDUCE-4366 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4366 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 1.0.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4366-branch-1.patch > > > Negative waiting_maps and waiting_reduces count is observed in the mapred > metrics. MAPREDUCE-1238 partially fixed this but it appears there is still > issues as we are seeing it, but not as bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647808#comment-13647808 ] Robert Parker commented on MAPREDUCE-5205: -- Vinod I can integrate the patch into our QE builds to verify on a 20 node secure cluster once it is posted. > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647751#comment-13647751
]
Karthik Kambatla commented on MAPREDUCE-5156:
-
+1. Not particular on it: the comment on the {{void driver}} should probably be
"API compatible with Hadoop 1.x"
> Hadoop-examples-1.x.x.jar cannot run on Yarn
>
>
> Key: MAPREDUCE-5156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Blocker
> Attachments: MAPREDUCE-5156.1.patch
>
>
> M/R examples are run through ProgramDriver.driver. ProgramDriver.driver
> returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the
> function signatures in the example jar and yarn are incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647730#comment-13647730 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5205: I chased this all day yesterday. Even though I could successfully reproduce this on branch-2, it was passing on trunk. After a long day, I cornered it down to one of two things - I had stale installation - Or git-trunk branch didn't get updated with YARN-579. It is more likely the later because I asked [~ojoshi] the same day, and he also saw the same result - passing on trunk and failing on branch-2. Either ways, I have a patch that works, attaching it now. > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-5205) Apps fail in secure cluster setup on branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli moved YARN-626 to MAPREDUCE-5205: - Key: MAPREDUCE-5205 (was: YARN-626) Project: Hadoop Map/Reduce (was: Hadoop YARN) > Apps fail in secure cluster setup on branch-2 > - > > Key: MAPREDUCE-5205 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5205 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > > Found at YARN-579 by [~daryn]. Need to investigate if it was caused by > YARN-579 itself or something else. > Secure setup on trunk passes though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-5036: - Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Incompatible change,Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks Sandy. > Default shuffle handler port should not be 8080 > --- > > Key: MAPREDUCE-5036 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5036.patch > > > The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is > a pretty common port for web services, and is likely to cause unnecessary > port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647543#comment-13647543 ] Hudson commented on MAPREDUCE-5193: --- Integrated in Hadoop-Mapreduce-trunk #1417 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1417/]) MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the default minimum block size. Contributed by Andrew Wang. (Revision 1478250) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478250 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/hdfs-site.xml > A few MR tests use block sizes which are smaller than the default minimum > block size > > > Key: MAPREDUCE-5193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 2.0.5-beta >Reporter: Aaron T. Myers >Assignee: Andrew Wang > Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch > > > HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR > tests deliberately set much smaller block sizes. This JIRA is to update those > tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647488#comment-13647488 ] Hudson commented on MAPREDUCE-5193: --- Integrated in Hadoop-Hdfs-trunk #1390 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1390/]) MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the default minimum block size. Contributed by Andrew Wang. (Revision 1478250) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478250 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/hdfs-site.xml > A few MR tests use block sizes which are smaller than the default minimum > block size > > > Key: MAPREDUCE-5193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 2.0.5-beta >Reporter: Aaron T. Myers >Assignee: Andrew Wang > Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch > > > HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR > tests deliberately set much smaller block sizes. This JIRA is to update those > tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
[ https://issues.apache.org/jira/browse/MAPREDUCE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647424#comment-13647424 ] Hudson commented on MAPREDUCE-5193: --- Integrated in Hadoop-Yarn-trunk #201 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/201/]) MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the default minimum block size. Contributed by Andrew Wang. (Revision 1478250) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1478250 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/hdfs-site.xml > A few MR tests use block sizes which are smaller than the default minimum > block size > > > Key: MAPREDUCE-5193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 2.0.5-beta >Reporter: Aaron T. Myers >Assignee: Andrew Wang > Attachments: MAPREDUCE-5156.1.patch, mapreduce-5193-1.patch > > > HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR > tests deliberately set much smaller block sizes. This JIRA is to update those > tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
