Re: Hadoop CI with alternate architectures.
On Wed, May 18, 2016 at 08:29AM, Allen Wittenauer wrote: > That’s really a question for infrastructure-...@apache.org . They > manage the ASF build infrastructure which Apache Hadoop and lots of > other projects utilize. (Bigtop uses something custom, which I think > is funded by Cloudera.) Nope, Bigtop CI is not sponsored by Cloudera. Besides, Bigtop is running itw own CI because we are producing a lot of binary artifacts and don't won't to run a risk of clogging ASF-wide CI. Cos -- Take care, Konstantin (Cos) Boudnik - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-2230) Fault-injection tests are executed multiple times if invoked with run-test-hdfs-fault-inject target
[
https://issues.apache.org/jira/browse/MAPREDUCE-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik resolved MAPREDUCE-2230.
---
Resolution: Won't Fix
Release Note: AOP tests were removed from the src tree long time ago -
closing this.
> Fault-injection tests are executed multiple times if invoked with
> run-test-hdfs-fault-inject target
> ---
>
> Key: MAPREDUCE-2230
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2230
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.20.3
> Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
>
> When invoked with {{run-test-hdfs-fault-inject target}} fault injection tests
> are getting executed 4 times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)
I am clearly +1 (non-binding) on the release. With 8 +1s (5 binding), no -1s or 0s the vote passes. Thanks to all who verified the bits, I'll push them out shortly. Thanks, Cos On Thu, Aug 15, 2013 at 10:29PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1 The maven artifacts are available via repository.apache.org. The only difference between rc0 and rc1 is ASL added to releasenotes.html and updated release dates in CHANGES.txt files. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos signature.asc Description: Digital signature
Release Apache Hadoop 2.0.6-alpha
All, I have pushed the bits of 2.0.6-alpha into the open and released staging bits in Nexus. Site is updated with the release news. 2.0.6-alpha is officially live now. Thanks everybody for your help! Cos On Thu, Aug 22, 2013 at 11:09PM, Konstantin Boudnik wrote: I am clearly +1 (non-binding) on the release. With 8 +1s (5 binding), no -1s or 0s the vote passes. Thanks to all who verified the bits, I'll push them out shortly. Thanks, Cos On Thu, Aug 15, 2013 at 10:29PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1 The maven artifacts are available via repository.apache.org. The only difference between rc0 and rc1 is ASL added to releasenotes.html and updated release dates in CHANGES.txt files. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha
Darn CHANGES - apparently they just hate me. I agree that needs to be addressed. I will spin-up rc1 tonight and restart the vote. Thanks, Cos On Wed, Aug 14, 2013 at 10:40AM, Alejandro Abdelnur wrote: OK: * verified MD5 * verified signature * expanded source tar and did a build * configured pseudo cluster and run a couple of example MR jobs * did a few HTTP calls to HTTFS NOT OK: * CHANGES.txt files have 2.0.6 as UNRELEASED, they should have the date the RC vote ends * 'mvn apache-rat:check' fails, releasenotes HTML files don't have license headers, I think we need to address the NO OK points (specially the last one), they are trivial. Thanks. On Sat, Aug 10, 2013 at 5:46 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha
Alejandro, looking into the source code: it seems that release notes never had license boilerplate in it, hence 2.0.6-alpha doesn't have as well. I have fixed CHANGES with new optimistic date of the release and upload rc1 right now. Please let me know if you feel like we need start doing the license for the releasenotes in this release. Thanks, Cos On Wed, Aug 14, 2013 at 10:40AM, Alejandro Abdelnur wrote: OK: * verified MD5 * verified signature * expanded source tar and did a build * configured pseudo cluster and run a couple of example MR jobs * did a few HTTP calls to HTTFS NOT OK: * CHANGES.txt files have 2.0.6 as UNRELEASED, they should have the date the RC vote ends * 'mvn apache-rat:check' fails, releasenotes HTML files don't have license headers, I think we need to address the NO OK points (specially the last one), they are trivial. Thanks. On Sat, Aug 10, 2013 at 5:46 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos -- Alejandro signature.asc Description: Digital signature
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha
Sure Alejandro, not a big deal - I agree. Uploaded RC1 and restartig the vote right away. Thanks for catching this! Cos On Thu, Aug 15, 2013 at 09:08PM, Alejandro Abdelnur wrote: it should be straight forward adding the license headers to the release notes. please make sure apache-rat:check passes on the RC before publishing it. Arun, as you are about to cut the new RC for 2.1.0-beta, can you please make sure the license headers are used in the releasenotes HTML files? Thx On Thu, Aug 15, 2013 at 8:02 PM, Konstantin Boudnik c...@apache.org wrote: Alejandro, looking into the source code: it seems that release notes never had license boilerplate in it, hence 2.0.6-alpha doesn't have as well. I have fixed CHANGES with new optimistic date of the release and upload rc1 right now. Please let me know if you feel like we need start doing the license for the releasenotes in this release. Thanks, Cos On Wed, Aug 14, 2013 at 10:40AM, Alejandro Abdelnur wrote: OK: * verified MD5 * verified signature * expanded source tar and did a build * configured pseudo cluster and run a couple of example MR jobs * did a few HTTP calls to HTTFS NOT OK: * CHANGES.txt files have 2.0.6 as UNRELEASED, they should have the date the RC vote ends * 'mvn apache-rat:check' fails, releasenotes HTML files don't have license headers, I think we need to address the NO OK points (specially the last one), they are trivial. Thanks. On Sat, Aug 10, 2013 at 5:46 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos -- Alejandro -- Alejandro signature.asc Description: Digital signature
[VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)
All, I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1 The maven artifacts are available via repository.apache.org. The only difference between rc0 and rc1 is ASL added to releasenotes.html and updated release dates in CHANGES.txt files. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos signature.asc Description: Digital signature
[VOTE] Release Apache Hadoop 2.0.6-alpha
All, I have created a release candidate (rc0) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos signature.asc Description: Digital signature
[VOTE RESULT] Release Apache Hadoop 2.0.5-alpha (rc2)
I am clearly +1 (non-binding) on the release. With 8 +1s (3 binding), no -1s or 0s the vote passes. Thanks to all who verified the bits, I'll push them out shortly. Thanks, Cos On Mon, Jun 03, 2013 at 12:51PM, Konstantin Boudnik wrote: I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha. The difference between rc1 and rc2 is the optimistic release date is set for 06/06/2013 in the CHANGES.txt files. The binary artifact is the same - there's no need to rebuild it. The maven artifacts are the same. The difference between the two RCs: svn diff \ https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1/ \ https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2/ New RC builds are uploaded to the web. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2 I would like to extend the vote for another three days for it is such a minor change that doesn't affect anything but the recorded release date. Please cast your vote before 06/06/2013 5pm PDT. Thanks for your patience! Cos On Fri, May 31, 2013 at 09:27PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)
Good point Thomas - I have just fixed the name of the file. Nice catch! Sigh... these three text files... If anyone feels strongly about having them in place - I will update the tarballs and recalculate the checksums. Or just leave it as is, perhaps? So, either way is fine with me. Thanks, Cos On Tue, Jun 04, 2013 at 08:47PM, Thomas Graves wrote: Cos, The name of the mds file for the binary tar is missing a . in your apache directory: hadoop-2.0.5-alphatar.gz.mds http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/hadoop-2.0.5-alphatar .gz.mds Contents of it are fine though so you might just add the .. Also the NOTICE.txt, README.txt, and LICENSE.txt files are missing from the top level directory in both the src and binary tar ball. Everything else looks good. Verified checksums, signatures, built, ran single node cluster. Thanks, Tom On 6/3/13 2:51 PM, Konstantin Boudnik c...@apache.org wrote: I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha. The difference between rc1 and rc2 is the optimistic release date is set for 06/06/2013 in the CHANGES.txt files. The binary artifact is the same - there's no need to rebuild it. The maven artifacts are the same. The difference between the two RCs: svn diff \ https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc 1/ \ https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc 2/ New RC builds are uploaded to the web. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2 I would like to extend the vote for another three days for it is such a minor change that doesn't affect anything but the recorded release date. Please cast your vote before 06/06/2013 5pm PDT. Thanks for your patience! Cos On Fri, May 31, 2013 at 09:27PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc 1 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha
Ok, if this the consensus on the issue, then tonight I will cut rc2 with only CHANGES.txt release dates update. After that I will extend the vote once again. Sorry for the hassle to all who did voted before. Cos On Mon, Jun 03, 2013 at 12:18AM, Doug Cutting wrote: On Sun, Jun 2, 2013 at 9:01 AM, Alejandro Abdelnur t...@cloudera.com wrote: [...] an easy way to solve it it would be using as release date the date the vote ends. +1 This is what I've done for release candidate builds. Doug
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha
Alejandro, I believe this is chicken and egg problem: I can't put release date into unreleased tarball. Looking into 2.0.4-alpha source tarball I see the same situation: Release 2.0.4-alpha - UNRELEASED But in the trunk and branch-2 the release data is in place. I don't think this is an issue. But I would be happy to fix it if this seems to be a problem. Thanks, Cos On Sat, Jun 01, 2013 at 08:04PM, Alejandro Abdelnur wrote: On RC1, verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. CHANGES.txt files contents are correct now. Still, a minor NIT, they have 2.0.5 as UNRELEASED, shouldn't they have a date (I would assume the date the vote ends). Thanks On Fri, May 31, 2013 at 9:39 PM, J. Rottinghuis jrottingh...@gmail.comwrote: Thanks for fixing Cos. http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ looks good to me. +1 (non-binding) Thanks, Joep On Fri, May 31, 2013 at 8:25 PM, Konstantin Boudnik c...@apache.org wrote: Ok, WRT HDFS-4646 - it is all legit and the code is in branch-2.0.4-alpha and later. It has been committed as r1465124 The reason it isn't normally visible because of the weird commit message: svn merge -c1465121 from trunk So, we good. I am done with the CHANGES.txt fixed that you guys have noted earlier and will be re-spinning RC1 in a few. Cos On Fri, May 31, 2013 at 08:07PM, Konstantin Boudnik wrote: Alejandro, thanks for looking into this. Indeed - I missed the 2.0.5-alpha section in YARN CHANGES.txt. Added now. As for HDFS-4646: apparently I didn't get into to branch-2.0.4-alpha back then, although I distinctively remember doing this. Let me pull it into 2.0.5-alpha and update CHANGES.txt to reflect it. Also, I will do JIRA in a moment. Joep, appreciate the thorough examination. I have fixed the dates for the releases 2.0.4-alpha. As for the top-level readme file - sorry I wasn't aware about them. As for the binary: I am pretty sure we are only releasing source code, but I will put binaries into the rc1 respin. I will respin rc1 shortly. Appreciate the feedback! Cos On Fri, May 31, 2013 at 05:27PM, Alejandro Abdelnur wrote: Verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. Still, something seems odd. The HDFS CHANGES.txt has the following entry under 2.0.5-alpha: HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout value (Jagane Sundar via cos) but I don't see that in the branch. And, the YARN CHANGES.txt does not have the 2.0.5-alpha section (it should be there empty). Cos, can you please look at these 2 things and explain/fix? Thanks. On Fri, May 31, 2013 at 4:04 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc0 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos -- Alejandro -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha
This is my first Hadoop release, so I am all ears ;) I looked at how Arun was cutting the previous 2.0.x and followed the example. Thanks for your input and help Alejandro! Regards, Cos On Sun, Jun 02, 2013 at 09:01AM, Alejandro Abdelnur wrote: I know we had this issue before. But and easy way to solve it it would be using as release date the date the vote ends. Anyway, not a big deal if we are not cutting a new rc for other reason +1 on rc1 Thx On Jun 2, 2013, at 12:04 AM, Konstantin Boudnik c...@apache.org wrote: Alejandro, I believe this is chicken and egg problem: I can't put release date into unreleased tarball. Looking into 2.0.4-alpha source tarball I see the same situation: Release 2.0.4-alpha - UNRELEASED But in the trunk and branch-2 the release data is in place. I don't think this is an issue. But I would be happy to fix it if this seems to be a problem. Thanks, Cos On Sat, Jun 01, 2013 at 08:04PM, Alejandro Abdelnur wrote: On RC1, verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. CHANGES.txt files contents are correct now. Still, a minor NIT, they have 2.0.5 as UNRELEASED, shouldn't they have a date (I would assume the date the vote ends). Thanks On Fri, May 31, 2013 at 9:39 PM, J. Rottinghuis jrottingh...@gmail.comwrote: Thanks for fixing Cos. http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ looks good to me. +1 (non-binding) Thanks, Joep On Fri, May 31, 2013 at 8:25 PM, Konstantin Boudnik c...@apache.org wrote: Ok, WRT HDFS-4646 - it is all legit and the code is in branch-2.0.4-alpha and later. It has been committed as r1465124 The reason it isn't normally visible because of the weird commit message: svn merge -c1465121 from trunk So, we good. I am done with the CHANGES.txt fixed that you guys have noted earlier and will be re-spinning RC1 in a few. Cos On Fri, May 31, 2013 at 08:07PM, Konstantin Boudnik wrote: Alejandro, thanks for looking into this. Indeed - I missed the 2.0.5-alpha section in YARN CHANGES.txt. Added now. As for HDFS-4646: apparently I didn't get into to branch-2.0.4-alpha back then, although I distinctively remember doing this. Let me pull it into 2.0.5-alpha and update CHANGES.txt to reflect it. Also, I will do JIRA in a moment. Joep, appreciate the thorough examination. I have fixed the dates for the releases 2.0.4-alpha. As for the top-level readme file - sorry I wasn't aware about them. As for the binary: I am pretty sure we are only releasing source code, but I will put binaries into the rc1 respin. I will respin rc1 shortly. Appreciate the feedback! Cos On Fri, May 31, 2013 at 05:27PM, Alejandro Abdelnur wrote: Verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. Still, something seems odd. The HDFS CHANGES.txt has the following entry under 2.0.5-alpha: HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout value (Jagane Sundar via cos) but I don't see that in the branch. And, the YARN CHANGES.txt does not have the 2.0.5-alpha section (it should be there empty). Cos, can you please look at these 2 things and explain/fix? Thanks. On Fri, May 31, 2013 at 4:04 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc0 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos -- Alejandro -- Alejandro signature.asc Description: Digital signature
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha
On Sun, Jun 02, 2013 at 03:04PM, Arun C Murthy wrote: I record the 'release date' as the one on which the build is created. Not sure what Matt actually does on branch-1. Matt? Arun, but you do this not in the release source tarball, right? Just in the rest of integration branches, ie branch-2 and trunk. Am I correct? Thanks, Cos hth, Arun On Jun 2, 2013, at 12:33 PM, Konstantin Boudnik wrote: This is my first Hadoop release, so I am all ears ;) I looked at how Arun was cutting the previous 2.0.x and followed the example. Thanks for your input and help Alejandro! Regards, Cos On Sun, Jun 02, 2013 at 09:01AM, Alejandro Abdelnur wrote: I know we had this issue before. But and easy way to solve it it would be using as release date the date the vote ends. Anyway, not a big deal if we are not cutting a new rc for other reason +1 on rc1 Thx On Jun 2, 2013, at 12:04 AM, Konstantin Boudnik c...@apache.org wrote: Alejandro, I believe this is chicken and egg problem: I can't put release date into unreleased tarball. Looking into 2.0.4-alpha source tarball I see the same situation: Release 2.0.4-alpha - UNRELEASED But in the trunk and branch-2 the release data is in place. I don't think this is an issue. But I would be happy to fix it if this seems to be a problem. Thanks, Cos On Sat, Jun 01, 2013 at 08:04PM, Alejandro Abdelnur wrote: On RC1, verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. CHANGES.txt files contents are correct now. Still, a minor NIT, they have 2.0.5 as UNRELEASED, shouldn't they have a date (I would assume the date the vote ends). Thanks On Fri, May 31, 2013 at 9:39 PM, J. Rottinghuis jrottingh...@gmail.comwrote: Thanks for fixing Cos. http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ looks good to me. +1 (non-binding) Thanks, Joep On Fri, May 31, 2013 at 8:25 PM, Konstantin Boudnik c...@apache.org wrote: Ok, WRT HDFS-4646 - it is all legit and the code is in branch-2.0.4-alpha and later. It has been committed as r1465124 The reason it isn't normally visible because of the weird commit message: svn merge -c1465121 from trunk So, we good. I am done with the CHANGES.txt fixed that you guys have noted earlier and will be re-spinning RC1 in a few. Cos On Fri, May 31, 2013 at 08:07PM, Konstantin Boudnik wrote: Alejandro, thanks for looking into this. Indeed - I missed the 2.0.5-alpha section in YARN CHANGES.txt. Added now. As for HDFS-4646: apparently I didn't get into to branch-2.0.4-alpha back then, although I distinctively remember doing this. Let me pull it into 2.0.5-alpha and update CHANGES.txt to reflect it. Also, I will do JIRA in a moment. Joep, appreciate the thorough examination. I have fixed the dates for the releases 2.0.4-alpha. As for the top-level readme file - sorry I wasn't aware about them. As for the binary: I am pretty sure we are only releasing source code, but I will put binaries into the rc1 respin. I will respin rc1 shortly. Appreciate the feedback! Cos On Fri, May 31, 2013 at 05:27PM, Alejandro Abdelnur wrote: Verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. Still, something seems odd. The HDFS CHANGES.txt has the following entry under 2.0.5-alpha: HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout value (Jagane Sundar via cos) but I don't see that in the branch. And, the YARN CHANGES.txt does not have the 2.0.5-alpha section (it should be there empty). Cos, can you please look at these 2 things and explain/fix? Thanks. On Fri, May 31, 2013 at 4:04 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc0 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos -- Alejandro -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
Chris, with 2.0.5-alpha (rc1) out, would you please take a look at the release bits? I assume the four-digit numbering scheme issue has been resolved now. Regards, Cos On Thu, May 30, 2013 at 06:18PM, Chris Douglas wrote: On Thu, May 30, 2013 at 5:51 PM, Konstantin Boudnik c...@apache.org wrote: I have no issues of changing the version to 2.0.5-alpha and restarting to vote for the release content, e.g. 2 bug fixes. Shall I call 3 days re-vote because of the number change? +1 Sounds great. Does the result of bylaw vote nullifies the unfinished vote started by Arun? Sorry, I am dense, apparently. Yes, nobody should feel bound by either vote. The bylaw change clarifies that release plans are for RMs to solicit feedback and gauge PMC support for an artifact, not pre-approvals for doing work. Can we limit the vote thread to the merits of the release then? Happily. That sound like adding an insult to injury, if my forth-language skills do not mislead me. They do mislead you, or I've expressed the point imprecisely. We can take this offline. -C On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote: On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Why not include MAPREDUCE-4211 as well rather than create one release per patch? From Cos's description, it sounded like these were backports of fixes to help Sqoop2 and fix some build issues. If it's not just to fixup leftover bugs in 2.0.4 *once* so downstream projects can integrate against 2.0.4.1, and this a release series, then I've completely misunderstood the purpose. Cos, are you planning 2.0.4.2? Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? Good point. Since it contains only backports from branch-2, it would make sense for it to be an intermediate release. I shouldn't have to say this, but I'm changing my vote to -1 while we work this out. -C On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos signature.asc Description: Digital signature
Re: Heads up: moving from 2.0.4.1-alpha to 2.0.5-alpha
I changes are completed, I have updated branch-2 and trunk CHANGES.txt files as planned. Version of the branch-2 is set to 2.1.0-beta; 2.0.5-alpha release artifacts are deployed to the staging area. Thanks for your patience, guys! Cos On Fri, May 31, 2013 at 12:45PM, Konstantin Boudnik wrote: Guys, I will be performing some changes wrt to moving 2.0.4.1 release candidate to 2.0.5 space. As outline below by Alejandro: 1. I will create new 2.0.5-alpha branch from the current head of 2.0.4-alpha that contains 2.0.4.1 changes 2. consequently, set the artifacts version on the new branch to be 2.0.5-alpha 3. the CHANGES.txt will be updated accordingly on the new 2.0.5 branch 4. At this point I can cut an RC and put it out for re-vote. The staging can be done after the next two steps. I will be doing all these modifications in the next hour or so. Tomorrow at 1 pm PDT I would like to: 1. update the version of the artifacts on branch-2 to become 2.1.0-SNAPSHOT 2. update the CHANGES.txt in the trunk and branch-2 to reflect new version names 3. at this point it should safe to do the staging for 2.0.5-alpha RC To avoid any collisions during the last two steps - especially 2. - I would ask everyone to hold off the modifications of the CHANGES.txt files on trunk and branch-2 between 1 pm and 2 pm PDT. Please let me know if you see any flaw above, questions. Cos As we change from 2.0.4.1 to 2.0.5 you'll need to do the following housekeeping as you work the new RC. * rename the svn branch * update the versions in the POMs * update the CHANGES.txt in trunk, branch-2 and the release branch * change the current 2.0.5 version in JIRA to 2.1.0, create a new 2.0.5 version, change the fix version of the 2 JIRAs that make the RC I renamed 2.0.5-beta to 2.1.0-beta and 2.0.4.1-alpha to 2.0.5-alpha versions in jira for HADOOP, HDFS, YARN MAPREDUCE. Please take care of the rest. Also, in branch-2, the version should be 2.1.0-SNAPSHOT. signature.asc Description: Digital signature
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
Thanks for the headstart Arun! I will be sending a separate email to the *-dev@ lists outlining the plan and the schedule of the changes, so we won't step on each other feet, like Vinod expressed earlier. Cos On Fri, May 31, 2013 at 11:20AM, Arun C Murthy wrote: On May 30, 2013, at 8:41 PM, Alejandro Abdelnur wrote: Konstantin, Cos, As we change from 2.0.4.1 to 2.0.5 you'll need to do the following housekeeping as you work the new RC. * rename the svn branch * update the versions in the POMs * update the CHANGES.txt in trunk, branch-2 and the release branch * change the current 2.0.5 version in JIRA to 2.1.0, create a new 2.0.5 version, change the fix version of the 2 JIRAs that make the RC I renamed 2.0.5-beta to 2.1.0-beta and 2.0.4.1-alpha to 2.0.5-alpha versions in jira for HADOOP, HDFS, YARN MAPREDUCE. Please take care of the rest. Also, in branch-2, the version should be 2.1.0-SNAPSHOT. thanks, Arun Thanks. On Thu, May 30, 2013 at 6:18 PM, Chris Douglas cdoug...@apache.org wrote: On Thu, May 30, 2013 at 5:51 PM, Konstantin Boudnik c...@apache.org wrote: I have no issues of changing the version to 2.0.5-alpha and restarting to vote for the release content, e.g. 2 bug fixes. Shall I call 3 days re-vote because of the number change? +1 Sounds great. Does the result of bylaw vote nullifies the unfinished vote started by Arun? Sorry, I am dense, apparently. Yes, nobody should feel bound by either vote. The bylaw change clarifies that release plans are for RMs to solicit feedback and gauge PMC support for an artifact, not pre-approvals for doing work. Can we limit the vote thread to the merits of the release then? Happily. That sound like adding an insult to injury, if my forth-language skills do not mislead me. They do mislead you, or I've expressed the point imprecisely. We can take this offline. -C On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote: On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Why not include MAPREDUCE-4211 as well rather than create one release per patch? From Cos's description, it sounded like these were backports of fixes to help Sqoop2 and fix some build issues. If it's not just to fixup leftover bugs in 2.0.4 *once* so downstream projects can integrate against 2.0.4.1, and this a release series, then I've completely misunderstood the purpose. Cos, are you planning 2.0.4.2? Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? Good point. Since it contains only backports from branch-2, it would make sense for it to be an intermediate release. I shouldn't have to say this, but I'm changing my vote to -1 while we work this out. -C On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Heads up: moving from 2.0.4.1-alpha to 2.0.5-alpha
Guys, I will be performing some changes wrt to moving 2.0.4.1 release candidate to 2.0.5 space. As outline below by Alejandro: 1. I will create new 2.0.5-alpha branch from the current head of 2.0.4-alpha that contains 2.0.4.1 changes 2. consequently, set the artifacts version on the new branch to be 2.0.5-alpha 3. the CHANGES.txt will be updated accordingly on the new 2.0.5 branch 4. At this point I can cut an RC and put it out for re-vote. The staging can be done after the next two steps. I will be doing all these modifications in the next hour or so. Tomorrow at 1 pm PDT I would like to: 1. update the version of the artifacts on branch-2 to become 2.1.0-SNAPSHOT 2. update the CHANGES.txt in the trunk and branch-2 to reflect new version names 3. at this point it should safe to do the staging for 2.0.5-alpha RC To avoid any collisions during the last two steps - especially 2. - I would ask everyone to hold off the modifications of the CHANGES.txt files on trunk and branch-2 between 1 pm and 2 pm PDT. Please let me know if you see any flaw above, questions. Cos As we change from 2.0.4.1 to 2.0.5 you'll need to do the following housekeeping as you work the new RC. * rename the svn branch * update the versions in the POMs * update the CHANGES.txt in trunk, branch-2 and the release branch * change the current 2.0.5 version in JIRA to 2.1.0, create a new 2.0.5 version, change the fix version of the 2 JIRAs that make the RC I renamed 2.0.5-beta to 2.1.0-beta and 2.0.4.1-alpha to 2.0.5-alpha versions in jira for HADOOP, HDFS, YARN MAPREDUCE. Please take care of the rest. Also, in branch-2, the version should be 2.1.0-SNAPSHOT.
Re: Heads up: moving from 2.0.4.1-alpha to 2.0.5-alpha
Indeed! The second part is happening on Saturday, JUN01 2013 1PM-2PM PST. On Fri, May 31, 2013 at 01:01PM, Alejandro Abdelnur wrote: Cos, just to be clear, this is happening SAT JUN01 1PM-2PM PST, not now (FRI MAY31 1PM PST). Correct? Thx On Fri, May 31, 2013 at 12:45 PM, Konstantin Boudnik c...@apache.org wrote: Guys, I will be performing some changes wrt to moving 2.0.4.1 release candidate to 2.0.5 space. As outline below by Alejandro: 1. I will create new 2.0.5-alpha branch from the current head of 2.0.4-alpha that contains 2.0.4.1 changes 2. consequently, set the artifacts version on the new branch to be 2.0.5-alpha 3. the CHANGES.txt will be updated accordingly on the new 2.0.5 branch 4. At this point I can cut an RC and put it out for re-vote. The staging can be done after the next two steps. I will be doing all these modifications in the next hour or so. Tomorrow at 1 pm PDT I would like to: 1. update the version of the artifacts on branch-2 to become 2.1.0-SNAPSHOT 2. update the CHANGES.txt in the trunk and branch-2 to reflect new version names 3. at this point it should safe to do the staging for 2.0.5-alpha RC To avoid any collisions during the last two steps - especially 2. - I would ask everyone to hold off the modifications of the CHANGES.txt files on trunk and branch-2 between 1 pm and 2 pm PDT. Please let me know if you see any flaw above, questions. Cos As we change from 2.0.4.1 to 2.0.5 you'll need to do the following housekeeping as you work the new RC. * rename the svn branch * update the versions in the POMs * update the CHANGES.txt in trunk, branch-2 and the release branch * change the current 2.0.5 version in JIRA to 2.1.0, create a new 2.0.5 version, change the fix version of the 2 JIRAs that make the RC I renamed 2.0.5-beta to 2.1.0-beta and 2.0.4.1-alpha to 2.0.5-alpha versions in jira for HADOOP, HDFS, YARN MAPREDUCE. Please take care of the rest. Also, in branch-2, the version should be 2.1.0-SNAPSHOT. -- Alejandro
[VOTE] Release Apache Hadoop 2.0.5-alpha
All, I have created a release candidate (rc0) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc0 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha
Alejandro, thanks for looking into this. Indeed - I missed the 2.0.5-alpha section in YARN CHANGES.txt. Added now. As for HDFS-4646: apparently I didn't get into to branch-2.0.4-alpha back then, although I distinctively remember doing this. Let me pull it into 2.0.5-alpha and update CHANGES.txt to reflect it. Also, I will do JIRA in a moment. Joep, appreciate the thorough examination. I have fixed the dates for the releases 2.0.4-alpha. As for the top-level readme file - sorry I wasn't aware about them. As for the binary: I am pretty sure we are only releasing source code, but I will put binaries into the rc1 respin. I will respin rc1 shortly. Appreciate the feedback! Cos On Fri, May 31, 2013 at 05:27PM, Alejandro Abdelnur wrote: Verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. Still, something seems odd. The HDFS CHANGES.txt has the following entry under 2.0.5-alpha: HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout value (Jagane Sundar via cos) but I don't see that in the branch. And, the YARN CHANGES.txt does not have the 2.0.5-alpha section (it should be there empty). Cos, can you please look at these 2 things and explain/fix? Thanks. On Fri, May 31, 2013 at 4:04 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc0 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos -- Alejandro signature.asc Description: Digital signature
Re: [VOTE] Release Apache Hadoop 2.0.5-alpha
Ok, WRT HDFS-4646 - it is all legit and the code is in branch-2.0.4-alpha and later. It has been committed as r1465124 The reason it isn't normally visible because of the weird commit message: svn merge -c1465121 from trunk So, we good. I am done with the CHANGES.txt fixed that you guys have noted earlier and will be re-spinning RC1 in a few. Cos On Fri, May 31, 2013 at 08:07PM, Konstantin Boudnik wrote: Alejandro, thanks for looking into this. Indeed - I missed the 2.0.5-alpha section in YARN CHANGES.txt. Added now. As for HDFS-4646: apparently I didn't get into to branch-2.0.4-alpha back then, although I distinctively remember doing this. Let me pull it into 2.0.5-alpha and update CHANGES.txt to reflect it. Also, I will do JIRA in a moment. Joep, appreciate the thorough examination. I have fixed the dates for the releases 2.0.4-alpha. As for the top-level readme file - sorry I wasn't aware about them. As for the binary: I am pretty sure we are only releasing source code, but I will put binaries into the rc1 respin. I will respin rc1 shortly. Appreciate the feedback! Cos On Fri, May 31, 2013 at 05:27PM, Alejandro Abdelnur wrote: Verified MD5 signature, built, configured pseudo cluster, run a couple of sample jobs, tested HTTPFS. Still, something seems odd. The HDFS CHANGES.txt has the following entry under 2.0.5-alpha: HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout value (Jagane Sundar via cos) but I don't see that in the branch. And, the YARN CHANGES.txt does not have the 2.0.5-alpha section (it should be there empty). Cos, can you please look at these 2 things and explain/fix? Thanks. On Fri, May 31, 2013 at 4:04 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc0) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc0 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos -- Alejandro signature.asc Description: Digital signature
[VOTE] Release Apache Hadoop 2.0.5-alpha (rc1)
All, I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1 The maven artifacts will be available via repository.apache.org on Sat, June 1st, 2013 at 2 pm PDT as outlined here http://s.apache.org/WKD Please try the release bits and vote; the vote will run for the 3 days, because this is just a version name change. The bits are identical to the ones voted on before in http://s.apache.org/2041move Thanks for your voting Cos
[jira] [Reopened] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik reopened MAPREDUCE-5211: --- This needs to go into the next stabilization release of 2.0.4.2 as well. Adding the label to that effect and re-opening the ticket. Reducer intermediate files can collide during merge --- Key: MAPREDUCE-5211 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Labels: 2.0.4.2 Fix For: 0.23.8 Attachments: MAPREDUCE-5211.branch-0.23.patch The OnDiskMerger.merge method constructs an output path that is not unique to a reduce attempt, and as a result can result in a file collision with other reducers from the same app that are running on the same node. In addition the name of the output file is based on MapOutput.toString which may not be unique in light of multi-pass merges on disk since the mapId will be null and the basename ends up as MapOutput(null, DISK) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
On Thu, May 30, 2013 at 10:57AM, Arun C Murthy wrote: I see you just re-opened MAPREUDCE-5211. Why not include MAPREDUCE-5211 as well rather than create one release per patch? Arun, it is unclear if MAPREDUCE-5211 has implications in 2.0.4 as per https://issues.apache.org/jira/browse/MAPREDUCE-5211?focusedCommentId=13670574page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670574 Hence, there's a good chance that it never will be backported. And I don't have any plans to created 'a release per patch'. Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? There were precedents in four-numbered schemes before: 0.20.20[3-5].0 comes to mind. As for 2.0.5-alpha: The release numbering games and votes that had happened in the last few weeks are very confusing. Some of them never been concluded, the branches are moved and artifact versions seem to be colliding. 2.0.4.x seems to work well for the stabilization purposes and it will allow to unblock downstream and integration projects quickly. Cos Arun On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
There's no misunderstanding Chris - this release is to unblock downstream. As for your question: I don't have a crystal ball; I wish though. I think the answer depends on will be there more blocking bugs found in the later releases of Bigtop or other downstream components. This is bugfix release and, I guess, if there are more bugs found in the future - more releases would have to be cut. Isn't this is why the software is being released? Now, the -1: I am not clear about the justification. What exactly we expect to work out? Cos On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote: On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Why not include MAPREDUCE-4211 as well rather than create one release per patch? From Cos's description, it sounded like these were backports of fixes to help Sqoop2 and fix some build issues. If it's not just to fixup leftover bugs in 2.0.4 *once* so downstream projects can integrate against 2.0.4.1, and this a release series, then I've completely misunderstood the purpose. Cos, are you planning 2.0.4.2? Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? Good point. Since it contains only backports from branch-2, it would make sense for it to be an intermediate release. I shouldn't have to say this, but I'm changing my vote to -1 while we work this out. -C On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
On Thu, May 30, 2013 at 03:18PM, Chris Douglas wrote: On Thu, May 30, 2013 at 2:39 PM, Konstantin Boudnik c...@apache.org wrote: There's no misunderstanding Chris - this release is to unblock downstream. As for your question: I don't have a crystal ball; I wish though. I think the answer depends on will be there more blocking bugs found in the later releases of Bigtop or other downstream components. This is bugfix release and, I guess, if there are more bugs found in the future - more releases would have to be cut. Isn't this is why the software is being released? Sure, but they're all backports from the release currently marked for 2.0.5. Either (a) these are really blocker bugs and we should roll a patch release or (b) some bleeding-edge work needs to work around this while branch-2 is released in the next few weeks. If it's not severe enough to justify disrupting the versioning of snapshot maven artifacts in branch-2, then we're clearly not in case (a). I thought this was the result of extensive testing, and 2.0.4.1 was a release to enable additional integration before 2.0.5. If we plan to roll more releases as a subset of the bug fixes committed to branch-2 then just call it 2.0.5. Please make sure it- and any future, intermediate release- is worth the disruption. There's no plans to release anything else at this point - this is a bug-fix release, as I pointed out on a numerous occasions. There's no new features - just 2 fixes. 2.0.5 matter became and still is too controversial at some point. The vote started by Arun to override the results of the Konstantin's vote never been closed. The downstream projects are handing in the middle of the air because of that confusion. Now, the -1: I am not clear about the justification. What exactly we expect to work out? It's become fashionable to close threads and count votes in the middle of the discussion. I changed my vote instead of trusting you. -C Have I missed something or you just called me a cheater and a lair right to my face? Cos On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote: On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Why not include MAPREDUCE-4211 as well rather than create one release per patch? From Cos's description, it sounded like these were backports of fixes to help Sqoop2 and fix some build issues. If it's not just to fixup leftover bugs in 2.0.4 *once* so downstream projects can integrate against 2.0.4.1, and this a release series, then I've completely misunderstood the purpose. Cos, are you planning 2.0.4.2? Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? Good point. Since it contains only backports from branch-2, it would make sense for it to be an intermediate release. I shouldn't have to say this, but I'm changing my vote to -1 while we work this out. -C On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
On Thu, May 30, 2013 at 03:11PM, Matt Foley wrote: Hi Cos, I would also request that you renumber the release candidate to just three-numbers, hence 2.0.5-alpha. Arun, are you willing to start the 2.1.x name-space for your next release, so that 2.0.x-alpha can become an intermediate stabilization branch as Cos and Konst want? Let's get the facts straight, Matt, please: this want has been expressed in the official vote here http://s.apache.org/ZMf Apparently, 2.0.5-alpha is blocked now because of the another vote that hasn't been closed yet for whatever reason. In order to unblock a number of releases in downstream component I have moved forward with this release. Do you have any material objections to the release that pursue this goal? I just think that using four-number schemes was symptomatic of the near-forking we had back in the 0.20.xxx.y days, and I really don't want to go back there. Especially since you could say that 0.20.xxx.y is just three significant numbers, the leading zero being inconsequential. I dare to remind that forth part of the version is reserved - not in a parallel universe, of course - for patch level aka bug fixes. It hardly can be taken a sign of 'forking' by any definition. Cos So, would you please consider using 2.0.5-alpha? As for the 2.0.5-SNAPSHOT in the branch-2 versioning, that's standard usage. Whoever makes the 2.0.5 release (or any next release) is expected to update the parent branch's SNAPSHOT default versioning, per HowToReleasePostMavenization#Branchinghttps://wiki.apache.org/hadoop/HowToReleasePostMavenization#Branching, step 6. Thanks, --Matt On Thu, May 30, 2013 at 11:52 AM, Konstantin Boudnik c...@apache.org wrote: On Thu, May 30, 2013 at 10:57AM, Arun C Murthy wrote: I see you just re-opened MAPREUDCE-5211. Why not include MAPREDUCE-5211 as well rather than create one release per patch? Arun, it is unclear if MAPREDUCE-5211 has implications in 2.0.4 as per https://issues.apache.org/jira/browse/MAPREDUCE-5211?focusedCommentId=13670574page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670574 Hence, there's a good chance that it never will be backported. And I don't have any plans to created 'a release per patch'. Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? There were precedents in four-numbered schemes before: 0.20.20[3-5].0 comes to mind. As for 2.0.5-alpha: The release numbering games and votes that had happened in the last few weeks are very confusing. Some of them never been concluded, the branches are moved and artifact versions seem to be colliding. 2.0.4.x seems to work well for the stabilization purposes and it will allow to unblock downstream and integration projects quickly. Cos Arun On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
On Thu, May 30, 2013 at 04:08PM, Jean-Daniel Cryans wrote: FWIW, not that I have a dog in this fight, but the only release with a 4th number (not including .0 like the 0.20.20x releases did) we had was: http://hadoop.6.n7.nabble.com/VOTE-Release-0-17-2-1-rc-0-td13398.html 0.17.2 was missing some native libs so 0.17.2.1 was released to fix that critical issue instead of calling it .3 Exactly the point - the _bigfix_ release. Thanks for pointing out the similarities. Cos J-D On Thu, May 30, 2013 at 3:52 PM, Konstantin Boudnik c...@apache.org wrote: On Thu, May 30, 2013 at 03:11PM, Matt Foley wrote: Hi Cos, I would also request that you renumber the release candidate to just three-numbers, hence 2.0.5-alpha. Arun, are you willing to start the 2.1.x name-space for your next release, so that 2.0.x-alpha can become an intermediate stabilization branch as Cos and Konst want? Let's get the facts straight, Matt, please: this want has been expressed in the official vote here http://s.apache.org/ZMf Apparently, 2.0.5-alpha is blocked now because of the another vote that hasn't been closed yet for whatever reason. In order to unblock a number of releases in downstream component I have moved forward with this release. Do you have any material objections to the release that pursue this goal? I just think that using four-number schemes was symptomatic of the near-forking we had back in the 0.20.xxx.y days, and I really don't want to go back there. Especially since you could say that 0.20.xxx.y is just three significant numbers, the leading zero being inconsequential. I dare to remind that forth part of the version is reserved - not in a parallel universe, of course - for patch level aka bug fixes. It hardly can be taken a sign of 'forking' by any definition. Cos So, would you please consider using 2.0.5-alpha? As for the 2.0.5-SNAPSHOT in the branch-2 versioning, that's standard usage. Whoever makes the 2.0.5 release (or any next release) is expected to update the parent branch's SNAPSHOT default versioning, per HowToReleasePostMavenization#Branchinghttps://wiki.apache.org/hadoop/HowToReleasePostMavenization#Branching, step 6. Thanks, --Matt On Thu, May 30, 2013 at 11:52 AM, Konstantin Boudnik c...@apache.org wrote: On Thu, May 30, 2013 at 10:57AM, Arun C Murthy wrote: I see you just re-opened MAPREUDCE-5211. Why not include MAPREDUCE-5211 as well rather than create one release per patch? Arun, it is unclear if MAPREDUCE-5211 has implications in 2.0.4 as per https://issues.apache.org/jira/browse/MAPREDUCE-5211?focusedCommentId=13670574page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670574 Hence, there's a good chance that it never will be backported. And I don't have any plans to created 'a release per patch'. Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? There were precedents in four-numbered schemes before: 0.20.20[3-5].0 comes to mind. As for 2.0.5-alpha: The release numbering games and votes that had happened in the last few weeks are very confusing. Some of them never been concluded, the branches are moved and artifact versions seem to be colliding. 2.0.4.x seems to work well for the stabilization purposes and it will allow to unblock downstream and integration projects quickly. Cos Arun On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
On Thu, May 30, 2013 at 05:30PM, Chris Douglas wrote: On Thu, May 30, 2013 at 3:25 PM, Konstantin Boudnik c...@apache.org wrote: There's no plans to release anything else at this point - this is a bug-fix release, as I pointed out on a numerous occasions. There's no new features - just 2 fixes. If you're worried about extending the voting by a week, I don't think anyone can reasonably object to a truncated schedule if the only change is the version number. Downstream fixes discovered in Bigtop are a sufficient reason for a patch release and I think we'd all like them to become routine. Not just in 2.0.x, but in later release branches. I have no issues of changing the version to 2.0.5-alpha and restarting to vote for the release content, e.g. 2 bug fixes. Shall I call 3 days re-vote because of the number change? 2.0.5 matter became and still is too controversial at some point. The vote started by Arun to override the results of the Konstantin's vote never been closed. For the nth time, neither release plan vote was binding. We recently amended the bylaws to eliminate this confusion. There is no ambiguity between votes when neither is in scope. Does the result of bylaw vote nullifies the unfinished vote started by Arun? Sorry, I am dense, apparently. The downstream projects are handing in the middle of the air because of that confusion. Can we please ground our discussion when discussing compatibility and bugs? Breathless alarm is not proportional to the severity, here. Good point. Can we limit the vote thread to the merits of the release then? Have I missed something or you just called me a cheater and a lair right to my face? I called you neither. The prenominate votes were closed, counted, and declared to be binding over objections. I'm annoyed that I have to toggle my vote to prevent that tactic, but based on recent experience I don't expect you to forgo it. I'd be happy to learn my caution is unnecessary. -C That sound like adding an insult to injury, if my forth-language skills do not mislead me. Cos On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote: On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Why not include MAPREDUCE-4211 as well rather than create one release per patch? From Cos's description, it sounded like these were backports of fixes to help Sqoop2 and fix some build issues. If it's not just to fixup leftover bugs in 2.0.4 *once* so downstream projects can integrate against 2.0.4.1, and this a release series, then I've completely misunderstood the purpose. Cos, are you planning 2.0.4.2? Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? Good point. Since it contains only backports from branch-2, it would make sense for it to be an intermediate release. I shouldn't have to say this, but I'm changing my vote to -1 while we work this out. -C On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha
Thanks Alejandro, that's what my plan for the morning. Thanks for putting together the check-list - would be easier for me not to miss anything. I am aiming to have the bits out by noon or so. Appreciate the help! Cos On Thu, May 30, 2013 at 08:41PM, Alejandro Abdelnur wrote: Konstantin, Cos, As we change from 2.0.4.1 to 2.0.5 you'll need to do the following housekeeping as you work the new RC. * rename the svn branch * update the versions in the POMs * update the CHANGES.txt in trunk, branch-2 and the release branch * change the current 2.0.5 version in JIRA to 2.1.0, create a new 2.0.5 version, change the fix version of the 2 JIRAs that make the RC Thanks. On Thu, May 30, 2013 at 6:18 PM, Chris Douglas cdoug...@apache.org wrote: On Thu, May 30, 2013 at 5:51 PM, Konstantin Boudnik c...@apache.org wrote: I have no issues of changing the version to 2.0.5-alpha and restarting to vote for the release content, e.g. 2 bug fixes. Shall I call 3 days re-vote because of the number change? +1 Sounds great. Does the result of bylaw vote nullifies the unfinished vote started by Arun? Sorry, I am dense, apparently. Yes, nobody should feel bound by either vote. The bylaw change clarifies that release plans are for RMs to solicit feedback and gauge PMC support for an artifact, not pre-approvals for doing work. Can we limit the vote thread to the merits of the release then? Happily. That sound like adding an insult to injury, if my forth-language skills do not mislead me. They do mislead you, or I've expressed the point imprecisely. We can take this offline. -C On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote: On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote: Why not include MAPREDUCE-4211 as well rather than create one release per patch? From Cos's description, it sounded like these were backports of fixes to help Sqoop2 and fix some build issues. If it's not just to fixup leftover bugs in 2.0.4 *once* so downstream projects can integrate against 2.0.4.1, and this a release series, then I've completely misunderstood the purpose. Cos, are you planning 2.0.4.2? Also, this is the first time we are seeing a four-numbered scheme in Hadoop. Why not call this 2.0.5-alpha? Good point. Since it contains only backports from branch-2, it would make sense for it to be an intermediate release. I shouldn't have to say this, but I'm changing my vote to -1 while we work this out. -C On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos -- Alejandro signature.asc Description: Digital signature
[VOTE] Release Apache Hadoop 2.0.4.1-alpha
All, I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues discovered in the testing with BigTop 0.6.0 release candidate. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos signature.asc Description: Digital signature
[jira] [Reopened] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik reopened MAPREDUCE-5240:
---
I will be backporting this to 2.0.4.1 release
inside of FileOutputCommitter the initialized Credentials cache appears to be
empty
---
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
MAPREDUCE-5240-20130513.txt
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text(mykey), mysecret.getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Versions - Confusion
I have put together this chart some time ago and am trying to keep it up to date (more or less). The most latest update is available here: http://is.gd/axRlgJ Sometimes picture worth a lot... you know ;) Cos On Fri, Apr 26, 2013 at 11:09AM, Suresh S wrote: Hello, I was confused with Hadoop versioning. I found that, some people working on version starting with 0. Some others, working on version starting with 2. Also, i was confused with branch. Which version is really current version. *Regards* *S.Suresh,* *Research Scholar,* *Department of Computer Applications,* *National Institute of Technology,* *Tiruchirappalli - 620015.* *+91-9941506562* signature.asc Description: Digital signature
Re: MAPREDUCE-5069
On Mon, Apr 22, 2013 at 10:51AM, Sangjin Lee wrote: Hi, I just wanted to ping the committers again for your review of the patch for MAPREDUCE-5069. I would greatly appreciate it if you could take a look at the patch, and let me know if you have any feedback or accept the patch. On a related note, do I need to generate a separate patch for branch-2 for it to get accepted onto branch-2? Or is the trunk patch sufficient? Thanks in advance! If trunk patch cleanly applies on branch-2 then you don't need a separate patch. Otherwise, please generate a separate one. Cos
Re: [VOTE] Release Apache Hadoop 2.0.4-alpha
-0 the release is missing HADOOP-9704 that has critical effect on downstream projects e.g. build are affected. The issue has been raised for the first time back in 4/10/13 http://is.gd/OGb3GG and never been even sneezed upon. Cos On Sat, Apr 13, 2013 at 03:26AM, Arun C Murthy wrote: Folks, I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would like to release. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ signature.asc Description: Digital signature
[jira] [Resolved] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik resolved MAPREDUCE-5088.
---
Resolution: Fixed
The patch is merged to all corresponding branches now.
MR Client gets an renewer token exception while Oozie is submitting a job
-
Key: MAPREDUCE-5088
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Roman Shaposhnik
Assignee: Daryn Sharp
Priority: Blocker
Fix For: 3.0.0, 2.0.5-beta, 2.0.4-alpha
Attachments: HADOOP-9409.patch, HADOOP-9409.patch,
MAPREDUCE-5088.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.txt
After the fix for HADOOP-9299 I'm now getting the following bizzare exception
in Oozie while trying to submit a job. This also seems to be KRB related:
{noformat}
2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
TOKEN[] APP[MapReduce] JOB[001-130315123130987-oozie-oozi-W]
ACTION[001-130315123130987-oozie-oozi-W@Sleep] Error starting action
[Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
Message [UninitializedMessageException: Message missing required fields:
renewer]
org.apache.oozie.action.ActionExecutorException:
UninitializedMessageException: Message missing required fields: renewer
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.google.protobuf.UninitializedMessageException: Message missing
required fields: renewer
at
com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
at
org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
at
org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
at
org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
at
org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
... 10 more
2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
TOKEN[] APP[MapReduce] JOB[001-13031512313
{noformat}
--
This message is automatically
[jira] [Created] (MAPREDUCE-5101) hive smoke tests can
Konstantin Boudnik created MAPREDUCE-5101: - Summary: hive smoke tests can Key: MAPREDUCE-5101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5101 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Konstantin Boudnik -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Vote] Merge branch-trunk-win to trunk
Yes, you are right of course - the mis-merged commit is the cause. Thanks for pointing this out! I think it would be beneficial if we had branch-2 on going build in the Jenkins. I will go ahead and create one tonight. Cos On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote: Adding other mailing lists I missed earlier. Cos, There is progress being made on that ticket. Also it has nothing to do with that. Please follow the discussion here and why this happened due to an invalid commit that was reverted - https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650 Regards, Suresh On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik c...@apache.org wrote: It doesn't look like any progress has been done on the ticket below in the last 3 weeks. And now branch-2 can't be compiled because of hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15] WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from outside package That's exactly why I was -1'ing this... Cos On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote: Thanks, gentlemen. I've opened and taken responsibility for https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has agreed to help with the parts that require Jenkins admin access. Thanks, --Matt On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko shv.had...@gmail.comwrote: +1 on the merge. I am glad we agreed. Having Jira to track the CI effort is a good idea. Thanks, --Konstantin On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.com wrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com wrote: Hi Konstantin, I'd like to point out two things: First, I already committed in this thread (email of Thu, Feb 28, 2013 at 6:01 PM) to providing CI for Windows builds. So please stop acting like I'm resisting this idea or something. Second, you didn't answer my question, you just kvetched about the phrasing. So I ask again: Will providing full test-patch integration
Re: [Vote] Merge branch-trunk-win to trunk
Here's a daily build for hadoop-2 branch https://builds.apache.org/job/Hadoop-branch2 It just builds the full Hadoop project without running any tests (for now). Can be easily extended to do test runs/artifact deployment, if needed. Cos On Mon, Mar 25, 2013 at 07:14PM, Konstantin Boudnik wrote: Yes, you are right of course - the mis-merged commit is the cause. Thanks for pointing this out! I think it would be beneficial if we had branch-2 on going build in the Jenkins. I will go ahead and create one tonight. Cos On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote: Adding other mailing lists I missed earlier. Cos, There is progress being made on that ticket. Also it has nothing to do with that. Please follow the discussion here and why this happened due to an invalid commit that was reverted - https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650 Regards, Suresh On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik c...@apache.org wrote: It doesn't look like any progress has been done on the ticket below in the last 3 weeks. And now branch-2 can't be compiled because of hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15] WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from outside package That's exactly why I was -1'ing this... Cos On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote: Thanks, gentlemen. I've opened and taken responsibility for https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has agreed to help with the parts that require Jenkins admin access. Thanks, --Matt On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko shv.had...@gmail.comwrote: +1 on the merge. I am glad we agreed. Having Jira to track the CI effort is a good idea. Thanks, --Konstantin On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley mfo...@hortonworks.com wrote: Thanks. I agree Windows -1's in test-patch should not block commits. --Matt On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko shv.had...@gmail.com wrote: On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley mfo...@hortonworks.com wrote: Konstantine, you have voted -1, and stated some requirements before you'll withdraw that -1. As I plan to do work to fulfill those requirements, I want to make sure that what I'm proposing will, in fact, satisfy you. That's why I'm asking, if we implement full test-patch integration for Windows, does it seem to you that that would provide adequate support? Yes. I have learned not to presume that my interpretation is correct. My interpretation of item #1 is that test-patch provides pre-commit build, so it would satisfy item #1. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. I agree it will satisfy my item #1. I did not agree in my previous email, but I changed my mind based on the latest discussion. I have to explain why now. I was proposing nightly build because I did not want pre-commit build for Windows block commits to Linux. But if people are fine just ignoring -1s for the Windows part of the build it should be good. Regarding item #2, it is also my interpretation that test-patch provides an on-demand (perhaps 20-minutes deferred) Jenkins build and unit test, with logs available to the developer, so it would satisfy item #2. But rather than assuming that I am interpreting it correctly, I simply want your agreement that it would, or if not, clarification why it won't. It will satisfy my item #2 in the following way: I can duplicate your pre-commit build for Windows and add an input parameter, which would let people run the build on their patches chosen from local machine rather than attaching them to Jiras. Thanks, --Konstantin In agile terms, you are the Owner of these requirements. Please give me owner feedback as to whether my proposed work sounds like it will satisfy the requirements. Thank you, --Matt On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Didn't I explain in details what I am asking for? Thanks, --Konst On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley mfo...@hortonworks.com
Re: [Vote] Merge branch-trunk-win to trunk
Ok, looks like we are converging on this across a few hundred emails ;) So, as has been stated elsewhere: test-patch will be improved to fully support Windows; furthermore -1 from Windows' test-patch won't block Linux commits. This is ok with me. Can we have a JIRA ticket for that test-patch work assigned to the real owner, so it can be tracked? I am +1 in this case. Cos On Fri, Mar 01, 2013 at 04:57PM, Chris Douglas wrote: On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko shv.had...@gmail.com wrote: Commitment is a good thing. I think the two builds that I proposed are a prerequisite for Win support. If we commit windows patch people will start breaking it the next day. Which we wont know without the nightly build and wont be able to fix without the on-demand one. As several people have pointed out already, the surface of possible conflicts is relatively limited, and- as you did in 2007- the devs on Windows will report and fix bugs in that platform as they find them. CI is important for detecting and preventing bugs, but this isn't software we're launching into orbit. Making two builds is less than 2 days work, imho, given that there is a Windows node available and that mvn targets are in place. Correct me if I missed any complications in the process. On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik c...@apache.org wrote: It seems that with the HW in place, the matter of setting at least nightly build is trivial for anyone with up to date Windows knowledge. I wish I could help. Going without a validation is a recipe for a disaster IMO. Fair enough, though that also implies that the window for regressions is small, and it leaves little room to doubt that this will receive priority. Until it's merged, spurious notifications that the current trunk breaks Windows are an awkward introduction to devs' workflow. The order of merge/CI is a choice between mild annoyances, really. But it might be moot. Giri: you're doing the work on this. When do you think it can be complete? -C
Re: [Vote] Merge branch-trunk-win to trunk
It seems that with the HW in place, the matter of setting at least nightly build is trivial for anyone with up to date Windows knowledge. I wish I could help. Going without a validation is a recipe for a disaster IMO. -1 until some reasonable solution is implemented. Cos On Fri, Mar 01, 2013 at 01:57PM, Konstantin Shvachko wrote: Commitment is a good thing. I think the two builds that I proposed are a prerequisite for Win support. If we commit windows patch people will start breaking it the next day. Which we wont know without the nightly build and wont be able to fix without the on-demand one. Making two builds is less than 2 days work, imho, given that there is a Windows node available and that mvn targets are in place. Correct me if I missed any complications in the process. Thanks, --Konst On Fri, Mar 1, 2013 at 1:28 PM, Chris Douglas cdoug...@apache.org wrote: Konstantin- There's no debate on the necessity of CI and related infrastructure to support the platform well. Suresh outlined the support to effect this here: http://s.apache.org/s1 Is the commitment to establish this infrastructure after the merge sufficient? -C On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko shv.had...@gmail.com wrote: -1 We should have a CI infrastructure in place before we can commit to supporting Windows platform. Eric is right Win/Cygwin was supported since day one. I had a Windows box under my desk running nightly builds back in 2006-07. People were irritated but I was filing windows bugs until 0.22 release. Times changing and I am glad to see wider support for Win platform. But in order to make it work you guys need to put the CI process in place 1. windows jenkins build: could be nightly or PreCommit. - Nightly would mean that changes can be committed to trunk based on linux PreCommit build. And people will file bugs if the change broke Windows nightly build. - PreCommit-win build will mean automatic reporting failed tests to respective jira blocking commits the same way as it is now with linux PreCommit builds. We should discuss which way is more efficient for developers. 2. On-demand-windows Jenkins build. I see it as a build to which I can attach my patch and the build will run my changes on a dedicated windows box. That way people can test their changes without having personal windows nodes. I think this is the minimal set of requirement for us to be able to commit to the new platform. Right now I see only one windows related build https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/ Which was failing since Sept 8, 2012 and did not run in the last month. Thanks, --Konst On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler eri...@hortonworks.com wrote: +1 (non-binding) A few of observations: - Windows has actually been a supported platform for Hadoop since 0.1 . Doug championed supporting windows then and we've continued to do it with varying vigor over time. To my knowledge we've never made a decision to drop windows support. The change here is improving our support and dropping the requirement of cigwin. We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception. - A little pragmatism will go a long way. As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ... We should all plan to let new features optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data. - A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs. Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO. It is mostly abstracted from the OS APIs via Java and our design choices. Where it is not we can continue to add plugable abstractions.
Re: [Vote] Merge branch-trunk-win to trunk
Suresh, I appreciate all the troubles you're going through wrt syncing up the huge patch for a long time - I really do. I am not asking to have full test-patch process in place. But I think it is a real good idea to have a way to run the full test suite once in a while - or as Konstantin proposing - for a given patch, to make sure that codebase doesn't bitrot at the edges. Official support has nothing to do with the issue - you're trying to build a straw man argument around this. What is relevant, on the other hand, is that Windows is so different from _any_ Unix or pseudo-Unix flavors, including Windows with Cygwin - that even multi-platform Java has hard hard time dealing with it. This is enough, IMO, to warrant a separate checkpoint. I hope I have explained myself better. Cos On Fri, Mar 01, 2013 at 05:55PM, Suresh Srinivas wrote: It seems that with the HW in place, the matter of setting at least nightly build is trivial for anyone with up to date Windows knowledge. I wish I could help. Going without a validation is a recipe for a disaster IMO. -1 until some reasonable solution is implemented. Cos Cos, I have hard time understanding your veto? Here is my rationale for merge: Currently with all the cross platform support, the merge patch has +1 from Jenkins on Linux. So no regression has been introduced in Hadoop on Linux. As regards to Windows support I want to make two points: 1. Until Jenkins is setup and we are passing all the tests, I am okay, as Konstntin proposed, if we do not officially declare Windows as supported. I do not want to tie the patch merge with setting up Windows Jenkins. I have been maintaining this branch for a long time and keeping it in sync with trunk is non-trivial. 2. After Jenkins is setup, there is a concern in the community about -1 from Windows hindering patch commits. As others have already suggested in the thread, I am okay committing new patches even if -1 is posted by Jenkins on Windows. The team that worked on Windows will fix the issues. I do not see many such issues cropping up. signature.asc Description: Digital signature
Re: [Vote] Merge branch-trunk-win to trunk
On Thu, Feb 28, 2013 at 03:08PM, sanjay Radia wrote: +1 Java has done the bulk of the work in making Hadoop multi-platform. Windows specific code is a tiny percentage of the code. Jeninks support for windows is going help us keep the platform portable going forward. I expect that the vast majority of new commits have no problems. I propose that we start by fixing problems that Jenkins raises but not block new commits for too long if the author does not have a windows box or if a volunteer does not step up. Considering a typical set of software most of the people here work with it would be completely inappropriate to block commits for failing Windows specific features. After all, Microsoft never did bother to check what features or compatibilty matters they have broke in Java and elsewhere, so why should we? I believe this kind of rules have to be set and discussed before the merge is done. Cheers, Cos signature.asc Description: Digital signature
Re: Jenkins says -1 overall while each items of testing are all green
What was wrong with surefire.timeout in the build system? Why coming up with artificial enforcements like this? Cos On Fri, Feb 22, 2013 at 08:35AM, Surenkumar Nihalani wrote: Hey Tsuyoshi, The check for timeout in @Tests was recently introduced. This was a bug of the first commit. A correctional commit has been made ( See comments on https://issues.apache.org/jira/browse/HADOOP-9112 ). Hudson reported success in yarn and common. Failure in HDFS. So, I am guessing the correction hasn't propagated to map reduce yet since there wasn't a hudson comment. This is a known bug. It should be corrected soon. Thanks, Suren. On Feb 22, 2013, at 7:42 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Hi, I've written a patch and passed each items of testing, but Jenkins says that the patch is -1 overall. It seems to be odd for me. Is that a normal behaviour? https://issues.apache.org/jira/browse/MAPREDUCE-4502?focusedCommentId=13583133page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13583133 Thanks, - Tsuyoshi
Re: [VOTE] Release hadoop-2.0.3-alpha
We've created BigTop stack with 2.0.3 as the base. Ran YCSB, slive, and some other loads on up to 20 nodes clusters as a part of the release validation. Two issues were noted: - due to the known issue with Jetty we are seeing MR jobs hanging here and there - without properly configured capacity-scheduler.xml resource manager throws exception on the startup (BIGTOP-841). +1 Cos On Wed, Feb 06, 2013 at 07:59PM, Arun C Murthy wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ signature.asc Description: Digital signature
Re: [VOTE] Release hadoop-2.0.3-alpha
On Tue, Feb 12, 2013 at 07:44PM, Konstantin Boudnik wrote: We've created BigTop stack with 2.0.3 as the base. Ran YCSB, slive, and some other loads on up to 20 nodes clusters as a part of the release validation. Two issues were noted: - due to the known issue with Jetty we are seeing MR jobs hanging here and there The issue referred here is reported as MAPREDUCE-2980 - without properly configured capacity-scheduler.xml resource manager throws exception on the startup (BIGTOP-841). +1 Cos On Wed, Feb 06, 2013 at 07:59PM, Arun C Murthy wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ signature.asc Description: Digital signature
Re: [VOTE] Release hadoop-2.0.3-alpha
The issue with the configuration is raised (and adressed) in https://issues.apache.org/jira/browse/BIGTOP-841 Cos On Fri, Feb 08, 2013 at 04:25PM, Aaron T. Myers wrote: +1 (binding) I downloaded the src tar ball, built it with the native bits enabled, started up a little cluster, and ran some sample jobs. Things worked as expected. I also verified the signatures on the source artifact. I did bump into one little issue, but I don't think it should be considered a blocker. When I first tried to start up the RM, it failed to start with this error: 13/02/08 16:00:31 FATAL resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:710) And then this on shutdown: 13/02/08 16:00:31 INFO service.CompositeService: Error stopping ResourceManager java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:590) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) Presumably this is because I don't have the CapacityScheduler queues configured at all, and the default scheduler is now the CapacityScheduler. To work around this for my testing, I switched to the FairScheduler and the RM came up just fine. -- Aaron T. Myers Software Engineer, Cloudera On Wed, Feb 6, 2013 at 7:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ signature.asc Description: Digital signature
Re: Fault injection framework for testing
Hadoop-1 includes framework called Herriot that would allow you to develop on-the-cluster FI system tests. However, because of the some timing, it hasn't been hooked into the maven build system Hadoop-2 branches. Basically, I see two way of doing what you need to do here: - wait until the Herriot is integrated back (that might take a while, actually) - go along with MOP using Groovy and develop a cluster test for your feature. MOP won't require pretty much anything but a groovy jar to be added to the classpath of the java process(es) in question. With it in place you can instrument anything you want the way you need during the application bootstrap. In fact, I think Herriot would be better off with that approach instead of initial AspectJ build-time mechanism. Hope it helps, Cos On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote: Hi, I've created patch for MAPREDUCE-4502. Now, I confirmed that it works well for usual case, and I also added code to handle MapTask failure. As a next step, I need to add test code against MapTask failure. So I have questions: Is there fault injection in MapReduce testing framework? If the answer is negative, do you have any ideas to test it? Thanks, OZAWA Tsuyoshi signature.asc Description: Digital signature
Re: Fault injection framework for testing
On Wed, Jan 16, 2013 at 01:18PM, Tsuyoshi OZAWA wrote: Thanks for your comment. Your comment is helpful for me. I'd like to go with 2nd approach - MOP with Groovy. In that case, how can I add test code to the trunk? You go with a additional patch for the test and the test-time dependencies added. Is it acceptable for Hadoop project to add test code written in groovy? Groovy is a Java+ Having Groovy tests won't require any massive build up of the infrastructure - just an extra jar file, that will be visible in the test scope only. While there are might different opinions in the community, as f course, I don't see any real issues with that approach. Cos Thanks, Tsuyoshi On Wed, Jan 16, 2013 at 12:13 PM, Konstantin Boudnik c...@apache.org wrote: Hadoop-1 includes framework called Herriot that would allow you to develop on-the-cluster FI system tests. However, because of the some timing, it hasn't been hooked into the maven build system Hadoop-2 branches. Basically, I see two way of doing what you need to do here: - wait until the Herriot is integrated back (that might take a while, actually) - go along with MOP using Groovy and develop a cluster test for your feature. MOP won't require pretty much anything but a groovy jar to be added to the classpath of the java process(es) in question. With it in place you can instrument anything you want the way you need during the application bootstrap. In fact, I think Herriot would be better off with that approach instead of initial AspectJ build-time mechanism. Hope it helps, Cos On Wed, Jan 16, 2013 at 02:19AM, Tsuyoshi OZAWA wrote: Hi, I've created patch for MAPREDUCE-4502. Now, I confirmed that it works well for usual case, and I also added code to handle MapTask failure. As a next step, I need to add test code against MapTask failure. So I have questions: Is there fault injection in MapReduce testing framework? If the answer is negative, do you have any ideas to test it? Thanks, OZAWA Tsuyoshi -- OZAWA Tsuyoshi
Re: can we remove Ant stuff now that MAPREDUCE-3543 (gridmix) has been mavenized?
Herriot and fault injection work is still due, although it won't affect the build if these files will be removed. Cos On Thu, May 17, 2012 at 09:12AM, Alejandro Abdelnur wrote: First of all, Thomas, thanks for finalizing Gridmix mavenization. AFAIK this was the blocker to get rid of the Ant remnants from MR build. Can we delete all the build.xml and all the sources under hadoop-mapred/src that are in hadoop-tools and/or dead? thx -- Alejandro signature.asc Description: Digital signature
Re: can we remove Ant stuff now that MAPREDUCE-3543 (gridmix) has been mavenized?
Ok, makes sense. On Thu, May 17, 2012 at 09:36AM, Alejandro Abdelnur wrote: My take is that we should delete all the stuff from there and unwire completely ANT from the build. Things can be revived from SVN if necessary at porting time. AFAIK, the current status of the old code is: hadoop-mapreduce/src/java : this is old MR1 stuff, dead code in trunk, DELETE hadoop-mapreduce/src/contrib/data_join : already migrated to hadoop-tools, DELETE hadoop-mapreduce/src/contrib/gridmix : already migrated to hadoop-tools, DELETE hadoop-mapreduce/src/contrib/eclipse-plugin : not required anymore, DELETE hadoop-mapreduce/src/test : this has AOP stuff, migration has not been started hadoop-mapreduce/src/contrib/raid : migration of this to trunk is WIP hadoop-mapreduce/src/contrib/block_forensics : hadoop-mapreduce/src/contrib/vaidya : hadoop-mapreduce/src/contrib/vertica : thx. On Thu, May 17, 2012 at 9:15 AM, Konstantin Boudnik c...@apache.org wrote: Herriot and fault injection work is still due, although it won't affect the build if these files will be removed. Cos On Thu, May 17, 2012 at 09:12AM, Alejandro Abdelnur wrote: First of all, Thomas, thanks for finalizing Gridmix mavenization. AFAIK this was the blocker to get rid of the Ant remnants from MR build. Can we delete all the build.xml and all the sources under hadoop-mapred/src that are in hadoop-tools and/or dead? thx -- Alejandro -- Alejandro signature.asc Description: Digital signature
Re: What's the status of AspectJ files in branch-1?
Hi Ravi. You need to run Herriot build to make sure that everything is ok after your changes. The way to to it is as follows: % ant jar-system jar-test-system this will perform the compilation of Hadoop binaries with weaved Herriot APIs. More information about Herriot can be found here https://wiki.apache.org/hadoop/HowToUseSystemTestFramework Cos On Mon, May 07, 2012 at 04:45PM, Ravi Prakash wrote: Hi folks, I'm patching changes to StatisticsCollector in https://issues.apache.org/jira/browse/MAPREDUCE-4227 . A simple grep shows StatisticsCollector is also referenced in src/test/system/aop/org/apache/hadoop/mapred/StatisticsCollectorAspect.aj src/test/system/aop/org/apache/hadoop/mapred/JobTrackerAspect.aj How can I check whether my changes cause any issues or not in the AspectJ? (tests?) Thanks Ravi.
[jira] [Resolved] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-3863. --- Resolution: Fixed Fix Version/s: 0.22.1 Hadoop Flags: Reviewed I have just committed it. Thank you, Antony! 0.22 branch mvn deploy is not publishing hadoop-streaming JAR - Key: MAPREDUCE-3863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0, 0.22.1 Reporter: Alejandro Abdelnur Priority: Critical Fix For: 0.22.1 Attachments: MAPREDUCE-3863.patch, MAPREDUCE-3863.patch, MAPREDUCE-3863.patch Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2344) Exclude second Ant JAR from classpath in MR builds
[ https://issues.apache.org/jira/browse/MAPREDUCE-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2344. --- Resolution: Won't Fix As in HDFS-798 Exclude second Ant JAR from classpath in MR builds -- Key: MAPREDUCE-2344 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2344 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Minor Attachments: MAPREDUCE-2344.patch Counterpart of HDFS-798 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3429) Few contrib tests are failing because of the missing commons-lang dependency
Few contrib tests are failing because of the missing commons-lang dependency
Key: MAPREDUCE-3429
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3429
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/capacity-sched, contrib/gridmix
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
Assignee: Konstantin Boudnik
Fix For: 0.22.0
As the result of MAPREDUCE-3311 fix a transient {{commons-lang}} isn't
available anymore to contrib tests. This causing silent failure with timeout.
The problem is only seeing if tests are ran with {{-Dtest.output=yes}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3311) Bump jetty to 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-3311. --- Resolution: Fixed Hadoop Flags: Reviewed I have just committed the patch. Bump jetty to 6.1.26 Key: MAPREDUCE-3311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.22.0 Attachments: MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, MAPREDUCE-3311.patch MapReduce part of HADOOP-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3396) Mumak compilation is broken (for a while?)
[
https://issues.apache.org/jira/browse/MAPREDUCE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik resolved MAPREDUCE-3396.
---
Resolution: Duplicate
It really dups MAPREDUCE-3311, so closing this one.
Mumak compilation is broken (for a while?)
--
Key: MAPREDUCE-3396
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3396
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/mumak
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
Assignee: Konstantin Boudnik
Fix For: 0.22.0
Attachments: MAPREDUCE-3396.patch
{{contrib/mumak}} compilation is broken because of the missing
{{commons-lang}} dependency
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3396) Mumak compilation is broken (for a while?)
Mumak compilation is broken (for a while?)
--
Key: MAPREDUCE-3396
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3396
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/mumak
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
Assignee: Konstantin Boudnik
Fix For: 0.22.0
{{contrib/mumak}} compilation is broken because of the missing {{commons-lang}}
dependency
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3311) Bump jetty to 6.1.26
Bump jetty to 6.1.26 Key: MAPREDUCE-3311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311 Project: Hadoop Map/Reduce Issue Type: Task Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.22.0 MapReduce part of HADOOP-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3156) Allow TestMRCLI to be run against a cluster
Allow TestMRCLI to be run against a cluster --- Key: MAPREDUCE-3156 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3156 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Mapreduce part of HDFS-1762 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problem getting the code from SVN
Seems like it has been broken in the recent re-shuffle of the workspace due to the maven changes. This external now lives under hadoop-common, apparently. On Sun, Aug 21, 2011 at 07:20PM, Praveen Sripati wrote: Hi, When I try to get the code from svn, I get the below error. svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/ Atrunk/hadoop-mapreduce/bin/mapred-config.sh Atrunk/hadoop-mapreduce/bin/stop-mapred.sh Atrunk/hadoop-mapreduce/bin/mapred Atrunk/hadoop-mapreduce/bin/start-mapred.sh Atrunk/hadoop-project Atrunk/hadoop-project/pom.xml U trunk Fetching external item into 'trunk/hadoop-hdfs/src/test/bin' svn: warning: OPTIONS of ' https://svn.apache.org/repos/asf/hadoop/common/trunk/common/src/test/bin': Could not resolve hostname `svn.apache.org': No address associated with hostname (https://svn.apache.org) svn: warning: Error handling externals definition for 'trunk/hadoop-mapreduce/src/test/bin': svn: warning: OPTIONS of ' https://svn.apache.org/repos/asf/hadoop/common/trunk/common/src/test/bin': Could not resolve hostname `svn.apache.org': No address associated with hostname (https://svn.apache.org) Checked out revision 1159979. Thanks, Praveen
Re: Mavenizing the HDFS build
I see that the whole AOP stuff has been taken out along with Herriot work. I don't see any discussion about this on the JIRA. Neither I don't see any tickets created to track the effort (but a subtask for FI tests in HADOOP-7412). Any reason we are missing a big chunk of validation infrastructure in the mavenized build? Cos On Fri, Aug 19, 2011 at 10:55AM, Tom White wrote: HDFS-2096 is now committed to trunk. The instructions for building HDFS can be found in the top-level BUILDING.txt file. I added a script to https://issues.apache.org/jira/browse/HADOOP-7500 to help with migrating HDFS patches to the new layout. There are a few follow-up patches that need doing soon (e.g. HADOOP-7498, HADOOP-7496, MAPREDUCE-2856), but these shouldn't stop folks from doing development as usual. Thanks to everyone who helped with this! Cheers, Tom On Thu, Aug 18, 2011 at 11:30 AM, Tom White t...@cloudera.com wrote: Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, ═Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. ═Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
Re: Mavenizing the HDFS build
Right, Common has the tracking for AOP FI tests. There's Herrior work (system cluster tests) which is separate from FI although is using the same technology. My point was that HDFS ticket doesn't have any traces of it which makes me wonder if it has been removed or else. I see your point and I'll add new subtasks to the JIRAs. Cos On Fri, Aug 19, 2011 at 11:26AM, Alejandro Abdelnur wrote: Hey Cos, Given the size of the Mavenization work we broke into several pieces and we are working them incrementally. Because of that some disruption is happening to some parts of the build and components. There is a JIRA open for the AOP stuff ( https://issues.apache.org/jira/browse/HADOOP-7481). There you've indicated you'd take a stab at it. Given that you are familiar with AOP and how it is used in Hadoop, I guess you are the best person to take care of this part of the puzzle. Thanks. Alejandro On Fri, Aug 19, 2011 at 11:09 AM, Konstantin Boudnik c...@apache.org wrote: I see that the whole AOP stuff has been taken out along with Herriot work. I don't see any discussion about this on the JIRA. Neither I don't see any tickets created to track the effort (but a subtask for FI tests in HADOOP-7412). Any reason we are missing a big chunk of validation infrastructure in the mavenized build? Cos On Fri, Aug 19, 2011 at 10:55AM, Tom White wrote: HDFS-2096 is now committed to trunk. The instructions for building HDFS can be found in the top-level BUILDING.txt file. I added a script to https://issues.apache.org/jira/browse/HADOOP-7500 to help with migrating HDFS patches to the new layout. There are a few follow-up patches that need doing soon (e.g. HADOOP-7498, HADOOP-7496, MAPREDUCE-2856), but these shouldn't stop folks from doing development as usual. Thanks to everyone who helped with this! Cheers, Tom On Thu, Aug 18, 2011 at 11:30 AM, Tom White t...@cloudera.com wrote: Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, ═Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. ═Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
[jira] [Resolved] (MAPREDUCE-2752) Build does not pass along properties to contrib builds
[ https://issues.apache.org/jira/browse/MAPREDUCE-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2752. --- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] I have just committed it. Thanks Joep. Build does not pass along properties to contrib builds -- Key: MAPREDUCE-2752 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2752 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Environment: RHEL 6.1 Ubuntu 11.04; Sun JDK 1.6_016 Sun JDK 1.6.0_26; Ant 1.8.2 Reporter: Joep Rottinghuis Priority: Minor Fix For: 0.22.0 Attachments: MAPREDUCE-2752.patch Subant call to compile contribs do not pass along parameters from parent build. Properties such as hadoop-common.version, asfrepo, offline, etc. are not passed along. Result is that build not connected to Internet fails, hdfs proxy refuses to build against own recently built common but rather downloads 0.22-SNAPSHOT from apache again. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2547) TestDFSIO fails on a physical cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik resolved MAPREDUCE-2547.
---
Resolution: Not A Problem
TestDFSIO fails on a physical cluster
-
Key: MAPREDUCE-2547
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2547
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Environment: Physical cluster based on 0.22-SNAPSHOT
Reporter: Konstantin Boudnik
Assignee: Konstantin Boudnik
Priority: Blocker
An attempt to run TestDSFIO on cluster fails because TestDFSIO tries to run
MR job with local runner. If JT is explicitly specified via {{-jt}} cmd. arg.
then everything is working as expected.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: wish to become mapred developer
http://hadoop.apache.org/mapreduce will be a great starting point for you. -- Take care, Konstantin (Cos) Boudnik On Sat, Apr 30, 2011 at 09:14, shashank shekhar shshn...@gmail.com wrote: hii.i want to contribute to mapreduce but i now i am in beginner stage so i have few question for which i need answer. first thing which i want to know is about bugs. Do i need to find bugs myself or the bugs are organised in some kind of list using which i can know about them and fix them. Pls do reply ..thanks
[jira] [Resolved] (MAPREDUCE-2202) Generalize CLITest structure and interfaces to facilitate upstream adoption (e.g. for web or system testing)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2202. --- Resolution: Fixed Patch has been reviewed as a part of parent issue. I have just committed this. Generalize CLITest structure and interfaces to facilitate upstream adoption (e.g. for web or system testing) Key: MAPREDUCE-2202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2202 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Attachments: MAPREDUCE-2202.patch Counterpart of HADOOP-7014 and HDFS-1486 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
That seems like settings.xml has been modified, but then it'd affect other builds as well. Unless we are using different machines/users for different component's builds. On Thu, Mar 3, 2011 at 13:41, Todd Lipcon t...@cloudera.com wrote: Anyone understand why we can't seem to publish to apache maven anymore? -Todd On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server hud...@hudson.apache.org wrote: See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/621/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 4464 lines...] [javac] Compiling 11 source files to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. jar-test-system: -do-jar-test: [jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar [jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-sources.jar set-version: [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy clean-sign: sign: signanddeploy: simpledeploy: [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots [artifact:deploy] [INFO] Retrieving previous build number from apache.snapshots.https [artifact:deploy] Uploading: org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar to apache.snapshots.https [artifact:deploy] Uploaded 1741K [artifact:deploy] An error has occurred while processing the Maven artifact tasks. [artifact:deploy] Diagnosis: [artifact:deploy] [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar. Return code is: 401 BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1528: Error deploying artifact 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar. Return code is: 401 Total time: 3 minutes 20 seconds == == STORE: saving artifacts == == mv: cannot stat `build/*.tar.gz': No such file or directory mv: cannot stat `build/test/findbugs': No such file or directory mv: cannot stat `build/docs/api': No such file or directory Build Failed [FINDBUGS] Skipping publisher since build result is FAILURE Recording fingerprints Archiving artifacts Recording test results Publishing Javadoc Publishing Clover coverage report... No Clover report will be published due to a Build Failure Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran. -- Todd Lipcon Software Engineer, Cloudera
Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Looks like the same symptoms are affecting Common trunk builds https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/516/ Have our build env. has been changed recently? -- Take care, Konstantin (Cos) Boudnik On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik c...@apache.org wrote: That seems like settings.xml has been modified, but then it'd affect other builds as well. Unless we are using different machines/users for different component's builds. On Thu, Mar 3, 2011 at 13:41, Todd Lipcon t...@cloudera.com wrote: Anyone understand why we can't seem to publish to apache maven anymore? -Todd On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server hud...@hudson.apache.org wrote: See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/621/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 4464 lines...] [javac] Compiling 11 source files to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. jar-test-system: -do-jar-test: [jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar [jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-sources.jar set-version: [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy clean-sign: sign: signanddeploy: simpledeploy: [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots [artifact:deploy] [INFO] Retrieving previous build number from apache.snapshots.https [artifact:deploy] Uploading: org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar to apache.snapshots.https [artifact:deploy] Uploaded 1741K [artifact:deploy] An error has occurred while processing the Maven artifact tasks. [artifact:deploy] Diagnosis: [artifact:deploy] [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar. Return code is: 401 BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1528: Error deploying artifact 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar. Return code is: 401 Total time: 3 minutes 20 seconds == == STORE: saving artifacts == == mv: cannot stat `build/*.tar.gz': No such file or directory mv: cannot stat `build/test/findbugs': No such file or directory mv: cannot stat `build/docs/api': No such file or directory Build Failed [FINDBUGS] Skipping publisher since build result is FAILURE Recording fingerprints Archiving artifacts Recording test results Publishing Javadoc Publishing Clover coverage report... No Clover report will be published due to a Build Failure Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran. -- Todd Lipcon Software Engineer, Cloudera
Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Thanks a lot Giri! I guess this is another reason to have CM in place for the build machines. On Fri, Mar 4, 2011 at 16:30, Giridharan Kesavan gkesa...@yahoo-inc.com wrote: It looks like settings.xml file is deleted by someone while cleaning the m2 repository (my guess). I ve copied it back. I'm triggering a build to verify this. -Giri On 3/4/11 4:21 PM, Giridharan Kesavan gkesa...@yahoo-inc.com wrote: I'm lookin into it.. -Giri On 3/4/11 3:04 PM, Konstantin Boudnik c...@apache.org wrote: Looks like the same symptoms are affecting Common trunk builds https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/516/ Have our build env. has been changed recently? -- Take care, Konstantin (Cos) Boudnik On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik c...@apache.org wrote: That seems like settings.xml has been modified, but then it'd affect other builds as well. Unless we are using different machines/users for different component's builds. On Thu, Mar 3, 2011 at 13:41, Todd Lipcon t...@cloudera.com wrote: Anyone understand why we can't seem to publish to apache maven anymore? -Todd On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server hud...@hudson.apache.org wrote: See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/621/ ## # ## LAST 60 LINES OF THE CONSOLE ### [...truncated 4464 lines...] [javac] Compiling 11 source files to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ b uild-fi/system/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. jar-test-system: -do-jar-test: [jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ b uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar [jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ b uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-sources.jar set-version: [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ i vy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ i vy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ i vy [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ i vy clean-sign: sign: signanddeploy: simpledeploy: [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots [artifact:deploy] [INFO] Retrieving previous build number from apache.snapshots.https [artifact:deploy] Uploading: org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110 3 03.213523-60.jar to apache.snapshots.https [artifact:deploy] Uploaded 1741K [artifact:deploy] An error has occurred while processing the Maven artifact tasks. [artifact:deploy] Diagnosis: [artifact:deploy] [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/ha d oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60. j ar. Return code is: 401 BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ b uild.xml:1528: Error deploying artifact 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/ha d oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60. j ar. Return code is: 401 Total time: 3 minutes 20 seconds == == STORE: saving artifacts == == mv: cannot stat `build/*.tar.gz': No such file or directory mv: cannot stat `build/test/findbugs': No such file or directory mv: cannot stat `build/docs/api': No such file or directory Build Failed [FINDBUGS] Skipping publisher since build result is FAILURE Recording fingerprints Archiving artifacts Recording test results Publishing Javadoc Publishing Clover coverage report... No Clover report will be published due to a Build Failure Email was triggered for: Failure Sending email for trigger: Failure
[jira] Resolved: (MAPREDUCE-2228) Remove java5 dependencies from build
[ https://issues.apache.org/jira/browse/MAPREDUCE-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2228. --- Resolution: Fixed Fix Version/s: 0.21.1 Hadoop Flags: [Reviewed] I have committed this to the trunk and branches 0.21, 0.22. Also, svn:externals src/test were re-pointed in 0.21 and 0.22 Remove java5 dependencies from build Key: MAPREDUCE-2228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2228 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.1 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.21.1 Attachments: MAPREDUCE-2228.patch As the first short-term step let's remove JDK5 dependency from build(s) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2200) TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to be conditional
TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to be conditional Key: MAPREDUCE-2200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2200 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.0 Reporter: Konstantin Boudnik TestUmbilicalProtocolWithJobToken requires Krb environment to be set. For testing some 'pseudo' environment is needed (similar to HDFS-1284). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2199) build is broken 0.22 branch creation
[ https://issues.apache.org/jira/browse/MAPREDUCE-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2199. --- Resolution: Fixed Hadoop Flags: [Reviewed] I have just committed it. build is broken 0.22 branch creation Key: MAPREDUCE-2199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2199 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Konstantin Boudnik Attachments: MAPREDUCE-2199.patch hdfs and common dep versions weren't updated properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2134) ant binary-system is broken in mapreduce project.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2134. --- Resolution: Fixed Fix Version/s: 0.21.1 Hadoop Flags: [Reviewed] I have just committed this to 0.21 and trunk. ant binary-system is broken in mapreduce project. - Key: MAPREDUCE-2134 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2134 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.1 Reporter: Vinay Kumar Thota Assignee: Konstantin Boudnik Fix For: 0.21.1 Attachments: MAPREDUCE-2134.patch Build failed due to unable to copy the commons instrumented jar. I could see the following error in the log. binary-system: [copy] Copying 5 files to /home/vinay/mapreduce/build-fi/system/hadoop-mapred-0.22.0-SNAPSHOT BUILD FAILED /home/vinay/mapreduce/build.xml:1307: Warning: Could not find file /home/vinay/mapreduce/build-fi/ivy/lib/Hadoop/system/hadoop-common-instrumented-0.22.0-SNAPSHOT.jar to copy. It's pointing to the wrong path to copy the file. Actually the correct path is, /home/vinay/mapreduce/build-fi/system/ivy/Hadoop/system -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2144) CLONE -ant binary-system is broken
CLONE -ant binary-system is broken -- Key: MAPREDUCE-2144 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2144 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.21.1 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Build failed due to unable to copy the commons instrumented jar. I could see the following error in the log. binary-system: [copy] Copying 5 files to /home/vinay/mapreduce/build-fi/system/hadoop-mapred-0.22.0-SNAPSHOT BUILD FAILED /home/vinay/mapreduce/build.xml:1307: Warning: Could not find file /home/vinay/mapreduce/build-fi/ivy/lib/Hadoop/system/hadoop-common-instrumented-0.22.0-SNAPSHOT.jar to copy. It's pointing to the wrong path to copy the file. Actually the correct path is, /home/vinay/mapreduce/build-fi/system/ivy/Hadoop/system -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2090) Clover build doesn't generate per-test coverage
[ https://issues.apache.org/jira/browse/MAPREDUCE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-2090. --- Release Note: This fix requires that test coverage is running under Clover 3.0+ Fix Version/s: 0.21.1 Resolution: Fixed The fix is exactly the same as for HADOOP-6971. Thus, no separate review seems needed. I have ran the test with Clover on and see that the problem is gone. Clover build doesn't generate per-test coverage --- Key: MAPREDUCE-2090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2090 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.21.1, 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Fix For: 0.21.1 Attachments: MAPREDUCE-2090.patch, MAPREDUCE-2090.patch Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1984) herriot TestCluster fails because exclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-1984. --- Hadoop Flags: [Reviewed] Assignee: Balaji Rajagopalan Fix Version/s: 0.21.1 Resolution: Fixed +1 patch looks good. I have just committed it to the trunk and 0.21 herriot TestCluster fails because exclusion is not there Key: MAPREDUCE-1984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.1 Reporter: Balaji Rajagopalan Assignee: Balaji Rajagopalan Fix For: 0.21.1 Attachments: MAPREDUCE-1984.patch, testcluster.patch restart is part of the test case which causes ioexceptions, and this needs to be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2092) Trunk can't be compiled
Trunk can't be compiled
---
Key: MAPREDUCE-2092
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2092
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Reporter: Konstantin Boudnik
Compilation of the trunk is broken because of an attempt to call
{{ServiceAuthorizationManager.refresh}} from a static content.
0.21 branch seems to be Ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1154) Large-scale, automated test framwork for Map-Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-1154. --- Resolution: Duplicate This has been addressed as HADOOP-6332 and derived work. Large-scale, automated test framwork for Map-Reduce --- Key: MAPREDUCE-1154 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154 Project: Hadoop Map/Reduce Issue Type: New Feature Components: test Reporter: Arun C Murthy Attachments: testing.patch HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop. This jira is meant to track relevant work to Map-Reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1942) 'compile-fault-inject' should never be called directly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-1942. --- Hadoop Flags: [Reviewed] Fix Version/s: 0.21.0 Resolution: Fixed I have committed it. 'compile-fault-inject' should never be called directly. Key: MAPREDUCE-1942 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1942 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Minor Fix For: 0.21.0 Attachments: MAPREDUCE-1942.patch Similar to HDFS-1299: prevent calls to helper targets. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1942) 'compile-fault-inject' should never be called directly.
'compile-fault-inject' should never be called directly. Key: MAPREDUCE-1942 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1942 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.21.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Minor Similar to HDFS-1299: prevent calls to helper targets. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1830) Ivy2.0 has bugs: let's upgrate to 2.1.0
Ivy2.0 has bugs: let's upgrate to 2.1.0 --- Key: MAPREDUCE-1830 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1830 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Affects Versions: 0.21.0 Reporter: Konstantin Boudnik Similar to HDFS-1177 Ivy needs to be upgraded up to 2.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1791) Remote cluster control functionality needs JavaDocs improvement
Remote cluster control functionality needs JavaDocs improvement --- Key: MAPREDUCE-1791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1791 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: test Affects Versions: 0.21.0 Reporter: Konstantin Boudnik This is MR part of HADOOP-6752. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1760) Herriot tests for MapReduce should support submission into a specific queue
Herriot tests for MapReduce should support submission into a specific queue --- Key: MAPREDUCE-1760 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1760 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Hadoop might be configured in a way to allow job submission only into a special queues. Herriot tests have to be able to read a queue name configuration parameter from system-test.xml file and use it for job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1614) TestDFSIO should allow to configure output directory
TestDFSIO should allow to configure output directory Key: MAPREDUCE-1614 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1614 Project: Hadoop Map/Reduce Issue Type: Improvement Components: benchmarks Affects Versions: 0.20.2 Reporter: Konstantin Boudnik TestDFSIO has a hardcoded location for its files to be written and read to or from. This location is /benchmarks. However, it might pose a problem if HDFS '/' doesn't allow anyone to write into it. It'd be convenient to have a command line option to specify an alternative location on demand. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1326) fi tests don't use fi-site.xml
fi tests don't use fi-site.xml --- Key: MAPREDUCE-1326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1326 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: test Affects Versions: 0.22.0 Reporter: Konstantin Boudnik When fault injection framework was ported to the Mapreduce fi-site.xml is missed from the testing process. E.g. when the tests run they won't use FI configuration. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1294) Build fails to pull latest hadoop-core-* artifacts
Build fails to pull latest hadoop-core-* artifacts -- Key: MAPREDUCE-1294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1294 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Priority: Critical This is the same as HDFS-825 for mapreduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1259) Add SureLogic annotations' jar into Ivy and Eclipse configs
Add SureLogic annotations' jar into Ivy and Eclipse configs --- Key: MAPREDUCE-1259 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1259 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik In order to use SureLogic analysis tools and allow their concurrency analysis annotations in HDFS code the annotations library has to be automatically pulled from a Maven repo. Also, it has to be added to Eclipse .classpath template. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1260) Update Eclipse configuration to match changes to Ivy configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-1260. --- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] I've committed this. Thanks Edwin! Update Eclipse configuration to match changes to Ivy configuration -- Key: MAPREDUCE-1260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1260 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Edwin Chan Fix For: 0.22.0 Attachments: mapReduceClasspath.patch The .eclipse_templates/.classpath file doesn't match the Ivy configuration, so I've updated it to match. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1217) TestMapReduceLocal fails intermittently. Assert message is unclear
TestMapReduceLocal fails intermittently. Assert message is unclear
Key: MAPREDUCE-1217
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1217
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
TestMapReduceLocal fails occasionally with the following assert message
{{MultiFileWordCount failed}}
Besides of the failure, the message is unclear and requires an extra analysis
effort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1146) Sqoop dependencies break Ecpilse build on Linux
Sqoop dependencies break Ecpilse build on Linux
---
Key: MAPREDUCE-1146
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1146
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/sqoop
Environment: Linux, Sun JDK6
Reporter: Konstantin Boudnik
Under Linux there's the error in the Eclipse Problems view:
{noformat}
- com.sun.tools cannot be resolved at line 166 of
org.apache.hadoop.sqoop.orm.CompilationManager
{noformat}
The problem doesn't appear on MacOS though
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1133) Eclipse .classpath template has outdated jar files and is missing some new ones.
Eclipse .classpath template has outdated jar files and is missing some new ones. Key: MAPREDUCE-1133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1133 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Eclipse environment is broken in trunk: it still uses *.21*.jar files and includes some libraries which aren't in use any more (similar to HDFS-726). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1084) Implementing aspects development and fault injeciton framework for MapReduce
Implementing aspects development and fault injeciton framework for MapReduce Key: MAPREDUCE-1084 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1084 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, test Reporter: Konstantin Boudnik Similar to HDFS-435 and HADOOP-6204 this JIRA will track the introduction of injection framework for MapReduce. After HADOOP-6204 is in place this particular modification should be very trivial and would take importing (via svn:external) of src/test/build and some tweaking of the build.xml file -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
