Re: [DISCUSS] branch-1
+1 Arun On May 8, 2015, at 10:41 AM, Allen Wittenauer a...@altiscale.com wrote: May we declare this branch dead and just close bugs (but not necessarily concepts, ideas, etc) with won’t fix? I don’t think anyone has any intention of working on the 1.3 release, especially given that 1.2.1 was Aug 2013 …. I guess we need a PMC member to declare a vote or whatever….
Re: Thinking ahead to hadoop-2.7
Sangjin/Karthik, How about planning on hadoop-2.8 by late Jan? Thoughts? thanks, Arun On Dec 2, 2014, at 11:09 AM, Sangjin Lee sjl...@gmail.com wrote: If 2.7 is being positioned as the JDK7-only release, then it would be good to know how 2.8 lines up in terms of timing. Our interest is landing the shared cache feature (YARN-1492)... Thanks. Sangjin On Mon, Dec 1, 2014 at 2:55 PM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for starting this thread, Arun. Your proposal seems reasonable to me. I suppose we would like new features and improvements to go into 2.8 then? If yes, what time frame are we looking at for 2.8? Looking at YARN, it would be nice to get a release with shared-cache and a stable version of reservation work. I believe they are well under way and should be ready in a few weeks. Regarding 2.7 release specifics, do you plan to create a branch off of current branch-2.6 and update all issues marked fixed for 2.7 to be fixed for 2.8? Thanks Karthik On Mon, Dec 1, 2014 at 2:42 PM, Arun Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.6 out it's time to think ahead. As we've discussed in the past, 2.6 was the last release which supports JDK6. I'm thinking it's best to try get 2.7 out in a few weeks (maybe by the holidays) with just the switch to JDK7 (HADOOP-10530) and possibly support for JDK-1.8 (as a runtime) via HADOOP-11090. This way we can start with the stable base of 2.6 and switch over to JDK7 to allow our downstream projects to use either for a short time (hadoop-2.6 or hadoop-2.7). I'll update the Roadmap wiki accordingly. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.6.0
With 24 +1s (10 binding) and no -1s the vote passes. Thanks for everyone who tried the release and provided feedback. I'll push the bits and send out a note. thanks, Arun On Nov 13, 2014, at 3:08 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.6.0
Tsuyoshi, I think you have to pass along -Pdist too… afaik that is how the Jenkins we use for building release artifacts operates. For now, I've marked HADOOP-11316 for inclusion into 2.6.1 if required. thanks, Arun On Nov 18, 2014, at 7:26 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Hi, I also found that a following command fails with hadoop-2.6.0-rc1-src.tar.gz: $ mvn package -Pdocs -DskipTests -Dtar I filed it as HADOOP-11316. We should fix it if possible. Or, should we fix it for 2.6.1? Thanks, - Tsuyoshi On Tue, Nov 18, 2014 at 3:04 PM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Suresh, Tsz-Wo, Thanks for your comments. Could you check a comment by Masatake on HDFS-6833? thanks, - Tsuyoshi On Tue, Nov 18, 2014 at 7:14 AM, Tsz Wo Sze szets...@yahoo.com.invalid wrote: I agree that HDFS-6833 is not a blocker. It is not a problem for the blocks with replication = 3 and it is not a regression (please correct me if I am wrong.)Tsz-Wo On Monday, November 17, 2014 10:06 AM, Suresh Srinivas sur...@hortonworks.com wrote: Tsuyoshi, thanks for bringing up HDFS-6833. However, given it is a boundary condition (and should not cause issues when for files with replication factor 3), we should perhaps target this into 2.6.1 and not block this release. Thoughts? On Mon, Nov 17, 2014 at 4:17 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: +0(non-binding) HDFS-6833 is critical issue for us - could you help us to merge it into 2.6? Thanks, - Tsuyoshi On Sat, Nov 15, 2014 at 1:03 PM, Hitesh Shah hit...@apache.org wrote: +1 (binding) Built Hadoop from source, compiled Tez against the hadoop jars pushed to staging repo and ran a few example Tez jobs on a single node cluster. — HItesh On Nov 13, 2014, at 3:08 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- - Tsuyoshi -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- - Tsuyoshi -- - Tsuyoshi -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.6.0
Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.6.0
Sounds good. I'll create an rc1. Thanks. Arun On Nov 11, 2014, at 2:06 PM, Robert Kanter rkan...@cloudera.com wrote: Hi Arun, We were testing the RC and ran into a problem with the recent fixes that were done for POODLE for Tomcat (HADOOP-11217 for KMS and HDFS-7274 for HttpFS). Basically, in disabling SSLv3, we also disabled SSLv2Hello, which is required for older clients (e.g. Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be clear, it does not mean SSLv2, which is insecure. This also affects the MR shuffle in HADOOP-11243. The fix is super simple, so I think we should reopen these 3 JIRAs and put in addendum patches and get them into 2.6.0. thanks - Robert On Tue, Nov 11, 2014 at 1:04 PM, Ravi Prakash ravi...@ymail.com wrote: Hi Arun! We are very close to completion on YARN-1964 (DockerContainerExecutor). I'd also like HDFS-4882 to be checked in. Do you think these issues merit another RC? ThanksRavi On Tuesday, November 11, 2014 11:57 AM, Steve Loughran ste...@hortonworks.com wrote: +1 binding -patched slider pom to build against 2.6.0 -verified build did download, which it did at up to ~8Mbps. Faster than a local build. -full clean test runs on OS/X Linux Windows 2012: Same thing. I did have to first build my own set of the windows native binaries, by checking out branch-2.6.0; doing a native build, copying the binaries and then purging the local m2 repository of hadoop artifacts to be confident I was building against. For anyone who wants those native libs they will be up on https://github.com/apache/incubator-slider/tree/develop/bin/windows/ once it syncs with the ASF repos. afterwords: the tests worked! On 11 November 2014 02:52, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.6.0 that I would like to see released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0 The RC tag in git is: release-2.6.0-rc0 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1012. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.6.0
Folks, I've created a release candidate (rc0) for hadoop-2.6.0 that I would like to see released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0 The RC tag in git is: release-2.6.0-rc0 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1012. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Guava
… has been a constant pain w.r.t compatibility etc. Should we consider adopting a policy to not use guava in Common/HDFS/YARN? MR doesn't matter too much since it's application-side issue, it does hurt end-users though since they still might want a newer guava-version, but at least they can modify MR. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to hadoop-2.6
Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week. Thanks everyone! Arun On Oct 14, 2014, at 11:25 AM, Arun C Murthy a...@hortonworks.com wrote: 2.6.0 is close now. Here are the remaining blockers, I'm hoping cut an RC in the next week or so: http://s.apache.org/hadoop-2.6.0-blockers thanks, Arun On Sep 30, 2014, at 10:42 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created branch-2.6 to stabilize the release. Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already. By default new features should now be targeted to the version 2.7 henceforth - I've ensure all the projects have that version on jira. thanks, Arun On Sep 26, 2014, at 1:08 AM, Arun Murthy a...@hortonworks.com wrote: Sounds good. I'll branch this weekend and we can merge the jiras we discussed in this thread as they they get wrapped next week. Thanks everyone. Arun On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: We can branch off in a week or two so that work on branch-2 itself can go ahead with other features that can't fit in 2.6. Independent of that, we can then decide on the timeline of the release candidates once branch-2.6 is close to being done w.r.t the planned features. Branching it off can let us focus on specific features that we want in for 2.6 and then eventually blockers for the release, nothing else. There is a trivial pain of committing to one more branch, but it's worth it in this case IMO. A lot of efforts are happening in parallel from the YARN side from where I see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we don't branch off and selectively try to get stuff in, it is likely to be in a perpetual delay. My 2 cents. +Vinod On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas sur...@hortonworks.com wrote: Given some of the features are in final stages of stabilization, Arun, we should hold off creating 2.6 branch or building an RC by a week? All the features in flux are important ones and worth delaying the release by a week. On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to turn off the feature does not change the compatibility story. I'm willing to devote time to help review these JIRAs and kick the tires on the APIs, but my point above was that I'm not sure it'd all be done by the end of the week. Testing might also reveal additional changes that need to be made, which also might not happen by end-of-week. I guess the question before us is if we're comfortable putting something in branch-2.6 and then potentially adding API changes after. I'm okay with that as long as we're all aware that this might happen. Arun, as RM is this cool with you? Again, I like this feature and I'm fine with it's inclusion, just a heads up that we might need some extra time to finalize things before an RC can be cut. Thanks, Andrew On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze s29752-hadoop...@yahoo.com.invalid wrote: Hi, I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2. It gives us an impression that the feature is not yet well tested. Indeed, transparent encryption is a complicated feature which changes the core part of HDFS. It is not easy to get everything right. For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature. It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS. When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories. Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before. As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2. Regards, Tsz-Wo On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas sur...@hortonworks.com wrote: I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun
Re: Thinking ahead to hadoop-2.6
2.6.0 is close now. Here are the remaining blockers, I'm hoping cut an RC in the next week or so: http://s.apache.org/hadoop-2.6.0-blockers thanks, Arun On Sep 30, 2014, at 10:42 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created branch-2.6 to stabilize the release. Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already. By default new features should now be targeted to the version 2.7 henceforth - I've ensure all the projects have that version on jira. thanks, Arun On Sep 26, 2014, at 1:08 AM, Arun Murthy a...@hortonworks.com wrote: Sounds good. I'll branch this weekend and we can merge the jiras we discussed in this thread as they they get wrapped next week. Thanks everyone. Arun On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: We can branch off in a week or two so that work on branch-2 itself can go ahead with other features that can't fit in 2.6. Independent of that, we can then decide on the timeline of the release candidates once branch-2.6 is close to being done w.r.t the planned features. Branching it off can let us focus on specific features that we want in for 2.6 and then eventually blockers for the release, nothing else. There is a trivial pain of committing to one more branch, but it's worth it in this case IMO. A lot of efforts are happening in parallel from the YARN side from where I see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we don't branch off and selectively try to get stuff in, it is likely to be in a perpetual delay. My 2 cents. +Vinod On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas sur...@hortonworks.com wrote: Given some of the features are in final stages of stabilization, Arun, we should hold off creating 2.6 branch or building an RC by a week? All the features in flux are important ones and worth delaying the release by a week. On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to turn off the feature does not change the compatibility story. I'm willing to devote time to help review these JIRAs and kick the tires on the APIs, but my point above was that I'm not sure it'd all be done by the end of the week. Testing might also reveal additional changes that need to be made, which also might not happen by end-of-week. I guess the question before us is if we're comfortable putting something in branch-2.6 and then potentially adding API changes after. I'm okay with that as long as we're all aware that this might happen. Arun, as RM is this cool with you? Again, I like this feature and I'm fine with it's inclusion, just a heads up that we might need some extra time to finalize things before an RC can be cut. Thanks, Andrew On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze s29752-hadoop...@yahoo.com.invalid wrote: Hi, I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2. It gives us an impression that the feature is not yet well tested. Indeed, transparent encryption is a complicated feature which changes the core part of HDFS. It is not easy to get everything right. For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature. It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS. When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories. Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before. As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2. Regards, Tsz-Wo On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas sur...@hortonworks.com wrote: I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's
Re: Thinking ahead to hadoop-2.6
Re-sending… doesn't look like it made through. On Sep 30, 2014, at 10:42 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created branch-2.6 to stabilize the release. Committers, please exercise caution henceforth on commits other than the ones we've discussed on this thread already. By default new features should now be targeted to the version 2.7 henceforth - I've ensure all the projects have that version on jira. thanks, Arun On Sep 26, 2014, at 1:08 AM, Arun Murthy a...@hortonworks.com wrote: Sounds good. I'll branch this weekend and we can merge the jiras we discussed in this thread as they they get wrapped next week. Thanks everyone. Arun On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: We can branch off in a week or two so that work on branch-2 itself can go ahead with other features that can't fit in 2.6. Independent of that, we can then decide on the timeline of the release candidates once branch-2.6 is close to being done w.r.t the planned features. Branching it off can let us focus on specific features that we want in for 2.6 and then eventually blockers for the release, nothing else. There is a trivial pain of committing to one more branch, but it's worth it in this case IMO. A lot of efforts are happening in parallel from the YARN side from where I see. 2.6 is a little bulky if only on the YARN side and I'm afraid if we don't branch off and selectively try to get stuff in, it is likely to be in a perpetual delay. My 2 cents. +Vinod On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas sur...@hortonworks.com wrote: Given some of the features are in final stages of stabilization, Arun, we should hold off creating 2.6 branch or building an RC by a week? All the features in flux are important ones and worth delaying the release by a week. On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to turn off the feature does not change the compatibility story. I'm willing to devote time to help review these JIRAs and kick the tires on the APIs, but my point above was that I'm not sure it'd all be done by the end of the week. Testing might also reveal additional changes that need to be made, which also might not happen by end-of-week. I guess the question before us is if we're comfortable putting something in branch-2.6 and then potentially adding API changes after. I'm okay with that as long as we're all aware that this might happen. Arun, as RM is this cool with you? Again, I like this feature and I'm fine with it's inclusion, just a heads up that we might need some extra time to finalize things before an RC can be cut. Thanks, Andrew On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze s29752-hadoop...@yahoo.com.invalid wrote: Hi, I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2. It gives us an impression that the feature is not yet well tested. Indeed, transparent encryption is a complicated feature which changes the core part of HDFS. It is not easy to get everything right. For HDFS-6584: Archival Storage, it is a relatively simple and low risk feature. It introduces a new storage type ARCHIVE and the concept of block storage policy to HDFS. When a cluster is configured with ARCHIVE storage, the blocks will be stored using the appropriate storage types specified by storage policies assigned to the files/directories. Cluster admin could disable the feature by simply not configuring any storage type and not setting any storage policy as before. As Suresh mentioned, HDFS-6584 is in the final stages to be merged to branch-2. Regards, Tsz-Wo On Wednesday, September 24, 2014 7:00 AM, Suresh Srinivas sur...@hortonworks.com wrote: I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584
[jira] [Created] (HADOOP-11144) Update website to reflect that we use git, not svn
Arun C Murthy created HADOOP-11144: -- Summary: Update website to reflect that we use git, not svn Key: HADOOP-11144 URL: https://issues.apache.org/jira/browse/HADOOP-11144 Project: Hadoop Common Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy We need to update http://hadoop.apache.org/version_control.html to reflect that we use git, not svn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Dropping support for JDK6 in Apache Hadoop
[Apologies for the wide distribution.] Dear HBase/Hive/Pig/Oozie communities, We, over at Hadoop are considering dropping support for JDK6 this year. As you maybe aware we just released hadoop-2.5.0 and are now considering making the next release i.e. hadoop-2.6.0 the *last* release of Apache Hadoop which supports JDK6. This means, from hadoop-2.7.0 onwards we will not support JDK6 anymore and we *may* start relying on JDK7-specific apis. Now, the above releases a proposal and we do not want to pull the trigger without talking to projects downstream - hence the request for you feedback. Please feel free to forward this to other communities you might deem to be at risk from this too. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
+1 (binding) Verified sigs and ran sample jobs. Thanks for taking the lead on this Karthik. Arun On Aug 6, 2014, at 1:59 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Thinking ahead to hadoop-2.6
Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-10963) Move compile-time dependency to JDK7
Arun C Murthy created HADOOP-10963: -- Summary: Move compile-time dependency to JDK7 Key: HADOOP-10963 URL: https://issues.apache.org/jira/browse/HADOOP-10963 Project: Hadoop Common Issue Type: Bug Reporter: Arun C Murthy Fix For: 2.7.0 As discussed on the *-d...@hadoop.apache.org mailing list, this jira tracks moving to JDK7 and dropping support for JDK6. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Moving to JDK7, JDK8 and new major releases
Thanks everyone for the discussion. Looks like we have come to a pragmatic and progressive conclusion. In terms of execution of the consensus plan, I think a little bit of caution is in order. Let's give downstream projects more of a runway. I propose we inform HBase, Pig, Hive etc. that we are considering making 2.6 (not 2.5) the last JDK6 release and solicit their feedback. Once they are comfortable we can pull the trigger in 2.7. thanks, Arun On Jun 27, 2014, at 11:34 AM, Karthik Kambatla ka...@cloudera.com wrote: As someone else already mentioned, we should announce one future release (may be, 2.5) as the last JDK6-based release before making the move to JDK7. I am comfortable calling 2.5 the last JDK6 release. On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, responding to multiple messages here, Arun, thanks for the clarification regarding MR classpaths. It sounds like the story there is improved and still improving. However, I think we still suffer from this at least on the HDFS side. We have a single JAR for all of HDFS, and our clients need to have all the fun deps like Guava on the classpath. I'm told Spark sticks a newer Guava at the front of the classpath and the HDFS client still works okay, but this is more happy coincidence than anything else. While we're leaking deps, we're in a scary situation. API compat to me means that an app should be able to run on a new minor version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and have nothing break. If we muck with the classpath, my understanding is that this could break. Owen, bumping the minimum JDK version in a minor release like this should be a one-time exception as Tucu stated. A number of people have pointed out how painful a forced JDK upgrade is for end users, and it's not something we should be springing on them in a minor release unless we're *very* confident like in this case. Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on JDK7 across the CDH stack, so I think that's an indication that most ecosystem projects are ready to make the jump. Is that sufficient in your mind? For the record, I'm also +1 on the Tucu plan. Is it too late to do this for 2.5? I'll offer to help out with some of the mechanics. Thanks, Andrew On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth cnaur...@hortonworks.com wrote: I understood the plan for avoiding JDK7-specific features in our code, and your suggestion to add an extra Jenkins job is a great way to guard against that. The thing I haven't seen discussed yet is how downstream projects will continue to consume our built artifacts. If a downstream project upgrades to pick up a bug fix, and the jar switches to 1.7 class files, but their project is still building with 1.6, then it would be a nasty surprise. These are the options I see: 1. Make sure all other projects upgrade first. This doesn't sound feasible, unless all other ecosystem projects have moved to JDK7 already. If not, then waiting on a single long pole project would hold up our migration indefinitely. 2. We switch to JDK7, but run javac with -target 1.6 until the whole ecosystem upgrades. I find this undesirable, because in a certain sense, it still leaves a bit of 1.6 lingering in the project. (I'll assume that end-of-life for JDK6 also means end-of-life for the 1.6 bytecode format.) 3. Just declare a clean break on some version (your earlier email said 2.5) and start publishing artifacts built with JDK7 and no -target option. Overall, this is my preferred option. However, as a side effect, this sets us up for longer-term maintenance and patch releases off of the 2.4 branch if a downstream project that's still on 1.6 needs to pick up a critical bug fix. Of course, this is all a moot point if all the downstream ecosystem projects have already made the switch to JDK7. I don't know the status of that off the top of my head. Maybe someone else out there knows? If not, then I expect I can free up enough in a few weeks to volunteer for tracking down that information. Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur t...@cloudera.com wrote: Chris, Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you are still using jdk7 libraries and you could use new APIs, thus breaking jdk6 both at compile and runtime. you need to compile with jdk6 to ensure you are not running into that scenario. that is why i was suggesting the nightly jdk6 build/test jenkins job. On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth cnaur...@hortonworks.com wrote: I'm also +1 for getting us to JDK7 within the 2.x line after reading the
Re: [VOTE] Release Apache Hadoop 2.4.1
Aaron, Since the amend was just to the test, I'll keep this RC as-is. I'll also comment on jira. thanks, Arun On Jun 27, 2014, at 2:40 PM, Aaron T. Myers a...@cloudera.com wrote: I'm -0 on rc1. Note the latest discussion on HDFS-6527 which first resulted in that patch being reverted from branch-2.4.1 because it was believed it wasn't necessary, and then some more discussion which indicates that in fact the patch for HDFS-6527 should be included in 2.4.1, but with a slightly different test case. I believe that rc1 was actually created after the first backport of HDFS-6527, but before the revert, so rc1 should be functionally correct, but the test case is not quite correct in rc1, and I believe that rc1 does not currently reflect the actual tip of branch-2.4.1. I'm not going to consider this a deal-breaker, but seems like we should probably clean it up. To get this all sorted out properly, if we wanted to, I believe we should do another backport of HDFS-6527 to branch-2.4.1 including only the amended test case, and create a new RC from that point. Best, Aaron -- Aaron T. Myers Software Engineer, Cloudera On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Change by-laws on release votes: 5 days instead of 7
Folks, As discussed, I'd like to call a vote on changing our by-laws to change release votes from 7 days to 5. I've attached the change to by-laws I'm proposing. Please vote, the vote will the usual period of 7 days. thanks, Arun [main]$ svn diff Index: author/src/documentation/content/xdocs/bylaws.xml === --- author/src/documentation/content/xdocs/bylaws.xml (revision 1605015) +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) @@ -344,7 +344,16 @@ pVotes are open for a period of 7 days to allow all active voters time to consider the vote. Votes relating to code changes are not subject to a strict timetable but should be -made as timely as possible./p/li +made as timely as possible./p + + ul + li strongProduct Release - Vote Timeframe/strong + pRelease votes, alone, run for a period of 5 days. All other + votes are subject to the above timeframe of 7 days./p + /li + /ul + /li + /ul /section /body -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Moving to JDK7, JDK8 and new major releases
On Jun 24, 2014, at 4:22 PM, Andrew Wang andrew.w...@cloudera.com wrote: Since Hadoop apps can and do depend on the Hadoop classpath, the classpath is effectively part of our API. I'm sure there are user apps out there that will break if we make incompatible changes to the classpath. I haven't read up on the MR JIRA Arun mentioned, but there MR isn't the only YARN app out there. I think there is a some confusion/misunderstanding here. With hadoop-2 the user is completely in control of his own classpath (we had a similar, but limited capability in hadoop-1 w/ https://issues.apache.org/jira/browse/MAPREDUCE-1938). Furthermore, it's probably not well known that in hadoop-2 the user application (MR or otherwise) can also pick the JDK version by using JAVA_HOME env for the container. So, in effect, MR applications can continue to use java6 while YARN is running java7 - this hasn't been tested extensively though. This capability did not exist in hadoop-1. We've also made some progress with https://issues.apache.org/jira/browse/MAPREDUCE-1700 to defuse user jar-deps from MR system jars. https://issues.apache.org/jira/browse/MAPREDUCE-4421 also helps by ensuring MR applications can pick exact version of MR jars they were compiled against; and not rely on cluster installs. Hope that helps somewhat. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: hadoop-2.5 - June end?
Folks, I'd appreciate some help here. Due to family reasons (all good ones), I'll be away for a couple of weeks. Can someone else pick up the RM duties for hadoop-2.5? Maybe Andrew since he's expressed interest in the past? I will pick up the thread again for hadoop-2.6, but I don't want to block hadoop-2.5 due to my non-availability. thanks, Arun On Jun 9, 2014, at 9:39 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As you can see from the Roadmap wiki, it looks like several items are still a bit away from being ready. I think rather than wait for them, it will be useful to create an intermediate release (2.5) this month - I think ATS security is pretty close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of the month, with a branch a couple of weeks prior. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: hadoop-2.5 - June end?
Thanks Karthik! I've updated https://wiki.apache.org/hadoop/Roadmap with features which are very close to completion. Let's see if that makes sense and if we get any further feedback. Arun On Jun 23, 2014, at 2:09 PM, Karthik Kambatla ka...@cloudera.com wrote: I can pick up the RM duties for 2.5. If I run into any HDFS doubts, I might need some help from someone more familiar with HDFS. On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I'd appreciate some help here. Due to family reasons (all good ones), I'll be away for a couple of weeks. Can someone else pick up the RM duties for hadoop-2.5? Maybe Andrew since he's expressed interest in the past? I will pick up the thread again for hadoop-2.6, but I don't want to block hadoop-2.5 due to my non-availability. thanks, Arun On Jun 9, 2014, at 9:39 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As you can see from the Roadmap wiki, it looks like several items are still a bit away from being ready. I think rather than wait for them, it will be useful to create an intermediate release (2.5) this month - I think ATS security is pretty close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of the month, with a branch a couple of weeks prior. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 0.23.11
+1 (binding) Thanks for all your efforts driving 0.23.* Thomas co! Arun On Jun 19, 2014, at 8:14 AM, Thomas Graves tgra...@yahoo-inc.com.INVALID wrote: Hey Everyone, There have been various bug fixes that have went into branch-0.23 since the 0.23.10 release. We think its time to do a 0.23.11. This is also the last planned release off of branch-0.23 we plan on doing. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.11-candidate-0/ The RC Tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.11-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days til June 26th. I am +1 (binding). thanks, Tom Graves -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.4.1
Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] Change by-laws on release votes: 5 days instead of 7
Uma, Voting periods are defined in *minimum* terms, so it already covers what you'd like to see i.e. the vote can continue longer. thanks, Arun On Jun 21, 2014, at 2:19 AM, Gangumalla, Uma uma.ganguma...@intel.com wrote: How about proposing vote for 5days and give chance to RM for extending vote for 2more days( total to 7days) if the rc did not receive enough vote within 5days? If a rc received enough votes in 5days, RM can close vote. I can see an advantage of 7days voting is, that will cover all the week and weekend days. So, if someone wants to test on weekend time(due to the weekday schedules), that will give chance to them. Regards, Uma -Original Message- From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Saturday, June 21, 2014 11:25 AM To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: [DISCUSS] Change by-laws on release votes: 5 days instead of 7 Folks, I'd like to propose we change our by-laws to reduce our voting periods on new releases from 7 days to 5. Currently, it just takes too long to turn around releases; particularly if we have critical security fixes etc. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Plans of moving towards JDK7 in trunk
Andrew, On Jun 21, 2014, at 8:01 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hi Steve, let me confirm that I understand your proposal correctly: - Release an intermediate Hadoop 3 a few months out, based on JDK7 and with bumped library versions - Release a Hadoop 4 mid next year, based on JDK8 I question the utility of an intermediate Hadoop 3 like this. Assuming that it gets out in September (i.e. roughly when a 2.6 would land), we're looking at a valid lifespan of about 7 months before JDK7 is EOL i JDK6 eol was Feb 2013 and, a year later, we are still have customers using it - which means we can't drop it yet. http://www.oracle.com/technetwork/java/eol-135779.html Given that, it seems highly unlikely everyone will suddenly jump to JDK8 by April of next year... I suspect this means we'd have to support JDK7 at least till late 2015. I think, that, is really key regardless of version numbers. Furthermore, if we, as a community, maintain discipline in terms of wire-compat, rolling-upgrades etc. we are better off making a major release every year - as you put, no more 'Big Bang' releases. We have to, as a development community, ourselves get over the 'trauma' of major releases - I do realize the irony here - but it's requisite to help our users feel confident in upgrading at a reasonable rate. So, something like this could work: # hadoop-2 / jdk6 - Oct 2013 # hadoop-3 / jdk7 - Oct 2014 # hadoop-4 / jdk8 - Oct 2015 Having said that, it would also be prudent to co-release hadoop-2/hadoop-3 hadoop-3/hadoop-4 with requisite jdk versions. Maybe even hadoop-4 beta by middle of 2015. As such, it a good idea to allow trunk to move to jdk7 now - it's good practice as we will have to do the same for jdk8. It does help, a lot, that we have now de-coupled user dependencies from the system with YARN. For e.g. we could run hadoop-2 MR on hadoop-3 YARN, even if there is some work remaining... see MAPREDUCE-4551. Future reliance on technologies like Docker will help further. Thoughts? Arun If this release also breaks compatibility by changing library versions, then it looks less and less appealing from a user perspective. I suspect it would end up seeing low adoption as everyone waits (at most) 7 months for the JDK8-based release to emerge. I'd be more okay with an intermediate release with no incompatible changes whatsoever besides bumping the JDK requirement to JDK7. However, it'd still be a weak release considering that branch-2 already runs fine on JDK7, and it looks somewhat bad publicly as we burn another major release number less than a year since 2.x going GA. This is why I'd like to keep my original proposal on the table: keep going with branch-2 in the near term, while working towards a JDK8-based Hadoop 3 by April next year. It doesn't need to be a big bang release either. I'd be delighted if we could rolling upgrade from one to the other. I just didn't want to rule out the inclusion of some very compelling feature outright. Trust me though, I'd be the first person to ask about compatibility if such a feature does come up. I'll also posit that people will shy away from using JDK8 features while branch-2 remains in active use. There's definitely some new shiny there, but nothing compelling enough to me personally when weighed against the pain of harder branch-2 backports. Let's try to keep this thread focused on the planning side of things though, deferring JDK-feature-related discussion to a different thread. We'd need to draw up a code-style doc on the wiki, but it sounds like something Steve and/or I could draft initially. Thanks, Andrew On Fri, Jun 20, 2014 at 10:02 PM, Arun C Murthy a...@hortonworks.com wrote: On Jun 20, 2014, at 9:51 PM, Steve Loughran ste...@hortonworks.com wrote: On 20 June 2014 21:35, Steve Loughran ste...@hortonworks.com wrote: This actually argues in favour of -renaming branch-2 branch-3 after a release -making trunk hadoop-4 -getting hadoop 3 released off the new branch-3 out in 2014, effectively being an iteration of branch-2 with updated java , moves of (off?) guava, off jetty, lib changes, but no other significant big bang features Hadoop 4.x then becomes the 2015 release, which can add more stuff. In particular, anything that goes into Hadoop 4 for which there's no intent to support in hadoop 2 3, can use the java 8 language features sooner rather than later. I should add that I'm willing to be the person who gets the Java-7 based Hadoop 3.x out the door later this year +1 that makes sense to me. Thanks for volunteering Steve - I'm glad to share the pain… ;-) Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended
Re: Plans of moving towards JDK7 in trunk
After further consideration, here is an alternate. On Jun 21, 2014, at 11:14 AM, Arun C. Murthy a...@hortonworks.com wrote: JDK6 eol was Feb 2013 and, a year later, we are still have customers using it - which means we can't drop it yet. http://www.oracle.com/technetwork/java/eol-135779.html Given that, it seems highly unlikely everyone will suddenly jump to JDK8 by April of next year... I suspect this means we'd have to support JDK7 at least till late 2015. I think, that, is really key regardless of version numbers. Furthermore, if we, as a community, maintain discipline in terms of wire-compat, rolling-upgrades etc. we are better off making a major release every year - as you put, no more 'Big Bang' releases. Looking at the big picture, I believe the users of Apache Hadoop would be better served by us if we prioritized operational aspects such as rolling upgrades, wire-compatibility, binary etc. for a couple of years. Since not everyone has moved to hadoop-2 yet, talk of more incompatibility between hadoop-2/hadoop-3 or between hadoop-3/hadoop-4 within the next 12 months would certainly be a big issue for users - especially w.r.t rolling upgrades, wire-compat etc. So, I think we should prioritize these operational aspects for users above everything else. Sure, jdk versions, features etc. are important, but lower in priority. I'd also like to reiterate my concern on *dropping* support for a JDK7 - we need to support it till end of 2015 at the very least; happy to ship a version of Hadoop which is JDK8 only in 2015 - it just needs to support rolling-upgrades from the JDK7 Hadoop till end of 2015. With that in mind... I actually like Andrew's suggestion below: On Jun 21, 2014, at 8:01 AM, Andrew Wang andrew.w...@cloudera.com wrote: I'd be more okay with an intermediate release with no incompatible changes whatsoever besides bumping the JDK requirement to JDK7. Taking that thought to it's logical conclusion, we can de-couple the dual concerns of JDK versions and major releases but bumping up our software dependencies (JDK, guice etc.) at well-defined and well-articulated releases. The reason to so would be to ensure we *do not* sneak in operational incompatibilities in the guise of bumping JDK versions. So, we could do something like: # hadoop-2.30+ is JDK7, but provides rolling upgrades and wire-compat with hadoop-2.2+; say in Oct 2014 # hadoop-2.50+ is JDK8, but provides rolling upgrades and wire-compat with hadoop-2.2+; say in June 2015 (or even earlier). This scheme certainly has some dis-advantages, however it has the significant advantage of making it *very* clear to end-users and administrators that we take operational aspects seriously. Also, this is something we already have done i.e. we updated some of our software deps in hadoop-2.4 v/s hadoop-2.2 - clearly not something as dramatic as JDK. Here are some examples: https://issues.apache.org/jira/browse/HADOOP-9991 https://issues.apache.org/jira/browse/HADOOP-10102 https://issues.apache.org/jira/browse/HADOOP-10103 https://issues.apache.org/jira/browse/HADOOP-10104 https://issues.apache.org/jira/browse/HADOOP-10503 In summary, the key goals we should keep in mind are: # Operational aspects such as rolling upgrades, wire-compat etc. for the next couple of years. # Support JDK7 till end of 2015 at least, even if we decide to support JDK8 sometime in 2015. Just ensure wire-compat, rolling-upgrades etc. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.4.1
Thanks for the feedback Vinod, Akira Kihwal. I'll re-spin rc1 with MAPREDUCE-5830 HDFS-6527. @Kihwal - Can you, please, merge HDFS-6527 to branch-2.4 and branch-2.4.1? thanks, Arun On Jun 20, 2014, at 7:32 AM, Kihwal Lee kih...@yahoo-inc.com.INVALID wrote: If we ever respin 2.4.1, I strongly suggest HDFS-6527 be included. Kihwal On 6/19/14, 4:56 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: I think we should include this issue in 2.4.1, so I uploaded a patch to fix it. I'll appreciate your review. Thanks, Akira (2014/06/18 12:13), Vinod Kumar Vavilapalli wrote: There is one item [MAPREDUCE-5830 HostUtil.getTaskLogUrl is not backwards binary compatible with 2.3] marked for 2.4. Should we include it? There is no patch there yet, it doesn't really help much other than letting older clients compile - even if we put the API back in, the URL returned is invalid. +Vinod On Jun 16, 2014, at 9:27 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.4.1 (bug-fix release) that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Plans of moving towards JDK7 in trunk
On Jun 20, 2014, at 9:51 PM, Steve Loughran ste...@hortonworks.com wrote: On 20 June 2014 21:35, Steve Loughran ste...@hortonworks.com wrote: This actually argues in favour of -renaming branch-2 branch-3 after a release -making trunk hadoop-4 -getting hadoop 3 released off the new branch-3 out in 2014, effectively being an iteration of branch-2 with updated java , moves of (off?) guava, off jetty, lib changes, but no other significant big bang features Hadoop 4.x then becomes the 2015 release, which can add more stuff. In particular, anything that goes into Hadoop 4 for which there's no intent to support in hadoop 2 3, can use the java 8 language features sooner rather than later. I should add that I'm willing to be the person who gets the Java-7 based Hadoop 3.x out the door later this year +1 that makes sense to me. Thanks for volunteering Steve - I'm glad to share the pain… ;-) Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[DISCUSS] Change by-laws on release votes: 5 days instead of 7
Folks, I'd like to propose we change our by-laws to reduce our voting periods on new releases from 7 days to 5. Currently, it just takes too long to turn around releases; particularly if we have critical security fixes etc. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.4.1
Folks, I've created a release candidate (rc0) for hadoop-2.4.1 (bug-fix release) that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: hadoop-2.5 - June end?
No, I'm currently in the middle of the 2.4.1 RC. Watch this space. thanks, Arun On Jun 10, 2014, at 11:46 AM, Ravi Prakash ravi...@ymail.com wrote: Does this also mean that there won't be a 2.4.1 Apache release? On Tuesday, June 10, 2014 9:45 AM, Suresh Srinivas sur...@hortonworks.com wrote: We should also include extended attributes feature for HDFS from HDFS-2006 for release 2.5. On Mon, Jun 9, 2014 at 9:39 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As you can see from the Roadmap wiki, it looks like several items are still a bit away from being ready. I think rather than wait for them, it will be useful to create an intermediate release (2.5) this month - I think ATS security is pretty close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of the month, with a branch a couple of weeks prior. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
hadoop-2.5 - June end?
Folks, As you can see from the Roadmap wiki, it looks like several items are still a bit away from being ready. I think rather than wait for them, it will be useful to create an intermediate release (2.5) this month - I think ATS security is pretty close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of the month, with a branch a couple of weeks prior. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
hadoop-2.4.1
Folks, Here is a handy short-cut to track 2.4.1: http://s.apache.org/hadoop-2.4.1-blockers I'm hoping we can get the majority of this in by end-of-week and have an RC for consideration. Committers - I appreciate if you could try treat review/commit of these as high-priority. Also, please feel free to add other *really* important fixes you'd like to see - let's also try be super cautious adding new content. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead
On Apr 12, 2014, at 10:09 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 for starting to think about 2.5. Early June seems a little early to me - we had talked about a quarterly release cadence and this would be about half that. We can debate whether or June is too early for hadoop-2.5 and it's contents, but I recollect there was consensus that releases every 6 weeks (or so) makes sense. Else, with lesser frequency, we'll see a mad rush for everything to get into *the* quarterly release. As an example: hadoop-2.3 helped reduce pressure on contents in hadoop-2.4. In my experience, even if we do target 6 weeks it will be closer to ~8 weeks before we can *corral* a release. OTOH, a 12 week cycle would result in a release every 4-5 months... this is why the ASF encourages projects/PMCs to make frequent releases. Arun I'm having trouble editing the wiki, but I think Timeline Server stability (e.g. security and locking down APIs) should go on that list. I think we should probably take YARN-1404 off the list - even with 3 months it's unlikely to be complete. On Sat, Apr 12, 2014 at 9:50 AM, Chris Nauroth cnaur...@hortonworks.comwrote: +1 The proposed content for 2.5 in the roadmap wiki looks good to me. On Apr 12, 2014 7:26 AM, Arun C Murthy a...@hortonworks.com wrote: Gang, With hadoop-2.4 out, it's time to think ahead. In the short-term hadoop-2.4.1 is in order; particularly with https://issues.apache.org/jira/browse/MAPREDUCE-5830 (it's a break to @Private API, unfortunately something Hive is using - sigh!). There are some other fixes which testing has uncovered; so it will be nice to pull them them in. I'm thinking of an RC by end of the coming week - committers, please be *very* conservative when getting stuff into 2.4.1 (i.e. merging to branch-2.4). Next up, hadoop-2.5. I've updated https://wiki.apache.org/hadoop/Roadmap with some candidates for consideration - please chime in and say 'aye'/'nay' or add new content. IAC, I suspect that list is too large. Rather than wait for everything it would be better to plan on releasing it on a time-bound manner; particularly around the Hadoop Summit. If that makes sense; I think we should target branching for 2.5 by mid-May to get it stable and released by early June. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead
Sandy, On Apr 12, 2014, at 10:09 AM, Sandy Ryza sandy.r...@cloudera.com wrote: I'm having trouble editing the wiki, but I think Timeline Server stability (e.g. security and locking down APIs) should go on that list. I'm very glad to see you are very passionate about security for ATS since you've asked about it a few times around here. I just opened https://issues.apache.org/jira/browse/YARN-1935. Would you be willing to volunteer to take this on? Much appreciated. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Thinking ahead
Gang, With hadoop-2.4 out, it's time to think ahead. In the short-term hadoop-2.4.1 is in order; particularly with https://issues.apache.org/jira/browse/MAPREDUCE-5830 (it's a break to @Private API, unfortunately something Hive is using - sigh!). There are some other fixes which testing has uncovered; so it will be nice to pull them them in. I'm thinking of an RC by end of the coming week - committers, please be *very* conservative when getting stuff into 2.4.1 (i.e. merging to branch-2.4). Next up, hadoop-2.5. I've updated https://wiki.apache.org/hadoop/Roadmap with some candidates for consideration - please chime in and say 'aye'/'nay' or add new content. IAC, I suspect that list is too large. Rather than wait for everything it would be better to plan on releasing it on a time-bound manner; particularly around the Hadoop Summit. If that makes sense; I think we should target branching for 2.5 by mid-May to get it stable and released by early June. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HADOOP-10382) Add Apache Tez to the Hadoop homepage as a related project
[ https://issues.apache.org/jira/browse/HADOOP-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved HADOOP-10382. Resolution: Fixed I just committed this. Add Apache Tez to the Hadoop homepage as a related project -- Key: HADOOP-10382 URL: https://issues.apache.org/jira/browse/HADOOP-10382 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: HADOOP-10382.patch, HADOOP-10382.patch Add Apache Tez to the Hadoop homepage as a related project -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [VOTE] Release Apache Hadoop 2.4.0
Gera,
Apologies for the late response, I decided to take the weekend off! *smile*
Thanks for the feedback, I took a look at your list and here are some
observations:
On Apr 4, 2014, at 5:06 PM, Gera Shegalov g...@shegalov.com wrote:
I built the release from the rc tag, enabled timeline history service and
ran a sleep job on a pseudo-distributed cluster.
I encourage another rc, for 2.4.0 (non-binding)
1) Despite the discussion on YARN-1701, timeline AHS still sets
yarn.timeline-service.generic-application-history.fs-history-store.uri to a
location under ${hadoop.log.dir} that is meant for local file system, but
uses it on HDFS by default.
This is certainly unfortunate, but essentially it's just a bad default config
value. As such, there is a pretty simple clear workaround which I'll
release-doc.
2) Critical patch for WebHdfs/Hftp to fix the filesystem contract HDFS-6143
is not included
I had already commented on the jira around the time of RC as you may remember.
Also, this isn't a regression - this bug has existed for quite a while now.
Furthermore, there are some back-compatibility concerns we still need to
address.
3) Several patches that already proved themselves useful for diagnostics in
production and have been available for some months are still not included.
MAPREDUCE-5044/YARN-1515 is the most obvious example. Our users need to see
where the task container JVM got stuck when it was timed out by AM.
Please accept my personal apologies, this wasn't on my radar to review. In
future, pls drop me a note if you don't see any activity on a jira you care
about - I'm sure others like Vinod, Jason etc. would love to help too; it's
just something that got missed by the community collectively. Having said that,
as you can imagine, new features cannot be a blocker to a release; this sets up
a dynamic where it's impossible to please everyone and we cannot ship any
release… as you may have seen it's been quite an effort for the last 3-4 weeks
just to corral everyone into fixing/reviewing/committing blocker bugs itself.
*smile*
Overall, please feel free to share your feedback, but the above list doesn't
seem to have a critically bad release blocker.
Also, I plan to follow up with a 2.4.1 in a couple of weeks; I'll make sure to
include as much as your list as possible - and future ones.
thanks,
Arun
Thanks,
Gera
On Fri, Apr 4, 2014 at 3:51 PM, Azuryy azury...@gmail.com wrote:
Arun,
Do you mean you will cut another RC for 2.4?
Sent from my iPhone5s
On 2014年4月5日, at 3:50, Arun C. Murthy a...@hortonworks.com wrote:
Thanks for helping Tsuyoshi. Pls mark them as Blockers and set the
fix-version to 2.4.1.
Thanks again.
Arun
On Apr 3, 2014, at 11:38 PM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
wrote:
Hi,
Updated a test result log based on the result of 2.4.0-rc0:
https://gist.github.com/oza/9965197
IMO, there are some blockers to be fixed:
* MAPREDUCE-5815(TestMRAppMaster failure)
* YARN-1872(TestDistributedShell failure)
* HDFS: TestSymlinkLocalFSFileSystem failure on Linux (I cannot find
JIRA about this failure)
Now I'm checking the problem reported by Azuryy.
Thanks,
- Tsuyoshi
On Fri, Apr 4, 2014 at 8:55 AM, Tsuyoshi OZAWA
ozawa.tsuyo...@gmail.com wrote:
Hi,
Ran tests and confirmed that some tests(TestSymlinkLocalFSFileSystem)
fail.
The log of the test failure is as follows:
https://gist.github.com/oza/9965197
Should we fix or disable the feature?
Thanks,
- Tsuyoshi
On Mon, Mar 31, 2014 at 6:22 PM, Arun C Murthy a...@hortonworks.com
wrote:
Folks,
I've created a release candidate (rc0) for hadoop-2.4.0 that I would
like to get released.
The RC is available at:
http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0
The RC tag in svn is here:
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0
The maven artifacts are available via repository.apache.org.
Please try the release and vote; the vote will run for the usual 7
days.
thanks,
Arun
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
entity to
which it is addressed and may contain information that is
confidential,
privileged and exempt from disclosure under applicable law. If the
reader
of this message is not the intended recipient, you are hereby
notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately
and delete it from your system. Thank You.
--
- Tsuyoshi
--
- Tsuyoshi
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to
which it is addressed and may contain information that is confidential,
privileged and exempt
Re: [VOTE] Release Apache Hadoop 2.4.0
With 11 +1s (4 binding) and no -1s the vote passes. Thanks to everyone who tried out the release and passed their feedback along. I'll send a note out once I actually get the bits out and the site updated etc. thanks, Arun On Mar 31, 2014, at 2:22 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.4.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.4.0
Thanks for helping Tsuyoshi. Pls mark them as Blockers and set the fix-version to 2.4.1. Thanks again. Arun On Apr 3, 2014, at 11:38 PM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Hi, Updated a test result log based on the result of 2.4.0-rc0: https://gist.github.com/oza/9965197 IMO, there are some blockers to be fixed: * MAPREDUCE-5815(TestMRAppMaster failure) * YARN-1872(TestDistributedShell failure) * HDFS: TestSymlinkLocalFSFileSystem failure on Linux (I cannot find JIRA about this failure) Now I'm checking the problem reported by Azuryy. Thanks, - Tsuyoshi On Fri, Apr 4, 2014 at 8:55 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Hi, Ran tests and confirmed that some tests(TestSymlinkLocalFSFileSystem) fail. The log of the test failure is as follows: https://gist.github.com/oza/9965197 Should we fix or disable the feature? Thanks, - Tsuyoshi On Mon, Mar 31, 2014 at 6:22 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.4.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- - Tsuyoshi -- - Tsuyoshi -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.4.0
Folks, I've created a release candidate (rc0) for hadoop-2.4.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to 2.4
Update: We are now down to just 8 blockers of which 4 are already PA. I know it's getting in good shape per our QE gang too. If things go well, I plan to create an RC later half of next week. thanks, Arun On Mar 11, 2014, at 7:11 AM, Arun C Murthy a...@hortonworks.com wrote: Sorry, the previous link had a bug, the correct one is: http://s.apache.org/hadoop-2.4.0-blockers. We are currently down to 12 blockers; with several PA. thanks, Arun On Mar 6, 2014, at 1:40 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, Most of the big-ticket items are already in, awesome! I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we get through the list of blockers. Here is a handy link: http://s.apache.org/hadoop-2.4-blockers If you find more, please set Target Version to 2.4.0 and mark it a blocker. I'll try nudging people to start closing these soon, appreciate any help! thanks, Arun On Feb 20, 2014, at 3:45 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Azuryy Suresh. I've updated the roadmap wiki to reflect this. Arun On Feb 20, 2014, at 2:01 PM, Suresh Srinivas sur...@hortonworks.com wrote: Arun, Some of the previously 2.4 targeted features were made available in 2.3: - Heterogeneous storage support - Datanode cache The following are being targeted for 2.4: - Use protobuf for fsimge (already in) - ACLs (in trunk. In a week or so, this will be merged to branch-2.4) - Rolling upgrades (last bunch of jiras being worked in feature branch. Will be in 2.4 in around two weeks. Currently testing is in progress) So HDFS features should be ready in two weeks. On Sat, Feb 15, 2014 at 4:47 PM, Azuryy azury...@gmail.com wrote: Hi， I think you omit some key pieces in 2.4 Protobuf fsimage, rolling upgrade are also targeting 2.4 Sent from my iPhone5s On 2014年2月16日, at 6:59, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended
Re: Release notes for 2.3.0 on website
Andrew, This needs a fix to the docs in the main tree, not just the site. Agree we should fix this going fwd. thanks, Arun On Mar 7, 2014, at 10:51 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I found this page when googling for Hadoop 2.3.0 documentation, looks like it's pretty out of date (references new features like federation and YARN). Arun, should we update this with your release blurb? I could also try my hand at pushing the site if you're busy. http://hadoop.apache.org/docs/r2.3.0/ Thanks, Andrew -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to 2.4
Sorry, the previous link had a bug, the correct one is: http://s.apache.org/hadoop-2.4.0-blockers. We are currently down to 12 blockers; with several PA. thanks, Arun On Mar 6, 2014, at 1:40 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, Most of the big-ticket items are already in, awesome! I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we get through the list of blockers. Here is a handy link: http://s.apache.org/hadoop-2.4-blockers If you find more, please set Target Version to 2.4.0 and mark it a blocker. I'll try nudging people to start closing these soon, appreciate any help! thanks, Arun On Feb 20, 2014, at 3:45 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Azuryy Suresh. I've updated the roadmap wiki to reflect this. Arun On Feb 20, 2014, at 2:01 PM, Suresh Srinivas sur...@hortonworks.com wrote: Arun, Some of the previously 2.4 targeted features were made available in 2.3: - Heterogeneous storage support - Datanode cache The following are being targeted for 2.4: - Use protobuf for fsimge (already in) - ACLs (in trunk. In a week or so, this will be merged to branch-2.4) - Rolling upgrades (last bunch of jiras being worked in feature branch. Will be in 2.4 in around two weeks. Currently testing is in progress) So HDFS features should be ready in two weeks. On Sat, Feb 15, 2014 at 4:47 PM, Azuryy azury...@gmail.com wrote: Hi， I think you omit some key pieces in 2.4 Protobuf fsimage, rolling upgrade are also targeting 2.4 Sent from my iPhone5s On 2014年2月16日, at 6:59, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to 2.4
Other than Azuryy I haven't heard from HDFS folks. To help move this discussion along, I've created branch-2.4. Let's continue discussions here. Committers, please exercise caution to ensure we only commit necessary bug-fixes henceforth. thanks, Arun On Feb 15, 2014, at 2:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to 2.4
Other than Azuryy I haven't heard from HDFS folks. To help move this discussion along, I've created branch-2.4. Let's continue discussions here. Committers, please exercise caution to ensure we only commit necessary bug-fixes henceforth. thanks, Arun On Feb 15, 2014, at 2:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to 2.4
On Feb 17, 2014, at 9:44 PM, Arun C Murthy a...@hortonworks.com wrote: Committers, please exercise caution to ensure we only commit necessary bug-fixes henceforth. Forgot to add… Committers: I've created version 2.5.0 in jira with the expectation that most things will now commit to that version. thanks, Arun On Feb 15, 2014, at 2:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Thinking ahead to 2.4
Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.3.0
Folks, I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun PS: Thanks to Andrew, Vinod Alejandro for all their help in various release activities. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Re-swizzle 2.3
I punted YARN-1444 to 2.4 since it's a long-standing issue. Jian is away and I don't see YARN-1577 YARN-1206 making much progress till he is back; so I'm inclined to push both to 2.4 too. Any objections? Looks like Daryn has both HADOOP-10301 HDFS-4564 covered. Overall, I'll try get this out in next couple of days if we can clear the list. thanks, Arun On Feb 3, 2014, at 12:14 PM, Arun C Murthy a...@hortonworks.com wrote: An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN. Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Re-swizzle 2.3
An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN. Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Thanks Vinod, appreciate it! I think we are very close. Here is a handy ref. to the list of blockers: http://s.apache.org/hadoop-2.3.0-blockers I'd appreciate if folks can help expedite these fixes, and, equally importantly bring up others they feel should be blockers for 2.3.0. thanks, Arun On Jan 30, 2014, at 12:42 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: That was quite some exercise, but I'm done with it now. Updated YARN's and MAPREDUCE's CHANGES.txt on trunk, branch-2 and branch-2.3. Let me know if you find some inaccuracies. Thanks, +Vinod On Jan 29, 2014, at 10:49 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Okay, I'll look at YARN and MR CHANGES.txt problems. Seems like they aren't addressed yet. +Vinod On Jan 29, 2014, at 3:24 PM, Andrew Wang andrew.w...@cloudera.com wrote: I just finished tuning up branch-2.3 and fixing up the HDFS and Common CHANGES.txt in trunk, branch-2, and branch-2.3. I had to merge back a few JIRAs committed between the swizzle and now where the fix version was 2.3 but weren't in branch-2.3. I think the only two HDFS and Common JIRAs that are marked for 2.4 are these: HDFS-5842 Cannot create hftp filesystem when using a proxy user ugi and a doAs on a secure cluster HDFS-5781 Use an array to record the mapping between FSEditLogOpCode and the corresponding byte value Jing, these both look safe to me if you want to merge them back, or I can just do it. Thanks, Andrew On Wed, Jan 29, 2014 at 1:21 PM, Doug Cutting cutt...@apache.org wrote: On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe jl...@yahoo-inc.com wrote: It is a bit concerning that the JIRA history showed that the target version was set at some point in the past but no record of it being cleared. Perhaps the version itself was renamed? Doug -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. signature.asc -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Mostly ready for a jira perspective. Committers - Henceforth, please use extreme caution while committing to branch-2.3. Please commit *only* blockers to 2.3. thanks, Arun On Jan 28, 2014, at 3:30 PM, Arun C Murthy a...@hortonworks.com wrote: Fixing up stuff now, thanks to Andrew for volunteering to help with Common/HDFS. Arun On Jan 28, 2014, at 10:54 AM, Arun C Murthy a...@hortonworks.com wrote: Sorry, missed this. Go ahead, I'll fix things up at the back end. Thanks. On Jan 28, 2014, at 12:11 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Going forward with commits because it seems like others have been doing so On Mon, Jan 27, 2014 at 1:31 PM, Sandy Ryza sandy.r...@cloudera.com wrote: We should hold off commits until that's done, right? On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy a...@hortonworks.comwrote: Yep, on it as we speak. :) Arun On Jan 27, 2014, at 12:36 PM, Jason Lowe jl...@yahoo-inc.com wrote: Thanks, Arun. Are there plans to update the Fix Versions and CHANGES.txt accordingly? There are a lot of JIRAs that are now going to ship in 2.3.0 but the JIRA and CHANGES.txt says they're not fixed until 2.4.0. Jason On 01/27/2014 08:47 AM, Arun C Murthy wrote: Done. I've re-created branch-2.3 from branch-2. thanks, Arun On Jan 23, 2014, at 2:40 AM, Arun Murthy a...@hortonworks.com wrote: Based on the discussion at common-dev@, we've decided to target 2.3 off the tip of branch-2 based on the 2 major HDFS features which are Heterogenous Storage (HDFS-2832) and HDFS Cache (HDFS-4949). I'll create a new branch-2.3 on (1/24) at 6pm PST. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Logistics for releasing 2.4
think a 2.4 with just HDFS-4949, HDFS-2832, and YARN-149 would be an attractive and stable release, and is something we could actually cut this week and vote on before the month is out. The other stuff we can ship in Feb or March when it's gotten a chance to bake for a bit, and culturally speaking, the fact that it's in 2.5 rather than 2.4 shouldn't be a big deal. Thanks, Andrew On Tue, Jan 21, 2014 at 11:37 AM, Suresh Srinivas sur...@hortonworks.comwrote: There is not much progress on symlinks issue. I think we should move forward with 2.4 release with symlinks disabled. Status of 2.4 features from HDFS so far: - HDFS-2832 Heterogeneous storage support has been merged - HDFS-5535 rolling upgrades work is in progress - HDFS-4685 ACL related work is close to completion - HDFS-4949 As Andrew has proposed, this will be soon merged into 2.4 Regards, Suresh On Tue, Jan 21, 2014 at 11:12 AM, Arun C Murthy a...@hortonworks.com wrote: Andrew, I'm almost ready to push out rc0 for 2.3 (been testing it overnight), I'm pretty sure I'll get that out tonight. However, AHS (YARN-321) is very close (merge vote going on) … so that will definitely make it in very soon. So, my plan is essentially the same i.e. release 2.4 end of the month (after a bit more testing of RM HA in secure mode). Thanks for the offer, I'll ping you if I need any help. OTOH, can someone from HDFS chime in on status of symlinks? Arun On Jan 20, 2014, at 4:19 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I'm pretty excited to see a 2.4 this month if possible. Since I think people were favorable to the idea of time-based releases, how do we feel about just cutting branch-2 and spinning up the release process for our January goal? Looking at the roadmap (https://wiki.apache.org/hadoop/Roadmap), on the HDFS side, I plan to post a branch-2 patch for HDFS-4949 this week, and HDFS-2832 is already in. On the YARN side, it appears that RM HA is in, but the other three features (AHS, unmanaged containers, and dynamic resource configuration) remain unresolved. I think a 2.4 with HDFS-4949, HDFS-2832, and YARN-149 is already a pretty nice release. If it'd help, I'm willing to volunteer as release manager to help get this out the door. Thoughts welcome! Thanks, Andrew -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You
Re: [VOTE] Merge YARN-321 Generic Application History Service to trunk
+1 (binding) Arun On Jan 17, 2014, at 12:53 PM, Zhijie Shen zs...@hortonworks.com wrote: Hi folks, As previously discussed here (http://markmail.org/message/iscvp7cedrtvmd6p), I would like to call a vote to merge the YARN-321 branch for Generic Application History Server into trunk. *Scope of the changes* The changes enable ResourceManager to record the historic information of the application, the application attempt and the container in terms of events via a history writer. In addition, the changes setup up an application history server, which allows users to access the recorded information via RPC interface, web UI and REST APIs. *Details of development* - Development of the feature is tracked in the jira - https://issues.apache.org/jira/browse/YARN-321. - Development has been done in a separate branch - https://svn.apache.org/repos/asf/hadoop/common/branches/YARN-321. - The feature development involved about 35 subtasks. - The up-to-date design is posted at - https://issues.apache.org/jira/secure/attachment/12619638/Generic Application History - Design-20131219.pdfhttps://issues.apache.org/jira/secure/attachment/12619638/Generic%C2%A0Application%20History%20-%20Design-20131219.pdf - The uber merge patch Jira - https://issues.apache.org/jira/browse/YARN-1587 *Testing* A number of unit tests have been added as a part of the feature. In addition, we’ve also done end-to-end functional tests, and performance tests for HDFS-based history storage and history events processing. Last but not least, we have updated branch YARN-321 against the latest trunk, edited merge conflicts, fixed test failures caused by merge, and corrected a bunch of bad source code issues. The uber merge patch that contains all the diff between branch YARN-321 and trunk has been run through Jenkins. *Pending work* - Make it work in secure mode - Pending bug fixes We wish to merge the branch now instead of waiting for later. The main reason for this is that as the branch grew in size, the cost of its maintenance became huge. Once the feature is merged into trunk, we will continue to work on pending work like security stuff, to test and fix any bugs that may be found on the trunk, and to refactor the code about to share some pieces in PRC and web interfaces. *Release status* If the security stuff and the pending fixes arrive by the time everything else planned for Release 2.4 is done, we can include it as well. This is what we are striving for. Otherwise, we will call AHS not-feature-complete and not stable. The bulk of the design and implementation was done by Mayank Bansal and me with contributions from Devaraj K and Vinod Kumar Vavilapalli amongst others. Also, thanks to Robert Joseph Evans and Sandy Ryza for providing feedback on the design discussions. This vote runs for a week and closes on 1/24/2014 at 11:59 pm PT. Thanks, Zhijie -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Next releases
Thanks Suresh Colin. Please update the Roadmap wiki with your proposals. As always, we will try our best to get these in - but we can collectively decide to slip some of these to subsequent releases based on timelines. Arun On Dec 6, 2013, at 10:43 AM, Suresh Srinivas sur...@hortonworks.com wrote: Arun, I propose the following changes for 2.3: - There have been a lot of improvements related to supporting http policy. - There is a still discussion going on, but I would like to deprecate BackupNode in 2.3 as well. - We are currently working on rolling upgrades related change in HDFS. We might add a couple of changes that enables rolling upgrades from 2.3 onwards (hopefully we can this done by December) I propose the following for 2.4 release, if they are tested and stable: - Heterogeneous storage support - HDFS-2832 - Datanode cache related change - HDFS-4949 - HDFS ACLs - HDFS-4685 - Rolling upgrade changes Let me know if you want me to update the wiki. Regards, Suresh On Dec 6, 2013, at 12:27 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: If 2.4 is released in January, I think it's very unlikely to include symlinks. There is still a lot of work to be done before they're usable. You can look at the progress on HADOOP-10019. For some of the subtasks, it will require some community discussion before any code can be written. For better or worse, symlinks have not been requested by users as often as features like NFS export, HDFS caching, ACLs, etc, so effort has been focused on those instead. For now, I think we should put the symlinks-disabling patches (HADOOP-10020, etc) into branch-2, so that they will be part of the next releases without additional effort. I would like to see HDFS caching make it into 2.4. The APIs and implementation are beginning to stabilize, and around January it should be ok to backport to a stable branch. best, Colin On Thu, Nov 7, 2013 at 6:42 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, Thinking through the next couple of releases here, appreciate f/b. # hadoop-2.2.1 I was looking through commit logs and there is a *lot* of content here (81 commits as on 11/7). Some are features/improvements and some are fixes - it's really hard to distinguish what is important and what isn't. I propose we start with a blank slate (i.e. blow away branch-2.2 and start fresh from a copy of branch-2.2.0) and then be very careful and meticulous about including only *blocker* fixes in branch-2.2. So, most of the content here comes via the next minor release (i.e. hadoop-2.3) In future, we continue to be *very* parsimonious about what gets into a patch release (major.minor.patch) - in general, these should be only *blocker* fixes or key operational issues. # hadoop-2.3 I'd like to propose the following features for YARN/MR to make it into hadoop-2.3 and punt the rest to hadoop-2.4 and beyond: * Application History Server - This is happening in a branch and is close; with it we can provide a reasonable experience for new frameworks being built on top of YARN. * Bug-fixes in RM Restart * Minimal support for long-running applications (e.g. security) via YARN-896 * RM Fail-over via ZKFC * Anything else? HDFS??? Overall, I feel like we have a decent chance of rolling hadoop-2.3 by the end of the year. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure
Re: Next releases
Ok, I've updated https://wiki.apache.org/hadoop/Roadmap with a initial strawman list for hadoop-2.4 which I feel we can get out in Jan. What else would folks like to see? Please keep timeframe in mind. thanks, Arun On Dec 2, 2013, at 10:55 AM, Arun C Murthy a...@hortonworks.com wrote: On Nov 13, 2013, at 1:55 PM, Jason Lowe jl...@yahoo-inc.com wrote: +1 to limiting checkins of patch releases to Blockers/Criticals. If necessary committers check into trunk/branch-2 only and defer to the patch release manager for the patch release merge. Then there should be fewer surprises for everyone what ended up in a patch release and less likely the patch release becomes destabilized from the sheer amount of code churn. Maybe this won't be necessary if everyone understands that the patch release isn't the only way to get a change out in timely manner. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect that we only put in Blocker/Critical bugs into Point Releases. Committers, from now, please exercise extreme caution when committing to a point release: they should only be limited to Blocker bugs. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 0.23.10
+1 (binding) Verified bits and ran sample jobs. Thanks for driving this Thomas. Arun On Dec 2, 2013, at 10:22 PM, Thomas Graves tgra...@yahoo-inc.com wrote: Hey Everyone, There have been lots of improvements and bug fixes that have went into branch-0.23 since the 0.23.9 release. We think its time to do a 0.23.10 so I have created a release candidate (rc0) for a Hadoop-0.23.10 release. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.10-rc0/ The RC Tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.10-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days til December 9th. I am +1 (binding). thanks, Tom Graves -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Next releases
On Dec 2, 2013, at 10:31 AM, Arun C Murthy a...@hortonworks.com wrote: Ok, looks like there are no objections. I'm starting the work to rename 2.2.1 to 2.3 now. Committers, please hold commits till I send out the all clear. Done. I've renamed 2.3 - 2.4 and 2.2.1 - 2.3. I'll create the first RC for 2.3 a week from now i.e. 12/9. thanks, Arun thanks, Arun On Nov 20, 2013, at 6:38 AM, Arun C Murthy a...@hortonworks.com wrote: Jason, I'm glad to see we are converging. I'll update the Roadmap wiki with details about major/minor/patch releases. Here is a straight-forward approach for now: I'll just roll contents of branch-2.2 as a 2.3-rc0 candidate right-away. This way we don't have to get embroiled in details of individual patches (there are too many). Next up, I'll roll 2.4 in December. Thoughts? thanks, Arun On Nov 13, 2013, at 1:55 PM, Jason Lowe jl...@yahoo-inc.com wrote: I think a lot of confusion comes from the fact that the 2.x line is starting to mature. Before this there wasn't such a big contention of what went into patch vs. minor releases and often the lines were blurred between the two. However now we have significant customers and products starting to use 2.x as a base, which means we need to start treating it like we treat 1.x. That means getting serious about what we should put into a patch release vs. what we postpone to a minor release. Here's my $0.02 on recent proposals: +1 to releasing more often in general. A lot of the rush to put changes into a patch release is because it can be a very long time between any kind of release. If minor releases are more frequent then I hope there would be less of a need to rush something or hold up a release. +1 to limiting checkins of patch releases to Blockers/Criticals. If necessary committers check into trunk/branch-2 only and defer to the patch release manager for the patch release merge. Then there should be fewer surprises for everyone what ended up in a patch release and less likely the patch release becomes destabilized from the sheer amount of code churn. Maybe this won't be necessary if everyone understands that the patch release isn't the only way to get a change out in timely manner. As for 2.2.1, again I think it's expectations for what that release means. If it's really just a patch release then there shouldn't be features in it and tons of code churn, but I think many were treating it as the next vehicle to deliver changes in general. If we think 2.2.1 is just as good or better than 2.2.0 then let's wrap it up and move to a more disciplined approach for subsequent patch releases and more frequent minor releases. Jason On 11/13/2013 12:10 PM, Arun C Murthy wrote: On Nov 12, 2013, at 1:54 PM, Todd Lipcon t...@cloudera.com wrote: On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote: To be honest, I'm not aware of anything in 2.2.1 that shouldn't be there. However, I have only been following the HDFS and common side of things so I may not have the full picture. Arun, can you give a specific example of something you'd like to blow away? There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a patch release, plus things like: HADOOP-9623 Update jets3t dependency to 0.9.0 Having said that, the HDFS devs know their code the best. I agree with Colin. If we've been backporting things into a patch release (third version component) which don't belong, we should explicitly call out those patches, so we can learn from our mistakes and have a discussion about what belongs. Good point. Here is a straw man proposal: A patch (third version) release should only include *blocker* bugs which are critical from an operational, security or data-integrity issues. This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) is always release-able, and more importantly, deploy-able at any point in time. Sandy did bring up a related point about timing of releases and the urge for everyone to cram features/fixes into a dot release. So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message
Re: Next releases
On Nov 13, 2013, at 1:55 PM, Jason Lowe jl...@yahoo-inc.com wrote: +1 to limiting checkins of patch releases to Blockers/Criticals. If necessary committers check into trunk/branch-2 only and defer to the patch release manager for the patch release merge. Then there should be fewer surprises for everyone what ended up in a patch release and less likely the patch release becomes destabilized from the sheer amount of code churn. Maybe this won't be necessary if everyone understands that the patch release isn't the only way to get a change out in timely manner. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect that we only put in Blocker/Critical bugs into Point Releases. Committers, from now, please exercise extreme caution when committing to a point release: they should only be limited to Blocker bugs. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Next releases
Jason, I'm glad to see we are converging. I'll update the Roadmap wiki with details about major/minor/patch releases. Here is a straight-forward approach for now: I'll just roll contents of branch-2.2 as a 2.3-rc0 candidate right-away. This way we don't have to get embroiled in details of individual patches (there are too many). Next up, I'll roll 2.4 in December. Thoughts? thanks, Arun On Nov 13, 2013, at 1:55 PM, Jason Lowe jl...@yahoo-inc.com wrote: I think a lot of confusion comes from the fact that the 2.x line is starting to mature. Before this there wasn't such a big contention of what went into patch vs. minor releases and often the lines were blurred between the two. However now we have significant customers and products starting to use 2.x as a base, which means we need to start treating it like we treat 1.x. That means getting serious about what we should put into a patch release vs. what we postpone to a minor release. Here's my $0.02 on recent proposals: +1 to releasing more often in general. A lot of the rush to put changes into a patch release is because it can be a very long time between any kind of release. If minor releases are more frequent then I hope there would be less of a need to rush something or hold up a release. +1 to limiting checkins of patch releases to Blockers/Criticals. If necessary committers check into trunk/branch-2 only and defer to the patch release manager for the patch release merge. Then there should be fewer surprises for everyone what ended up in a patch release and less likely the patch release becomes destabilized from the sheer amount of code churn. Maybe this won't be necessary if everyone understands that the patch release isn't the only way to get a change out in timely manner. As for 2.2.1, again I think it's expectations for what that release means. If it's really just a patch release then there shouldn't be features in it and tons of code churn, but I think many were treating it as the next vehicle to deliver changes in general. If we think 2.2.1 is just as good or better than 2.2.0 then let's wrap it up and move to a more disciplined approach for subsequent patch releases and more frequent minor releases. Jason On 11/13/2013 12:10 PM, Arun C Murthy wrote: On Nov 12, 2013, at 1:54 PM, Todd Lipcon t...@cloudera.com wrote: On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote: To be honest, I'm not aware of anything in 2.2.1 that shouldn't be there. However, I have only been following the HDFS and common side of things so I may not have the full picture. Arun, can you give a specific example of something you'd like to blow away? There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a patch release, plus things like: HADOOP-9623 Update jets3t dependency to 0.9.0 Having said that, the HDFS devs know their code the best. I agree with Colin. If we've been backporting things into a patch release (third version component) which don't belong, we should explicitly call out those patches, so we can learn from our mistakes and have a discussion about what belongs. Good point. Here is a straw man proposal: A patch (third version) release should only include *blocker* bugs which are critical from an operational, security or data-integrity issues. This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) is always release-able, and more importantly, deploy-able at any point in time. Sandy did bring up a related point about timing of releases and the urge for everyone to cram features/fixes into a dot release. So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Next releases
On Nov 13, 2013, at 12:38 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Here are few patches that I put into 2.2.1 and are minimally invasive, but I don't think are blockers: YARN-305. Fair scheduler logs too many Node offered to app messages. YARN-1335. Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication YARN-1333. Support blacklisting in the Fair Scheduler YARN-1109. Demote NodeManager Sending out status for container logs to debug (haosdent via Sandy Ryza) YARN-1388. Fair Scheduler page always displays blank fair share +1 to doing releases at some fixed time interval. To be clear, I still think we should be *very* clear about what features we target for each release (2.3, 2.4, etc.). Except, we don't wait infinitely for any specific feature - if we miss a 4-6 week window a feature goes to the next train. Makes sense? thanks, Arun -Sandy On Wed, Nov 13, 2013 at 10:10 AM, Arun C Murthy a...@hortonworks.com wrote: On Nov 12, 2013, at 1:54 PM, Todd Lipcon t...@cloudera.com wrote: On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: To be honest, I'm not aware of anything in 2.2.1 that shouldn't be there. However, I have only been following the HDFS and common side of things so I may not have the full picture. Arun, can you give a specific example of something you'd like to blow away? There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a patch release, plus things like: HADOOP-9623 Update jets3t dependency to 0.9.0 Having said that, the HDFS devs know their code the best. I agree with Colin. If we've been backporting things into a patch release (third version component) which don't belong, we should explicitly call out those patches, so we can learn from our mistakes and have a discussion about what belongs. Good point. Here is a straw man proposal: A patch (third version) release should only include *blocker* bugs which are critical from an operational, security or data-integrity issues. This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) is always release-able, and more importantly, deploy-able at any point in time. Sandy did bring up a related point about timing of releases and the urge for everyone to cram features/fixes into a dot release. So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs. Thoughts? thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Next releases
Gang, Thinking through the next couple of releases here, appreciate f/b. # hadoop-2.2.1 I was looking through commit logs and there is a *lot* of content here (81 commits as on 11/7). Some are features/improvements and some are fixes - it's really hard to distinguish what is important and what isn't. I propose we start with a blank slate (i.e. blow away branch-2.2 and start fresh from a copy of branch-2.2.0) and then be very careful and meticulous about including only *blocker* fixes in branch-2.2. So, most of the content here comes via the next minor release (i.e. hadoop-2.3) In future, we continue to be *very* parsimonious about what gets into a patch release (major.minor.patch) - in general, these should be only *blocker* fixes or key operational issues. # hadoop-2.3 I'd like to propose the following features for YARN/MR to make it into hadoop-2.3 and punt the rest to hadoop-2.4 and beyond: * Application History Server - This is happening in a branch and is close; with it we can provide a reasonable experience for new frameworks being built on top of YARN. * Bug-fixes in RM Restart * Minimal support for long-running applications (e.g. security) via YARN-896 * RM Fail-over via ZKFC * Anything else? HDFS??? Overall, I feel like we have a decent chance of rolling hadoop-2.3 by the end of the year. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Managing docs with hadoop-1 hadoop-2
On Oct 18, 2013, at 2:17 PM, Eli Collins e...@cloudera.com wrote: On Fri, Oct 18, 2013 at 2:10 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, Currently http://hadoop.apache.org/docs/stable/ points to hadoop-1. With hadoop-2 going GA, should we just point that to hadoop-2? Couple of options: # Have stable1/stable2 links: http://hadoop.apache.org/docs/stable1 - hadoop-1.x http://hadoop.apache.org/docs/stable2 - hadoop-2.x +1, would also make: current - stable2(since v2 is the latest) stable - stable1 (for compatibility) Let's point stable - stable2 current to current2 (for e.g. 2.3 in future). This way we all look ahead. Makes sense? thanks, Arun Thanks, Eli # Just point stable to hadoop-2 and create something new for hadoop-1: http://hadoop.apache.org/docs/hadoop1 - hadoop-1.x http://hadoop.apache.org/docs/stable - hadoop-2.x We have similar requirements for *current* link too. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Managing docs with hadoop-1 hadoop-2
Folks, Currently http://hadoop.apache.org/docs/stable/ points to hadoop-1. With hadoop-2 going GA, should we just point that to hadoop-2? Couple of options: # Have stable1/stable2 links: http://hadoop.apache.org/docs/stable1 - hadoop-1.x http://hadoop.apache.org/docs/stable2 - hadoop-2.x # Just point stable to hadoop-2 and create something new for hadoop-1: http://hadoop.apache.org/docs/hadoop1 - hadoop-1.x http://hadoop.apache.org/docs/stable - hadoop-2.x We have similar requirements for *current* link too. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.2.0
Sorry, I've been away sick and hence the silence. I just started a discussion on the *-dev@ lists on Managing docs... once we all agree, I'll fix the links. Makes sense? thanks, Arun On Oct 17, 2013, at 11:56 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hello, The current document (http://hadoop.apache.org/docs/current/) is still 2.1.0-beta. Would you tell me how to update? common-dev@ I sent to wrong address 'hadoop-general@'. I'm sorry to send the same mail again. thanks, Akira (2013/10/17 11:44), Akira AJISAKA wrote: Hello, The current document (http://hadoop.apache.org/docs/current/) is still 2.1.0-beta. Would you tell me how to update? thanks, Akira (2013/10/16 13:34), Akira AJISAKA wrote: Congrats! The current document (http://hadoop.apache.org/docs/current/) is now hadoop-2.1.0-beta. I want someone to update. thanks, Akira (2013/10/15 21:35), Arun C Murthy wrote: With 31 +1s (15 binding) and no -1s the vote passes. Congratulations to all, Hadoop 2 is now GA! thanks, Arun On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.2.0
With 31 +1s (15 binding) and no -1s the vote passes. Congratulations to all, Hadoop 2 is now GA! thanks, Arun On Oct 7, 2013, at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.2.0
Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (HADOOP-10022) Add support for per project https support
[ https://issues.apache.org/jira/browse/HADOOP-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved HADOOP-10022. Resolution: Fixed All subtasks are done. Add support for per project https support - Key: HADOOP-10022 URL: https://issues.apache.org/jira/browse/HADOOP-10022 Project: Hadoop Common Issue Type: Bug Reporter: Suresh Srinivas Priority: Blocker Current configuration hadoop.https.enable turns on https only support for all the daemons in hadoop. This is an umbrella jira to add per project https configuration. For more details, see the detailed proposal - https://issues.apache.org/jira/browse/HADOOP-8581?focusedCommentId=13784332page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13784332 The current scope of work is described in - https://issues.apache.org/jira/browse/HADOOP-8581?focusedCommentId=13786567page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13786567 -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)
Looks like we are all clear now, I'll create an RC presently. Thanks everyone. Arun On Oct 1, 2013, at 8:59 PM, Arun C Murthy a...@hortonworks.com wrote: Yes, sorry if it wasn't clear. As others seem to agree, I think we'll be better getting a protocol/api stable GA done and then iterating on bugs etc. I'm not super worried about HADOOP-9984 since symlinks just made it to branch-2.1 recently. Currently we only have 2 blockers: HADOOP-9984 MAPREDUCE-5530. Both of which are PA and I've reviewed MR-5530 and is good to go (thanks Robert). Hopefully we can finish up HADOOP-9984 asap and we'll be good. thanks, Arun On Oct 1, 2013, at 4:53 PM, Alejandro Abdelnur t...@cloudera.com wrote: Arun, Does this mean that you want to skip a beta release and go straight to GA with the next release? thx On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy a...@hortonworks.com wrote: Guys, I took a look at the content in 2.1.2-beta so far, other than the critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR, there is fairly little content (unit tests fixes etc.) Furthermore, it's standing up well in testing too. Plus, the protocols look good for now (I wrote a gohadoop to try convince myself), let's lock them in. Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than make another 2.1.x release. This will drop a short-lived release (2.1.2) and help us move forward on 2.3 which has a fair bunch of content already... Thoughts? thanks, Arun On Sep 24, 2013, at 4:24 PM, Zhijie Shen zs...@hortonworks.com wrote: I've added MAPREDUCE-5531 to the blocker list. - Zhijie On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy a...@hortonworks.com wrote: With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll make it clear on the release page, that there are some known issues and that we will follow up very shortly with another release. Meanwhile, let's fix the remaining blockers (please mark them as such with Target Version 2.1.2-beta). The current blockers are here: http://s.apache.org/hadoop-2.1.2-beta-blockers thanks, Arun On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- Arun C
2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)
Guys, I took a look at the content in 2.1.2-beta so far, other than the critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR, there is fairly little content (unit tests fixes etc.) Furthermore, it's standing up well in testing too. Plus, the protocols look good for now (I wrote a gohadoop to try convince myself), let's lock them in. Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than make another 2.1.x release. This will drop a short-lived release (2.1.2) and help us move forward on 2.3 which has a fair bunch of content already... Thoughts? thanks, Arun On Sep 24, 2013, at 4:24 PM, Zhijie Shen zs...@hortonworks.com wrote: I've added MAPREDUCE-5531 to the blocker list. - Zhijie On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy a...@hortonworks.com wrote: With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll make it clear on the release page, that there are some known issues and that we will follow up very shortly with another release. Meanwhile, let's fix the remaining blockers (please mark them as such with Target Version 2.1.2-beta). The current blockers are here: http://s.apache.org/hadoop-2.1.2-beta-blockers thanks, Arun On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)
Yes, sorry if it wasn't clear. As others seem to agree, I think we'll be better getting a protocol/api stable GA done and then iterating on bugs etc. I'm not super worried about HADOOP-9984 since symlinks just made it to branch-2.1 recently. Currently we only have 2 blockers: HADOOP-9984 MAPREDUCE-5530. Both of which are PA and I've reviewed MR-5530 and is good to go (thanks Robert). Hopefully we can finish up HADOOP-9984 asap and we'll be good. thanks, Arun On Oct 1, 2013, at 4:53 PM, Alejandro Abdelnur t...@cloudera.com wrote: Arun, Does this mean that you want to skip a beta release and go straight to GA with the next release? thx On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy a...@hortonworks.com wrote: Guys, I took a look at the content in 2.1.2-beta so far, other than the critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR, there is fairly little content (unit tests fixes etc.) Furthermore, it's standing up well in testing too. Plus, the protocols look good for now (I wrote a gohadoop to try convince myself), let's lock them in. Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than make another 2.1.x release. This will drop a short-lived release (2.1.2) and help us move forward on 2.3 which has a fair bunch of content already... Thoughts? thanks, Arun On Sep 24, 2013, at 4:24 PM, Zhijie Shen zs...@hortonworks.com wrote: I've added MAPREDUCE-5531 to the blocker list. - Zhijie On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy a...@hortonworks.com wrote: With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll make it clear on the release page, that there are some known issues and that we will follow up very shortly with another release. Meanwhile, let's fix the remaining blockers (please mark them as such with Target Version 2.1.2-beta). The current blockers are here: http://s.apache.org/hadoop-2.1.2-beta-blockers thanks, Arun On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed
Re: Coverity Scan (MAPREDUCE-5032)
Agree with Aaron. Let's move this discussion to security@. Thanks. On Sep 30, 2013, at 5:57 PM, Aaron T. Myers a...@apache.org wrote: I strongly recommend that we take this conversation over to the (committers-only) secur...@hadoop.apache.org mailing list. In general we try to follow the Apache recommendations when it comes to addressing security issues, which involves not publicly disclosing the vulnerability until there are released version(s) with the issue(s) addressed. Best, Aaron On Mon, Aug 26, 2013 at 8:24 PM, Jon Jarboe jjar...@coverity.com wrote: Thanks for the interest. I'm in the process of building the 2.1.0 beta as suggested by Roman. Jon (214) 531-3496 -Original Message- From: Ottenheimer, Davi [mailto:davi.ottenhei...@emc.com] Sent: Monday, August 26, 2013 1:11 PM To: common-dev@hadoop.apache.org Subject: RE: Coverity Scan (MAPREDUCE-5032) Perhaps open the JIRA with only a reference/link to the Coverity report, and limit access to only those working on the issues. Full disclosure, update the JIRA, after fix. -- Davi Ottenheimer Senior Director of Trust EMC Corporation davi.ottenhei...@emc.com | @daviottenheimer | +1-415-271-6259 blog: http://www.flyingpenguin.com/ -Original Message- From: shaposh...@gmail.com [mailto:shaposh...@gmail.com] On Behalf Of Roman Shaposhnik Sent: Monday, August 26, 2013 10:50 AM To: common-dev@hadoop.apache.org Subject: Re: Coverity Scan (MAPREDUCE-5032) On Mon, Aug 26, 2013 at 10:43 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Can you file a JIRA and attach the report there? That is the best way to move this forward. Last time I was involved in a Coverity scan was when they scanned another project I'm committer on (FFmpeg). The lesson there was that the value you get out of browsing on their site https://scan.coverity.com is immeasurably higher than from any static report that can be attached to a JIRA. Also, at least in FFmpeg's case, Coverity identified a few things that could've been used as potential exploits so it made perfect sense to have a white-list of project members who could get access to the initial report instead of going all public with it to begin with (which would happen if it just gets attached to a JIRA in its entirety). Just my 2c worth of working with them in the past. Thanks, Roman. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
Done. On Sep 27, 2013, at 9:07 AM, Robert Kanter rkan...@cloudera.com wrote: Hi, When will the 2.1.1-beta jars be published in maven so downstream projects can start using them? I only see 2.1.0-beta. thanks - Robert On Tue, Sep 24, 2013 at 4:24 PM, Zhijie Shen zs...@hortonworks.com wrote: I've added MAPREDUCE-5531 to the blocker list. - Zhijie On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy a...@hortonworks.com wrote: With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll make it clear on the release page, that there are some known issues and that we will follow up very shortly with another release. Meanwhile, let's fix the remaining blockers (please mark them as such with Target Version 2.1.2-beta). The current blockers are here: http://s.apache.org/hadoop-2.1.2-beta-blockers thanks, Arun On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
Rather than spin another RC, let's get this out and follow up with the next release - especially since it's not clear how long it will take for the symlink stuff to sort itself out. Getting this out will help downstream projects, even if it does so in small way. Arun On Sep 23, 2013, at 5:36 PM, Alejandro Abdelnur t...@cloudera.com wrote: Vote for the 2.1.1-beta release is closing tonight, while we had quite a few +1s, it seems we need to address the following before doing a release: symlink discussion: get a concrete and explicit understanding on what we will do and in what release(s). Also, the following JIRAs seem nasty enough to require a new RC: https://issues.apache.org/jira/browse/HDFS-5225 (no patch avail) https://issues.apache.org/jira/browse/HDFS-5228 (patch avail) https://issues.apache.org/jira/browse/YARN-1089 (patch avail) https://issues.apache.org/jira/browse/MAPREDUCE-5529 (patch avail) I won't -1 the release but I'm un-casting my vote as I think we should address these things before. Thanks. Alejandro On Tue, Sep 24, 2013 at 1:49 AM, Suresh Srinivas sur...@hortonworks.comwrote: +1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: # Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. # Ran some tests such as sleep job, TestDFSIO, NNBench etc. On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
I've created 2.1.2-beta release version. Please use that for any *critical* commits on branch-2.1-beta branch. Please be careful, let's keep #commits here very small. thanks, Arun On Sep 24, 2013, at 2:07 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, That plan sounds good to me, thanks for being on top of things. What's the new fix version we should be using (2.1.2 or 2.2.0)? Would be good to get the same clarification regarding which branches should be receiving commits. I think a 2.1.2 would be nice to get the symlinks changes in a beta release pre-GA. I'd also like to add HADOOP-9761 to tucu's list of JIRAs, a symlink+viewfs regression that's mistakenly only in branch-2. Thanks, Andrew On Tue, Sep 24, 2013 at 1:39 PM, Arun C Murthy a...@hortonworks.com wrote: Rather than spin another RC, let's get this out and follow up with the next release - especially since it's not clear how long it will take for the symlink stuff to sort itself out. Getting this out will help downstream projects, even if it does so in small way. Arun On Sep 23, 2013, at 5:36 PM, Alejandro Abdelnur t...@cloudera.com wrote: Vote for the 2.1.1-beta release is closing tonight, while we had quite a few +1s, it seems we need to address the following before doing a release: symlink discussion: get a concrete and explicit understanding on what we will do and in what release(s). Also, the following JIRAs seem nasty enough to require a new RC: https://issues.apache.org/jira/browse/HDFS-5225 (no patch avail) https://issues.apache.org/jira/browse/HDFS-5228 (patch avail) https://issues.apache.org/jira/browse/YARN-1089 (patch avail) https://issues.apache.org/jira/browse/MAPREDUCE-5529 (patch avail) I won't -1 the release but I'm un-casting my vote as I think we should address these things before. Thanks. Alejandro On Tue, Sep 24, 2013 at 1:49 AM, Suresh Srinivas sur...@hortonworks.com wrote: +1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: # Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. # Ran some tests such as sleep job, TestDFSIO, NNBench etc. On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received
Re: windows support in trunk?
On Sep 24, 2013, at 3:24 PM, Andrey Klochkov akloch...@griddynamics.com wrote: Is Windows support in trunk currently? Or should I still use trunk-win to experiment with Hadoop on Windows? I've seen number of windows related patches going into trunk that's why I'm asking. Thanks! I know I just need to ask Chris Nauroth, but sending here just in case it's useful for others. *smile* Yes, FTR, Windows support is in trunk, branch-2 and branch-2.1. Arun -- Andrey Klochkov -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll make it clear on the release page, that there are some known issues and that we will follow up very shortly with another release. Meanwhile, let's fix the remaining blockers (please mark them as such with Target Version 2.1.2-beta). The current blockers are here: http://s.apache.org/hadoop-2.1.2-beta-blockers thanks, Arun On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: symlink support in Hadoop 2 GA
On Sep 16, 2013, at 6:49 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I wanted to broadcast plans for putting the FileSystem symlinks work (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think it's pretty important we get it in since it's not a compatible change; if it misses the GA train, we're not going to have symlinks until the next major release. Just catching up, is this an incompatible change, or not? The above reads 'not an incompatible change'. Arun However, we're still dealing with ongoing issues revealed via testing. There's user-code out there that only handles files and directories and will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912 for a nice example where globStatus returning symlinks broke Pig; some of us had a conference call to talk it through, and one definite conclusion was that this wasn't solvable in a generally compatible manner. There are also still some gaps in symlink support right now. For example, the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink resolution, and tooling like the FsShell and Distcp still need to be updated as well. So, there's definitely work to be done, but there are a lot of users interested in the feature, and symlinks really should be in GA. Would appreciate any thoughts/input on the matter. Thanks, Andrew -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.1.1-beta
Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-9944) RpcRequestHeaderProto defines callId as uint32 while ipc.Client.CONNECTION_CONTEXT_CALL_ID is signed (-3)
Arun C Murthy created HADOOP-9944: - Summary: RpcRequestHeaderProto defines callId as uint32 while ipc.Client.CONNECTION_CONTEXT_CALL_ID is signed (-3) Key: HADOOP-9944 URL: https://issues.apache.org/jira/browse/HADOOP-9944 Project: Hadoop Common Issue Type: Bug Reporter: Arun C Murthy Priority: Blocker RpcRequestHeaderProto defines callId as uint32 while ipc.Client.CONNECTION_CONTEXT_CALL_ID is signed (-3). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: hadoop-2.1.1-beta hadoop-2.2.0 (GA)
I'm planning on creating a branch-2.1.1-beta by EOD today. Ok? thanks, Arun On Aug 16, 2013, at 7:39 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, I spent time looking through changes slated for hadoop-2.1.1-beta and things look fairly contained (~10 or so changes for each of Common, HDFS, YARN MapReduce). Can I, henceforth, request committers to exercise large dollops of caution when committing to branch-2.1-beta? This way I hope we can quickly turn around to make a hadoop-2.1.1-beta release in the next couple of weeks. This can be followed by a bit more testing so that we are in a position to release hadoop-2.2.0 (GA/stable). In other words, I'm hoping hadoop-2.1.1-beta can be, um, the 'golden master' for the final hadoop-2 GA release. I don't mean to jinx it but saying it out loud, but I feel we could look at pushing out hadoop-2 GA by mid-September… there I said it! *smile* Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: pre-commit admin is fixed.
Thanks Giri! On Aug 25, 2013, at 12:16 AM, Giridharan Kesavan gkesa...@hortonworks.com wrote: Pre-commit Admin job on jenkins is fixed and back online -Giri -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
With 9 +1s including mine (6 binding) and no -1s, the vote passes. Thanks to all who voted. Arun On Aug 15, 2013, at 2:15 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: hadoop-2.1.1-beta hadoop-2.2.0 (GA)
FYI - here is the end-point I'm using to track blockers on 2.1.1-beta: http://s.apache.org/hadoop-2.1.1-beta-blockers Essentially, these are *Blocker* bugs with *Target Version* set to 2.1.1-beta. thanks, Arun On Aug 16, 2013, at 7:39 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, I spent time looking through changes slated for hadoop-2.1.1-beta and things look fairly contained (~10 or so changes for each of Common, HDFS, YARN MapReduce). Can I, henceforth, request committers to exercise large dollops of caution when committing to branch-2.1-beta? This way I hope we can quickly turn around to make a hadoop-2.1.1-beta release in the next couple of weeks. This can be followed by a bit more testing so that we are in a position to release hadoop-2.2.0 (GA/stable). In other words, I'm hoping hadoop-2.1.1-beta can be, um, the 'golden master' for the final hadoop-2 GA release. I don't mean to jinx it but saying it out loud, but I feel we could look at pushing out hadoop-2 GA by mid-September… there I said it! *smile* Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Thanks for the heads up Aaron, I've changed fix-version of HDFS-4763 to 2.1.1-beta for now. Committers - please be careful setting fix-versions, this is a good anti-pattern to avoid… though, I'm willing to bet a lot of dough that this isn't the first Hadoop release with this issue… *smile* Arun On Aug 20, 2013, at 6:09 PM, Aaron T. Myers a...@cloudera.com wrote: I was evaluating the release bits when I noticed that the change done in HDFS-4763 to add support for starting the HDFS NFSv3 gateway, which is marked with a fix version of 2.1.0-beta and included in the release notes of RC2, is not in fact included in the RC2 release bits. It looks to me like the change is included in branch-2.1-beta, but not branch-2.1.0-beta. Particularly since the release notes in RC2 are incorrect in claiming that this change is in this release, it seems like a pretty serious issue. Ordinarily I'd say that this issue should result in a new RC, and I would vote -1 on RC2. But, given the previous discussion that folks are interested in releasing 2.1.0-beta with several fairly substantial bugs that we already know about, I'll withhold my vote. If RC2 ends up getting released as-is, we should be sure to change the fix version field on that JIRA to be correct. -- Aaron T. Myers Software Engineer, Cloudera On Thu, Aug 15, 2013 at 2:15 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)
+1 (binding) Verified bits and ran examples on a 10-node cluster. Looks good. Arun On Aug 15, 2013, at 10:29 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1 The maven artifacts are available via repository.apache.org. The only difference between rc0 and rc1 is ASL added to releasenotes.html and updated release dates in CHANGES.txt files. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Help on running JUnit test cases on Hadoop-Common-trunk
Elizabeth, Welcome to Apache Hadoop! This seems like we either have broken or flaky tests (TestZKFailoverController). Looks like you are already engaged on HADOOP-9745. If you are feeling adventurous, we'd love to get your help in fixing them too. thanks, Arun On Aug 16, 2013, at 8:57 PM, Elizabeth Thomas email2el...@gmail.com wrote: Folks, I am getting started on the http://wiki.apache.org/hadoop/HowToContribute section of Hadoop. I attempted to run all the unit test cases on a fresh checked-out code from the hadoop common trunk (http://svn.apache.org/repos/asf/hadoop/common/trunk: revision - 1513368). But I am seem to be getting the following failures/errors. === Results : Failed tests: testGracefulFailoverFailBecomingActive(org.apache.hadoop.ha.TestZKFailoverController): Did not fail to graceful failover when target failed to become active! testGracefulFailoverFailBecomingStandby(org.apache.hadoop.ha.TestZKFailoverController): expected:1 but was:0 testGracefulFailoverFailBecomingStandbyAndFailFence(org.apache.hadoop.ha.TestZKFailoverController): Failover should have failed when old node wont fence testCachingRelaysResolveQueries(org.apache.hadoop.net.TestStaticMapping): Expected two entries in the map Mapping: cached switch mapping relaying to static mapping with single switch = false(..) testCachingCachesNegativeEntries(org.apache.hadoop.net.TestStaticMapping): Expected two entries in the map Mapping: cached switch mapping relaying to static mapping with single switch = false(..) testResolve(org.apache.hadoop.net.TestTableMapping): expected:/[rack1] but was:/[default-rack] testTableCaching(org.apache.hadoop.net.TestTableMapping): expected:/[rack1] but was:/[default-rack] testClearingCachedMappings(org.apache.hadoop.net.TestTableMapping): expected:/[rack1] but was:/[default-rack] testNormalizeHostName(org.apache.hadoop.net.TestNetUtils): expected:[67.215.65.145] but was:[UnknownHost123] Tests in error: testGracefulFailover(org.apache.hadoop.ha.TestZKFailoverController): test timed out after 25000 milliseconds testChgrp(org.apache.hadoop.fs.TestFsShellReturnCode): test timed out after 3 milliseconds Tests run: 2096, Failures: 9, Errors: 2, Skipped: 71 === Is there a pre-requisite for running these JUnit test cases? or any service to be up and running on my Ubuntu machine? As given in the contribute section, I would like to get the unit cases to run successfully before I start compiling my patch changes. Could anybody help here? Thanks in advance, Elizabeth -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Ok, I'll spin RC3. Can you please help get this in asap? Thanks, much appreciated! Arun On Aug 16, 2013, at 10:13 AM, Kihwal Lee kih...@yahoo-inc.com wrote: We have found HADOOP-9880, which prevents Namenode HA from running with security. Kihwal From: Arun C Murthy a...@hortonworks.com To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org Sent: Thursday, August 15, 2013 4:15 PM Subject: [VOTE] Release Apache Hadoop 2.1.0-beta Folks, I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Yep, there are quite a number of such fixes in 2.1.1 ATM, I think it will serve us better to get 2.1.0 out and then quickly turn around to make 2.1.1. My current plan is to start work on 2.1.1 right after this release gets complete… hopefully next week. thanks, Arun On Aug 16, 2013, at 4:36 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: There are other such isolated and well understood bug-fixes that we pushed to 2.1.1 in the interesting of making progress with 2.1.0 and the corresponding API changes. 2.1.1 should happen soon enough after this. Thanks, +Vinod On Aug 16, 2013, at 4:22 PM, Roman Shaposhnik wrote: What are the downsides of getting this fix into the 2.1? It appears that the fix is pretty isolated and well understood. Thoughts? Thanks, Roman. On Fri, Aug 16, 2013 at 3:04 PM, Kihwal Lee kih...@yahoo-inc.com wrote: I've changed the target version of HADOOP-9880 to 2.1.1. Please change it back, if you feel that it needs to be in 2.1.0-beta. Kihwal From: Kihwal Lee kih...@yahoo-inc.com To: Arun Murthy a...@hortonworks.com; common-dev@hadoop.apache.org common-dev@hadoop.apache.org Cc: mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org Sent: Friday, August 16, 2013 4:55 PM Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta It's your call, Arun. I.e. as long you believe rc2 meets the expectations and objectives of 2.1.0-beta. Kihwal From: Arun Murthy a...@hortonworks.com To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org Cc: Kihwal Lee kih...@yahoo-inc.com; mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org Sent: Friday, August 16, 2013 3:44 PM Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta That makes sense too. On Aug 16, 2013, at 10:39 AM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: We need to make a call on what blockers will be. From my limited understanding, this doesn't seem like a API or a compatibility issue. Can we not fix it in subsequent bug-fix releases? I do see a lot of follow up releases to 2.1.0. Getting this release out will help downstream projects start testing with all the API stuff that has already gone in 2.1.0. Thanks, +Vinod On Aug 16, 2013, at 10:13 AM, Kihwal Lee wrote: We have found HADOOP-9880, which prevents Namenode HA from running with security. Kihwal From: Arun C Murthy a...@hortonworks.com To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org Sent: Thursday, August 15, 2013 4:15 PM Subject: [VOTE] Release Apache Hadoop 2.1.0-beta Folks, I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use
hadoop-2.1.1-beta hadoop-2.2.0 (GA)
Gang, I spent time looking through changes slated for hadoop-2.1.1-beta and things look fairly contained (~10 or so changes for each of Common, HDFS, YARN MapReduce). Can I, henceforth, request committers to exercise large dollops of caution when committing to branch-2.1-beta? This way I hope we can quickly turn around to make a hadoop-2.1.1-beta release in the next couple of weeks. This can be followed by a bit more testing so that we are in a position to release hadoop-2.2.0 (GA/stable). In other words, I'm hoping hadoop-2.1.1-beta can be, um, the 'golden master' for the final hadoop-2 GA release. I don't mean to jinx it but saying it out loud, but I feel we could look at pushing out hadoop-2 GA by mid-September… there I said it! *smile* Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[VOTE] Release Apache Hadoop 2.1.0-beta
Folks, I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Ok, thanks for heads up Daryn. I'll spin an RC2 once HADOOP-9816 gets in - I'd appreciate if you could help push the fix in ASAP. Thanks again! Arun On Aug 1, 2013, at 9:38 AM, Daryn Sharp da...@yahoo-inc.com wrote: I broke RPC QOP for integrity and privacy options. :( See blocker HADOOP-9816. I think I understand the problem and it shouldn't be hard to fix. The bug went unnoticed because sadly there are no unit tests for the QOP options, even though it just involves a conf setting. Daryn On Jul 29, 2013, at 5:00 PM, Arun C Murthy wrote: Ok, I think we are close to rc1 now - the last of blockers should be committed later today… I'll try and spin RC1 tonight. thanks, Arun On Jul 21, 2013, at 12:43 AM, Devaraj Das d...@hortonworks.com wrote: I have just raised https://issues.apache.org/jira/browse/HDFS-5016 .. This bug can easily be reproduced by some HBase tests. I'd like this to be considered before we make a beta release. Have spoken about this with some hdfs folks offline and I am told that it is being worked on. Thanks Devaraj On Wed, Jul 17, 2013 at 4:25 PM, Alejandro Abdelnur tuc...@gmail.comwrote: As I've mentioned in my previous email, if we get YARN-701 in, we should also get in the fix for unmanaged AMs in an un-secure setup in 2.1.0-beta. Else is a regression of a functionality it is already working. Because of that, to avoid continuing delaying the release, I'm suggesting to mention in the release notes the API changes and behavior changes that YARN-918 and YARN-701 will bring into the next beta or GA release. thx On Wed, Jul 17, 2013 at 4:14 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: On Jul 17, 2013, at 1:04 PM, Alejandro Abdelnur wrote: * YARN-701 It should be addressed before a GA release. Still, as it is this breaks unmanaged AMs and to me that would be a blocker for the beta. YARN-701 and the unmanaged AMs fix should be committed in tandem. * YARN-918 It is a consequence of YARN-701 and depends on it. YARN-918 is an API change. And YARN-701 is a behaviour change. We need both in 2.1.0. * YARN-926 It would be nice to have it addressed before GA release. Either ways. I'd get it in sooner than later specifically when we are trying to replace the old API with the new one. Thanks, +Vino -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: Getting started
Welcome John! Depending on your interest you might want to sign up on HDFS, YARN or MapReduce here… also take a look at open liras with 'newbie' tag. Take a look at http://wiki.apache.org/hadoop/HowToContribute also. Arun On Jul 31, 2013, at 11:12 AM, John Chilton j...@johnchilton.com wrote: Hi all, My name is John. I have software engineering experience in a production environment and would like to get started working on hadoop. I have been told by a seasoned engineer that it takes about 6 months of full-time before anybody should be making un-mentored changes to a new project. That in mind, can anybody hold my hand for a few commits to get me rolling? Thanks, John Chilton -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
[jira] [Created] (HADOOP-9800) Fix up mvn src profile
Arun C Murthy created HADOOP-9800: - Summary: Fix up mvn src profile Key: HADOOP-9800 URL: https://issues.apache.org/jira/browse/HADOOP-9800 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.1.0-beta Reporter: Arun C Murthy Currently making a release has a bunch of manual steps documented at http://wiki.apache.org/hadoop/HowToReleasePostMavenization. We should fix up maven 'src' profile to copy releasenotes.html, CHANGES.txt etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[VOTE] Release Apache Hadoop 2.1.0-beta
Folks, I've created another release candidate (rc1) for hadoop-2.1.0-beta that I would like to get released. This RC fixes a number of issues reported on the previous candidate. This release represents a *huge* amount of work done by the community (~650 fixes) which includes several major advances including: # HDFS Snapshots # Windows support # YARN API stabilization # MapReduce Binary Compatibility with hadoop-1.x # Substantial amount of integration testing with rest of projects in the ecosystem The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: creating 2.2.0 version in JIRA
Sounds good. I'll re-create branch-2.1.0-beta from branch-2.1-beta when the last 2 blockers are in. thanks, Arun On Jul 10, 2013, at 10:56 AM, Alejandro Abdelnur t...@cloudera.com wrote: If that is the case, then I'll like to push the following JIRAs that have been committed to branch-2 to branch-2.1 when the first RC was just out and we didn't know that many more things would come in. I'm planning to push these JIRAs mid afternoon PST today. If there is any that should not make it, please speak up. Thanks. common: HADOOP-9661. Allow metrics sources to be extended. (sandyr via tucu) HADOOP-9370. Write FSWrapper class to wrap FileSystem and FileContext for better test coverage. (Andrew Wang via Colin Patrick McCabe) HADOOP-9355. Abstract symlink tests to use either FileContext or FileSystem. (Andrew Wang via Colin Patrick McCabe) HADOOP-9673. NetworkTopology: when a node can't be added, print out its location for diagnostic purposes. (Colin Patrick McCabe) HADOOP-9414. Refactor out FSLinkResolver and relevant helper methods. (Andrew Wang via Colin Patrick McCabe) HADOOP-9416. Add new symlink resolution methods in FileSystem and FileSystemLinkResolver. (Andrew Wang via Colin Patrick McCabe) hdfs: HDFS-4908. Reduce snapshot inode memory usage. (szetszwo) yarn: YARN-866. Add test for class ResourceWeights. (ywskycn via tucu) YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) YARN-883. Expose Fair Scheduler-specific queue metrics. (sandyr via tucu) mapreduce: MAPREDUCE-5333. Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities. (ywskycn via tucu) On Tue, Jul 9, 2013 at 10:54 AM, Arun C Murthy a...@hortonworks.com wrote: On Jul 2, 2013, at 3:54 PM, Alejandro Abdelnur t...@cloudera.com wrote: We need clarification on this then. I was under the impression that branch-2 would be 2.2.0. Sorry, I missed this thread - thanks to Jason for pointing me. As we discussed, the idea was that we are not adding new features to the the beta release (2.1.x-beta) so that we can focus on stabilizing it and releasing as hadoop-2.2.0 i.e. GA of hadoop-2. See http://s.apache.org/lZ8 . Hence, by default, new features goto branch-2 with fix-version as 2.3.x. Hope that makes sense. I'll fix branch-2 to set version to 2.3.0-SNAPSHOT to ease further confusion. thanks, Arun thx On Tue, Jul 2, 2013 at 2:38 PM, Jason Lowe jl...@yahoo-inc.com wrote: I thought Arun intends for 2.2.0 to be created off of branch-2.1.0-beta and not off of branch-2. As I understand it, only critical blockers will be the delta between 2.1.0-beta and 2.2.0 and items checked into branch-2 should be marked as fixed in 2.3.0. Part of the confusion is that currently branch-2 builds as 2.2.0-SNAPSHOT, but I believe Arun intended it to be 2.3.0-SNAPSHOT. Jason On 06/21/2013 12:05 PM, Alejandro Abdelnur wrote: Thanks Suresh, didn't know that, will do. On Fri, Jun 21, 2013 at 9:48 AM, Suresh Srinivas sur...@hortonworks.com wrote: I have added in to HDFS, HADOOP, MAPREDUCE projects. Can someone add it for YARN? On Fri, Jun 21, 2013 at 9:35 AM, Alejandro Abdelnur t...@cloudera.com wrote: When Arun created branch-2.1-beta he stated: The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet). I working/committing some JIRAs that I'm putting in branch-2 (testcases and improvements) but I don't want to put them in branch-2.1-beta as they are not critical and I don't won't add unnecessary noise to the branch-2.1-beta release work. Currently branch-2 POMs have a version 2.2.0 and the CHANGES.txt files as well. But because we did not create a JIRA version I cannot close those JIRAs. Can we please create the JIRA versions? later we can rename them. Thx -- Alejandro -- http://hortonworks.com/**download/ http://hortonworks.com/download/ -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: creating 2.2.0 version in JIRA
On Jul 2, 2013, at 3:54 PM, Alejandro Abdelnur t...@cloudera.com wrote: We need clarification on this then. I was under the impression that branch-2 would be 2.2.0. Sorry, I missed this thread - thanks to Jason for pointing me. As we discussed, the idea was that we are not adding new features to the the beta release (2.1.x-beta) so that we can focus on stabilizing it and releasing as hadoop-2.2.0 i.e. GA of hadoop-2. See http://s.apache.org/lZ8. Hence, by default, new features goto branch-2 with fix-version as 2.3.x. Hope that makes sense. I'll fix branch-2 to set version to 2.3.0-SNAPSHOT to ease further confusion. thanks, Arun thx On Tue, Jul 2, 2013 at 2:38 PM, Jason Lowe jl...@yahoo-inc.com wrote: I thought Arun intends for 2.2.0 to be created off of branch-2.1.0-beta and not off of branch-2. As I understand it, only critical blockers will be the delta between 2.1.0-beta and 2.2.0 and items checked into branch-2 should be marked as fixed in 2.3.0. Part of the confusion is that currently branch-2 builds as 2.2.0-SNAPSHOT, but I believe Arun intended it to be 2.3.0-SNAPSHOT. Jason On 06/21/2013 12:05 PM, Alejandro Abdelnur wrote: Thanks Suresh, didn't know that, will do. On Fri, Jun 21, 2013 at 9:48 AM, Suresh Srinivas sur...@hortonworks.com wrote: I have added in to HDFS, HADOOP, MAPREDUCE projects. Can someone add it for YARN? On Fri, Jun 21, 2013 at 9:35 AM, Alejandro Abdelnur t...@cloudera.com wrote: When Arun created branch-2.1-beta he stated: The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet). I working/committing some JIRAs that I'm putting in branch-2 (testcases and improvements) but I don't want to put them in branch-2.1-beta as they are not critical and I don't won't add unnecessary noise to the branch-2.1-beta release work. Currently branch-2 POMs have a version 2.2.0 and the CHANGES.txt files as well. But because we did not create a JIRA version I cannot close those JIRAs. Can we please create the JIRA versions? later we can rename them. Thx -- Alejandro -- http://hortonworks.com/**download/ http://hortonworks.com/download/ -- Alejandro -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Thanks Hitesh Roman, I'll roll RC1 once these are fixed. Arun On Jun 28, 2013, at 12:25 PM, Roman Shaposhnik r...@apache.org wrote: I think it would very useful if somebody more familiar with HDFS (Suresh?) could take a look at: https://issues.apache.org/jira/browse/HDFS-4940 Suresh was very helpful in unblocking the client side of things, but even after I followed his recommendations on the heap size, etc. The NN still OOMs it just takes longer. A quick look at the provided heap dump could help us understand whether this is something serious or not. This is the last bit that blocks the Bigtop side of things as far as 2.1 is concerned. Thanks, Roman. On Wed, Jun 26, 2013 at 1:17 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.0-beta that I would like to get released. This release represents a *huge* amount of work done by the community (639 fixes) which includes several major advances including: # HDFS Snapshots # Windows support # YARN API stabilization # MapReduce Binary Compatibility with hadoop-1.x # Substantial amount of integration testing with rest of projects in the ecosystem The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
