RE: Reminder: Apache committers have access to a free MSDN license
Thanks Chris :-) Regards, Yi Liu -Original Message- From: Chris Nauroth [mailto:cnaur...@hortonworks.com] Sent: Sunday, June 07, 2015 12:35 AM To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: Reminder: Apache committers have access to a free MSDN license If you are a committer on any Apache project (not just Hadoop), then you have access to a free MSDN license. The details are described here. https://svn.apache.org/repos/private/committers/donated-licenses/msdn-licen se-grants.txt You'll need to authenticate with your Apache credentials. This means that all Hadoop committers, and a large number of contributors who are also committers on other Apache projects, are empowered to review and test patches on Windows. After getting the free MSDN license, you can download the installation iso for Windows Server 2008 or 2010 and run it in a VirtualBox VM (or your hypervisor of choice). Instructions for setting up a Windows development environment have been in BUILDING.txt for a few years. This would prevent situations where patches are blocked from getting committed while waiting for me or any other individual to test. --Chris Nauroth
RE: Looking to a Hadoop 3 release
+1 Regards, Yi Liu -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: Tuesday, March 03, 2015 7:20 AM To: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: Looking to a Hadoop 3 release Hi devs, It's been a year and a half since 2.x went GA, and I think we're about due for a 3.x release. Notably, there are two incompatible changes I'd like to call out, that will have a tremendous positive impact for our users. First, classpath isolation being done at HADOOP-11656, which has been a long-standing request from many downstreams and Hadoop users. Second, bumping the source and target JDK version to JDK8 (related to HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months from now). In the past, we've had issues with our dependencies discontinuing support for old JDKs, so this will future-proof us. Between the two, we'll also have quite an opportunity to clean up and upgrade our dependencies, another common user and developer request. I'd like to propose that we start rolling a series of monthly-ish series of 3.0 alpha releases ASAP, with myself volunteering to take on the RM and other cat herding responsibilities. There are already quite a few changes slated for 3.0 besides the above (for instance the shell script rewrite) so there's already value in a 3.0 alpha, and the more time we give downstreams to integrate, the better. This opens up discussion about inclusion of other changes, but I'm hoping to freeze incompatible changes after maybe two alphas, do a beta (with no further incompat changes allowed), and then finally a 3.x GA. For those keeping track, that means a 3.x GA in about four months. I would also like to stress though that this is not intended to be a big bang release. For instance, it would be great if we could maintain wire compatibility between 2.x and 3.x, so rolling upgrades work. Keeping branch-2 and branch-3 similar also makes backports easier, since we're likely maintaining 2.x for a while yet. Please let me know any comments / concerns related to the above. If people are friendly to the idea, I'd like to cut a branch-3 and start working on the first alpha. Best, Andrew
RE: MRAppMaster running on a docker container failed to load openssl cipher.
Hi Chen He, If native is not available, JCE will be used. So you can ignore that debug info. Actually CryptoCodec is not necessary in org.apache.hadoop.fs.Hdfs, and I have created a JIRA to fix it. Regards, Yi Liu -Original Message- From: Chen He [mailto:airb...@gmail.com] Sent: Tuesday, December 16, 2014 12:31 PM To: common-dev@hadoop.apache.org Subject: MRAppMaster running on a docker container failed to load openssl cipher. Try to run teragen based on hadoop 2.6.0 using docker and met following error: 2014-12-15 04:15:21,385 DEBUG [main] org.apache.hadoop.crypto.OpensslCipher: Failed to load OpenSSL Cipher. java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl()Z at org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl(Native Method) at org.apache.hadoop.crypto.OpensslCipher.clinit(OpensslCipher.java:84) at org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.init(OpensslAesCtrCryptoCodec.java:50) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129) at org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:67) at org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:100) at org.apache.hadoop.fs.Hdfs.init(Hdfs.java:91) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:129) at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:157) at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:242) at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:334) at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331) at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448) at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.getDefaultFileContext(JobHistoryUtils.java:247) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.ensurePathInDefaultFileSystem(JobHistoryUtils.java:277) at org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.getConfiguredHistoryStagingDirPrefix(JobHistoryUtils.java:191) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:147) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:444) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1499) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429) 2014-12-15 04:15:21,390 DEBUG [main] org.apache.hadoop.util.PerformanceAdvisory: Crypto codec org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available. Regards! Chen
RE: [VOTE] Release Apache Hadoop 2.6.0
+1 (binding) - Built from source - Setup a security cluster with 4 nodes, and verified various fs operations. - Verified MapReduce examples in security mode. - Verify HDFS encryption at rest in security mode. Regards, Yi Liu -Original Message- From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Friday, November 14, 2014 7:08 AM To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: [VOTE] Release Apache Hadoop 2.6.0 Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: [VOTE] Release Apache Hadoop 2.6.0
Sorry, my vote should be non-binding. Regards, Yi Liu -Original Message- From: Liu, Yi A [mailto:yi.a@intel.com] Sent: Tuesday, November 18, 2014 10:03 AM To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: RE: [VOTE] Release Apache Hadoop 2.6.0 +1 (non-binding) - Built from source - Setup a security cluster with 4 nodes, and verified various fs operations. - Verified MapReduce examples in security mode. - Verify HDFS encryption at rest in security mode. Regards, Yi Liu -Original Message- From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Friday, November 14, 2014 7:08 AM To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: [VOTE] Release Apache Hadoop 2.6.0 Folks, I've created another release candidate (rc1) for hadoop-2.6.0 based on the feedback. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1 The RC tag in git is: release-2.6.0-rc1 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1013. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: [VOTE] Release Apache Hadoop 2.6.0
Thanks Chris for pushing it :) Regards, Yi Liu -Original Message- From: Chris Nauroth [mailto:cnaur...@hortonworks.com] Sent: Friday, November 14, 2014 4:52 AM To: common-dev@hadoop.apache.org Cc: yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; Ravi Prakash; hdfs-...@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.6.0 I have committed HDFS-7385 down through branch-2.6.0. Thank you! Chris Nauroth Hortonworks http://hortonworks.com/ On Thu, Nov 13, 2014 at 9:14 AM, Chris Nauroth cnaur...@hortonworks.com wrote: I'm helping to expedite a complete, approved patch for HDFS-7385 now. Then, we can make a final decision on its inclusion in 2.6.0. Thank you for bringing it up, Yi. Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, Nov 12, 2014 at 7:31 PM, Liu, Yi A yi.a@intel.com wrote: Arun, could you wait for HDFS-7385? It will cause issue of HDFS ACL and XAttrs in some case, the fix is very easy but I think the issue is critical. I'm helping review it, and expect to commit today. Thanks. Regards, Yi Liu -Original Message- From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Thursday, November 13, 2014 12:58 AM To: yarn-...@hadoop.apache.org Cc: mapreduce-...@hadoop.apache.org; Ravi Prakash; hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.6.0 Sounds good. I'll create an rc1. Thanks. Arun On Nov 11, 2014, at 2:06 PM, Robert Kanter rkan...@cloudera.com wrote: Hi Arun, We were testing the RC and ran into a problem with the recent fixes that were done for POODLE for Tomcat (HADOOP-11217 for KMS and HDFS-7274 for HttpFS). Basically, in disabling SSLv3, we also disabled SSLv2Hello, which is required for older clients (e.g. Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be clear, it does not mean SSLv2, which is insecure. This also affects the MR shuffle in HADOOP-11243. The fix is super simple, so I think we should reopen these 3 JIRAs and put in addendum patches and get them into 2.6.0. thanks - Robert On Tue, Nov 11, 2014 at 1:04 PM, Ravi Prakash ravi...@ymail.com wrote: Hi Arun! We are very close to completion on YARN-1964 (DockerContainerExecutor). I'd also like HDFS-4882 to be checked in. Do you think these issues merit another RC? ThanksRavi On Tuesday, November 11, 2014 11:57 AM, Steve Loughran ste...@hortonworks.com wrote: +1 binding -patched slider pom to build against 2.6.0 -verified build did download, which it did at up to ~8Mbps. Faster than a local build. -full clean test runs on OS/X Linux Windows 2012: Same thing. I did have to first build my own set of the windows native binaries, by checking out branch-2.6.0; doing a native build, copying the binaries and then purging the local m2 repository of hadoop artifacts to be confident I was building against. For anyone who wants those native libs they will be up on https://github.com/apache/incubator-slider/tree/develop/bin/window s/ once it syncs with the ASF repos. afterwords: the tests worked! On 11 November 2014 02:52, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.6.0 that I would like to see released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0 The RC tag in git is: release-2.6.0-rc0 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1012. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You
RE: [VOTE] Release Apache Hadoop 2.6.0
Arun, could you wait for HDFS-7385? It will cause issue of HDFS ACL and XAttrs in some case, the fix is very easy but I think the issue is critical. I'm helping review it, and expect to commit today. Thanks. Regards, Yi Liu -Original Message- From: Arun C Murthy [mailto:a...@hortonworks.com] Sent: Thursday, November 13, 2014 12:58 AM To: yarn-...@hadoop.apache.org Cc: mapreduce-...@hadoop.apache.org; Ravi Prakash; hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.6.0 Sounds good. I'll create an rc1. Thanks. Arun On Nov 11, 2014, at 2:06 PM, Robert Kanter rkan...@cloudera.com wrote: Hi Arun, We were testing the RC and ran into a problem with the recent fixes that were done for POODLE for Tomcat (HADOOP-11217 for KMS and HDFS-7274 for HttpFS). Basically, in disabling SSLv3, we also disabled SSLv2Hello, which is required for older clients (e.g. Java 6 with openssl 0.9.8x) so they can't connect without it. Just to be clear, it does not mean SSLv2, which is insecure. This also affects the MR shuffle in HADOOP-11243. The fix is super simple, so I think we should reopen these 3 JIRAs and put in addendum patches and get them into 2.6.0. thanks - Robert On Tue, Nov 11, 2014 at 1:04 PM, Ravi Prakash ravi...@ymail.com wrote: Hi Arun! We are very close to completion on YARN-1964 (DockerContainerExecutor). I'd also like HDFS-4882 to be checked in. Do you think these issues merit another RC? ThanksRavi On Tuesday, November 11, 2014 11:57 AM, Steve Loughran ste...@hortonworks.com wrote: +1 binding -patched slider pom to build against 2.6.0 -verified build did download, which it did at up to ~8Mbps. Faster than a local build. -full clean test runs on OS/X Linux Windows 2012: Same thing. I did have to first build my own set of the windows native binaries, by checking out branch-2.6.0; doing a native build, copying the binaries and then purging the local m2 repository of hadoop artifacts to be confident I was building against. For anyone who wants those native libs they will be up on https://github.com/apache/incubator-slider/tree/develop/bin/windows/ once it syncs with the ASF repos. afterwords: the tests worked! On 11 November 2014 02:52, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.6.0 that I would like to see released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0 The RC tag in git is: release-2.6.0-rc0 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1012. Please try the release and vote; the vote will run for the usual 5 days. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: [VOTE] Release Apache Hadoop 2.5.1 RC0
+1 (non-binding) * built from source tarball * Ran a pseudo-distributed cluster and Rand some MR example jobs * basic test on HttpFS and WebHDFS * basic test for XAttr. * checked CHANGES.txt Regards, Yi Liu -Original Message- From: Alejandro Abdelnur [mailto:t...@cloudera.com] Sent: Thursday, September 11, 2014 3:16 AM To: yarn-...@hadoop.apache.org Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.5.1 RC0 Thanks Karthik. +1. + verified MD5 for source tarball + verified signature for source tarball + successfully run apache-rat:check + checked CHANGES, LICENSE, README, NOTICE files. + built from source tarball + started pseudo cluster + run a couple of MR example jobs + basic test on HttpFS On Wed, Sep 10, 2014 at 10:10 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for reporting the mistake in the documentation, Akira. While it is good to fix it, I am not sure it is big enough to warrant another RC, particularly because 2.5.1 is very much 2.5.0 done right. I just updated the how-to-release wiki to capture this step in the release process, so we don't miss it in the future. On Mon, Sep 8, 2014 at 11:37 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -0 (non-binding) In the document, Apache Hadoop 2.5.1 is a minor release in the 2.x.y release line, buliding upon the previous stable release 2.4.1. Hadoop 2.5.1 is a point release. Filed HADOOP-11078 to track this. Regards, Akira (2014/09/09 0:51), Karthik Kambatla wrote: +1 (non-binding) Built the source tarball, brought up a pseudo-distributed cluster and ran a few MR jobs. Verified documentation and size of the binary tarball. On Fri, Sep 5, 2014 at 5:18 PM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (RC0) for Hadoop 2.5.1. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.1-RC0/ The RC git tag is release-2.5.1-RC0 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-101 0/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks Karthik -- Alejandro
RE: Git repo ready to use
Thanks Karthik for making Git repo ready. Regards, Yi Liu -Original Message- From: Karthik Kambatla [mailto:ka...@cloudera.com] Sent: Thursday, August 28, 2014 2:26 PM To: common-dev@hadoop.apache.org Subject: Re: Git repo ready to use We just got HADOOP-11001 in. If you have access, can you please try modifying the Jenkins jobs taking the patch on HADOOP-11001 into consideration. On Wed, Aug 27, 2014 at 4:38 PM, Ted Yu yuzhih...@gmail.com wrote: I have access. I can switch the repository if you think it is time to do so. On Wed, Aug 27, 2014 at 4:35 PM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for reporting it, Ted. We are aware of it - second follow-up item in my earlier email. Unfortunately, I don't have access to the builds to fix them and don't quite know the procedure to get access either. I am waiting for someone with access to help us out. On Wed, Aug 27, 2014 at 3:45 PM, Ted Yu yuzhih...@gmail.com wrote: Precommit builds are still using svn : https://builds.apache.org/job/PreCommit-HDFS-Build/configure https://builds.apache.org/job/PreCommit-YARN-Build/configure FYI On Wed, Aug 27, 2014 at 7:00 AM, Ted Yu yuzhih...@gmail.com wrote: Currently Jenkins builds still use subversion as source. Should Jenkins point to git ? Cheers On Wed, Aug 27, 2014 at 1:40 AM, Karthik Kambatla ka...@cloudera.com wrote: Oh.. a couple more things. The git commit hashes have changed and are different from what we had on our github. This might interfere with any build automations that folks have. Another follow-up item: email and JIRA integration On Wed, Aug 27, 2014 at 1:33 AM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I am very excited to let you know that the git repo is now writable. I committed a few changes (CHANGES.txt fixes and branching for 2.5.1) and everything looks good. Current status: 1. All branches have the same names, including trunk. 2. Force push is disabled on trunk, branch-2 and tags. 3. Even if you are experienced with git, take a look at https://wiki.apache.org/hadoop/HowToCommitWithGit . Particularly, let us avoid merge commits. Follow-up items: 1. Update rest of the wiki documentation 2. Update precommit Jenkins jobs and get HADOOP-11001 committed (reviews appreciated). Until this is done, the precommit jobs will run against our old svn repo. 3. git mirrors etc. to use the new repo instead of the old svn repo. Thanks again for your cooperation through the migration process. Please reach out to me (or the list) if you find anything missing or have suggestions. Cheers! Karthik
RE: [VOTE] Merge fs-encryption branch to trunk
+1 (non-binding) I involved in feature development and participated in JIRA reviews in this branch. With help from many committers/PMCs/contributors, this feature goes with good quality, I think it is ready for merge. Regards, Yi Liu -Original Message- From: Stephen Chu [mailto:s...@cloudera.com] Sent: Saturday, August 09, 2014 3:18 PM To: common-dev@hadoop.apache.org Cc: hdfs-...@hadoop.apache.org Subject: Re: [VOTE] Merge fs-encryption branch to trunk +1 (non-binding) I have been testing encryption in conjunction with the Hadoop KMS and right now the integration looks good to merge to trunk. I also tested on platforms with outdated openssl and no encryption configs to verify no regressions when users don't want to use this feature. Thanks to those who worked on this enhancement and fixed the bugs found in testing. Stephen On Fri, Aug 8, 2014 at 4:37 PM, Alejandro Abdelnur t...@cloudera.com wrote: +1 I've been following the work closely, specially on the crypto streams and key handling, and providing dev support as well. Kudos to Andrew, Yi and Charles for doing the bulk of the work. thx On Fri, Aug 8, 2014 at 2:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: I should add that this vote will run for the standard 7 days for a non-release vote, so will close at 12PM Pacific on August 15th. On Fri, Aug 8, 2014 at 11:45 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I'd like to call a vote to merge the fs-encryption branch to trunk. Development of this feature has been ongoing since March on HDFS-6134 and HADOOP-10150, totally approximately 50 commits. The fs-encryption branch introduces support for transparent, end-to-end encryption within an encryption zone. Each file stored within an encryption zone is automatically encrypted and decrypted with a unique key. These per-file keys are encrypted with an encryption key only accessible by the client, ensuring that only the client is able to decrypt sensitive data. Furthermore, there is support for native, hardware-accelerated AES encryption. For further details, please see the design doc on HDFS-6134. In terms of merge readiness, we've posted some successful consolidated patches to the JIRA for Jenkins runs. distcp and fs -cp support has also recently been completed, allowing users to securely copy encrypted files without first decrypting them. There is ongoing work to add support for WebHDFS, HttpFS, and other alternative access methods. Stephen Chu has also posted a test plan, and has already identified a few issues that have been fixed. Design and development of this feature was also a cross-company effort with many different contributors. I'd like to thank Charles Lamb, Yi Liu, Uma Maheswara Rao G, Colin McCabe, and Juan Yu for their code contributions and reviews. Alejandro Abdelnur was also instrumental, doing a lot of the design work and as well as writing most of the Hadoop Key Mangement Server (KMS). Finally, I'd like to thank everyone who gave feedback on the JIRAs. This includes Owen, Sanjay, Larry, Mike Y, ATM, Todd, Nicholas, and Andy, among others. With that, here's my +1 to merge this to trunk. Thanks, Andrew -- Alejandro
RE: [DISCUSS] Migrate from svn to git for source control?
+1, Git is more convenient for development and we can switch easily between branches and can have local branches. Regards, Yi Liu -Original Message- From: Karthik Kambatla [mailto:ka...@cloudera.com] Sent: Saturday, August 02, 2014 7:44 AM To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: [DISCUSS] Migrate from svn to git for source control? Hi folks, From what I hear, a lot of devs use the git mirror for development/reviews and use subversion primarily for checking code in. I was wondering if it would make more sense just to move to git. In addition to subjective liking of git, I see the following advantages in our workflow: 1. Feature branches - it becomes easier to work on them and keep rebasing against the latest trunk. 2. Cherry-picks between branches automatically ensures the exact same commit message and tracks the lineage as well. 3. When cutting new branches and/or updating maven versions etc., it allows doing all the work locally before pushing it to the main branch. 4. Opens us up to potentially using other code-review tools. (Gerrit?) 5. It is just more convenient. I am sure this was brought up before in different capacities. I believe the support for git in ASF is healthy now and several downstream projects have moved. Again, from what I hear, ASF INFRA folks make the migration process fairly easy. What do you all think? Thanks Karthik
