Re: [VOTE] Migration from subversion to git for version control
+1 (non-binding), I don't use subversion at all in creating patches. After the migration, we need to change test-patch.sh to use git. Thanks, Akira (2014/08/11 7:18), Mit Desai wrote: +1 non binding. -M!T On Aug 10, 2014 4:12 PM, Chen He airb...@gmail.com wrote: +1 (non-binding) On Sun, Aug 10, 2014 at 3:44 PM, Jason Lowe jl...@yahoo-inc.com.invalid wrote: +1 Jason On 08/08/2014 09:57 PM, Karthik Kambatla wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members.
Re: [VOTE] Migration from subversion to git for version control
+1 (non-binding) On Sat, Aug 9, 2014 at 8:27 AM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members. -- Thanks Devaraj K
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
+1 (non-binding), pending the document issue Akira found out + downloaded src tarball and did build + deployed two pseudo-distributed clusters + ran some sample mapreduce job on one cluster + ran basic distcp test between the two clusters Thanks. --Yongjun On Sun, Aug 10, 2014 at 9:35 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -1 (non-binding) + downloaded tarball and deployed HDFS-HA cluster + confirmed HDFS Rolling Upgrade from 2.4.0 + ran some sample jobs - documents are missing in tarball In 2.4.1 release, the documents were deployed at share/doc/hadoop/ . Would you please re-create a tarball with adding -Pdoc option? Thanks, Akira (2014/08/11 9:12), Chen He wrote: +1 (non-binding) download source code and build from it (OS is Fedora); deploy to single server; run word count and sleep job succeed. On Sun, Aug 10, 2014 at 6:00 PM, Jason Lowe jl...@yahoo-inc.com.invalid wrote: +1 (binding) - verified signatures and digests - built from source - deployed a single-node cluster - ran some sample jobs Jason On 08/06/2014 03:59 PM, Karthik Kambatla wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/ orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
+1 (non-binding), also pending doc issue Akira found - Brought up pseudo-distributed cluster from downloaded tarball - Ran basic XAttr, ACL, snapshot commands successfully Thanks, Stephen On Sun, Aug 10, 2014 at 11:52 PM, Yongjun Zhang yzh...@cloudera.com wrote: +1 (non-binding), pending the document issue Akira found out + downloaded src tarball and did build + deployed two pseudo-distributed clusters + ran some sample mapreduce job on one cluster + ran basic distcp test between the two clusters Thanks. --Yongjun On Sun, Aug 10, 2014 at 9:35 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: -1 (non-binding) + downloaded tarball and deployed HDFS-HA cluster + confirmed HDFS Rolling Upgrade from 2.4.0 + ran some sample jobs - documents are missing in tarball In 2.4.1 release, the documents were deployed at share/doc/hadoop/ . Would you please re-create a tarball with adding -Pdoc option? Thanks, Akira (2014/08/11 9:12), Chen He wrote: +1 (non-binding) download source code and build from it (OS is Fedora); deploy to single server; run word count and sleep job succeed. On Sun, Aug 10, 2014 at 6:00 PM, Jason Lowe jl...@yahoo-inc.com.invalid wrote: +1 (binding) - verified signatures and digests - built from source - deployed a single-node cluster - ran some sample jobs Jason On 08/06/2014 03:59 PM, Karthik Kambatla wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~ kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/ orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
+1 (non-binding) - downloaded source - verified signature - built and deployed a single-node cluster - ran some sample MapReduce jobs Thanks, Mit Desai On Aug 10, 2014 7:13 PM, Chen He airb...@gmail.com wrote: +1 (non-binding) download source code and build from it (OS is Fedora); deploy to single server; run word count and sleep job succeed. On Sun, Aug 10, 2014 at 6:00 PM, Jason Lowe jl...@yahoo-inc.com.invalid wrote: +1 (binding) - verified signatures and digests - built from source - deployed a single-node cluster - ran some sample jobs Jason On 08/06/2014 03:59 PM, Karthik Kambatla wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks
Re: [VOTE] Migration from subversion to git for version control
+1 (binding)
as this gets rolled out, I think we may also want define a branch naming
policy like feature/${JIRA}-text for features, as well as a policy for
deleting retired branches (tag the last commit then rm the branch)...then
we can cull some branches that are in SVN.
-steve
On 9 August 2014 03:57, Karthik Kambatla ka...@cloudera.com wrote:
I have put together this proposal based on recent discussion on this topic.
Please vote on the proposal. The vote runs for 7 days.
1. Migrate from subversion to git for version control.
2. Force-push to be disabled on trunk and branch-* branches. Applying
changes from any of trunk/branch-* to any of branch-* should be through
git cherry-pick -x.
3. Force-push on feature-branches is allowed. Before pulling in a
feature, the feature-branch should be rebased on latest trunk and the
changes applied to trunk through git rebase --onto or git cherry-pick
commit-range.
4. Every time a feature branch is rebased on trunk, a tag that
identifies the state before the rebase needs to be created (e.g.
tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
the feature is pulled into trunk and the tags are no longer useful.
5. The relevance/use of tags stay the same after the migration.
Thanks
Karthik
PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
vote and will be Lazy 2/3 majority of PMC members.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
-1 (non-binding). * Downloaded source and verified signature. * Built from source. * Ran tests and some MR jobs. * Ran RM-HA with manual failover mode. - Documents are missing. Good catch, Akira. As he mentioned, the documents including javadocs are missing from the binary tar ball. We should include them. It only includes following docs: $ find . -name *.html ./share/doc/hadoop/httpfs/dependency-analysis.html ./share/doc/hadoop/httpfs/UsingHttpTools.html ./share/doc/hadoop/httpfs/project-reports.html ./share/doc/hadoop/httpfs/ServerSetup.html ./share/doc/hadoop/httpfs/index.html ./share/hadoop/httpfs/tomcat/webapps/ROOT/index.html ./share/hadoop/hdfs/webapps/secondary/status.html ./share/hadoop/hdfs/webapps/secondary/index.html ./share/hadoop/hdfs/webapps/hdfs/dfshealth.html ./share/hadoop/hdfs/webapps/hdfs/explorer.html ./share/hadoop/hdfs/webapps/hdfs/index.html ./share/hadoop/hdfs/webapps/journal/index.html ./share/hadoop/hdfs/webapps/datanode/index.html ./share/hadoop/tools/sls/html/showSimulationTrace.html Karthik, could you create new tar ball with the documentations? Thanks, - Tsuyoshi On Thu, Aug 7, 2014 at 5:59 AM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks -- - Tsuyoshi
Re: [VOTE] Migration from subversion to git for version control
+1(non-binding) Thanks, - Tsuyoshi On Sat, Aug 9, 2014 at 11:57 AM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members. -- - Tsuyoshi
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
Thanks Akira for catching the missing docs. Let me work on *updating* the binary tarball to include the docs *without* modifying the source. Will send an update as soon as I sort that out. The vote itself is on the source and the binary is just a convenience. I hope we can let the vote continue so long as we don't change the source. Thanks Karthik On Mon, Aug 11, 2014 at 5:51 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: -1 (non-binding). * Downloaded source and verified signature. * Built from source. * Ran tests and some MR jobs. * Ran RM-HA with manual failover mode. - Documents are missing. Good catch, Akira. As he mentioned, the documents including javadocs are missing from the binary tar ball. We should include them. It only includes following docs: $ find . -name *.html ./share/doc/hadoop/httpfs/dependency-analysis.html ./share/doc/hadoop/httpfs/UsingHttpTools.html ./share/doc/hadoop/httpfs/project-reports.html ./share/doc/hadoop/httpfs/ServerSetup.html ./share/doc/hadoop/httpfs/index.html ./share/hadoop/httpfs/tomcat/webapps/ROOT/index.html ./share/hadoop/hdfs/webapps/secondary/status.html ./share/hadoop/hdfs/webapps/secondary/index.html ./share/hadoop/hdfs/webapps/hdfs/dfshealth.html ./share/hadoop/hdfs/webapps/hdfs/explorer.html ./share/hadoop/hdfs/webapps/hdfs/index.html ./share/hadoop/hdfs/webapps/journal/index.html ./share/hadoop/hdfs/webapps/datanode/index.html ./share/hadoop/tools/sls/html/showSimulationTrace.html Karthik, could you create new tar ball with the documentations? Thanks, - Tsuyoshi On Thu, Aug 7, 2014 at 5:59 AM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks -- - Tsuyoshi
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
Thank you for dealing with this problem, Karthik. It sounds reasonable to me to update binary tarball without modifying source code. +1(non-binding) to continue the voting. Thanks, - Tsuyoshi On Tue, Aug 12, 2014 at 3:47 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks Akira for catching the missing docs. Let me work on *updating* the binary tarball to include the docs *without* modifying the source. Will send an update as soon as I sort that out. The vote itself is on the source and the binary is just a convenience. I hope we can let the vote continue so long as we don't change the source. Thanks Karthik On Mon, Aug 11, 2014 at 5:51 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: -1 (non-binding). * Downloaded source and verified signature. * Built from source. * Ran tests and some MR jobs. * Ran RM-HA with manual failover mode. - Documents are missing. Good catch, Akira. As he mentioned, the documents including javadocs are missing from the binary tar ball. We should include them. It only includes following docs: $ find . -name *.html ./share/doc/hadoop/httpfs/dependency-analysis.html ./share/doc/hadoop/httpfs/UsingHttpTools.html ./share/doc/hadoop/httpfs/project-reports.html ./share/doc/hadoop/httpfs/ServerSetup.html ./share/doc/hadoop/httpfs/index.html ./share/hadoop/httpfs/tomcat/webapps/ROOT/index.html ./share/hadoop/hdfs/webapps/secondary/status.html ./share/hadoop/hdfs/webapps/secondary/index.html ./share/hadoop/hdfs/webapps/hdfs/dfshealth.html ./share/hadoop/hdfs/webapps/hdfs/explorer.html ./share/hadoop/hdfs/webapps/hdfs/index.html ./share/hadoop/hdfs/webapps/journal/index.html ./share/hadoop/hdfs/webapps/datanode/index.html ./share/hadoop/tools/sls/html/showSimulationTrace.html Karthik, could you create new tar ball with the documentations? Thanks, - Tsuyoshi On Thu, Aug 7, 2014 at 5:59 AM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks -- - Tsuyoshi -- - Tsuyoshi
Re: [VOTE] Migration from subversion to git for version control
+1 (binding) Kihwal On 8/8/14, 9:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members.
Re: [VOTE] Migration from subversion to git for version control
+1 On Mon, Aug 11, 2014 at 1:43 PM, Kihwal Lee kih...@yahoo-inc.com.invalid wrote: +1 (binding) Kihwal On 8/8/14, 9:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Migration from subversion to git for version control
+1. best, Colin On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members.
Re: [VOTE] Migration from subversion to git for version control
+1 non-binding On Mon, Aug 11, 2014 at 4:16 PM, jgho...@gmail.com wrote: +1 From: Colin McCabe Sent: Monday, August 11, 2014 3:40 PM To: Hadoop Common Cc: hdfs-...@hadoop.apache.org, yarn-...@hadoop.apache.org, mapreduce-...@hadoop.apache.org +1. best, Colin On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-10956) Fix create-release script to include docs in the binary
Karthik Kambatla created HADOOP-10956: - Summary: Fix create-release script to include docs in the binary Key: HADOOP-10956 URL: https://issues.apache.org/jira/browse/HADOOP-10956 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 2.5.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Blocker The create-release script doesn't include docs in the binary tarball. We should fix that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10958) TestGlobPaths should do more tests of globbing by unprivileged users
Colin Patrick McCabe created HADOOP-10958: - Summary: TestGlobPaths should do more tests of globbing by unprivileged users Key: HADOOP-10958 URL: https://issues.apache.org/jira/browse/HADOOP-10958 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.3.0 Reporter: Colin Patrick McCabe TestGlobPaths should do more tests of globbing by unprivileged users. Right now, most of the tests are of globbing by the superuser, but this tends to hide permission exception issues such as HADOOP-10957. We should keep a few tests operating with privileged globs, but do most of them unprivileged. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10959) A Complement and Short Term Solution to TokenAuth Based on Kerberos Pre-Authentication Framework
Kai Zheng created HADOOP-10959: -- Summary: A Complement and Short Term Solution to TokenAuth Based on Kerberos Pre-Authentication Framework Key: HADOOP-10959 URL: https://issues.apache.org/jira/browse/HADOOP-10959 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng To implement and integrate pluggable authentication providers, enhance desirable single sign on for end users, and help enforce centralized access control on the platform, the community has widely discussed and concluded token based authentication could be the appropriate approach. TokenAuth (HADOOP-9392) was proposed and is under development to implement another Authentication Method in lieu with Simple and Kerberos. It is a big and long term effort to support TokenAuth across the entire ecosystem. We here propose a short term replacement based on Kerberos that can complement to TokenAuth. Our solution involves less codes changes with limited risk and the main development work has already been done in our POC. Users can use our solution as a short term solution to support token inside Hadoop. This effort and resultant solution will be fully described in the design document to be attached soon. Below is the brief introduction. We proposed to add token-preauth mechanism similar to PKINIT and OTP for Kerberos based on the Pre-Authentication framework, which allows users to authenticate to KDC using a JWT token instead of password. KDC authenticates the JWT token and issues TGT as it would trust the token authority/issuer via PKI mechanism. The proposal was submitted to Kerberos and IETF Kitten WG and they’re interested. Currently we’re collaborating with MIT team to work on the draft and standardize the mechanism. We also did a POC which implemented the token-preauth mechanism as a MIT Kerberos plugin. The plugin can be separately packaged as a Linux .so module and deployed additionally for existing installations. MIT also wish we could contribute the codes and make it available in their future releases. Before that we can make the plugin binary and source codes available to the community for experimental usage and review. So ideally token-preauth plugin can be deployed to a MIT Kerberos installation, the end users can authenticate to 3rd party JWT token authorities and get tokens, and then use the tokens to acquire Kerberos TGT from KDC. Based on that, we implemented the token authentication for Hadoop, with only a few of central modifications into the code base, as we don’t have to add another Authentication Method and the solution leverages the existing Kerberos support. We added KrbTokenLoginModule that extends the Krb5LoginModule and adds to support logging in using a token or token cache. The new module is compatible with Krb5LoginModule in configuration and functionality, thus can be used safely. We also added KerberosTokenAuthenticationHandler to support Hadoop web interfaces. It extends KerberosAuthenticationHandler and adds to support token authentication and perform the SPNEGO negotiation purely in server side in the new handler. Again the new handler is compatible with KerberosAuthenticationHandler and can be used safely. Token is used to exchange Kerberos ticket and ticket goes to Hadoop services as normally does. In addition to that, to employ the token attributes to enforce fine-grained authorization or whatever, a token derivation is encapsulated into ticket as Authorization data when KDC issues the ticket with the token. Then in service (Hadoop services) side, token can be queried and extracted from service ticket. We made this happen in both GSSAPI and SASL contexts as the both are used in Hadoop. As we can see or think of, the main concern for this solution may be that it requires to deploy additional plugin for existing Kerberos installations, and involves necessary identity accounts sync from identity management systems to Kerberos KDC. Most importantly, it requires Kerberos deployment as its prerequisite setup. We’re also discussing with MIT team about how to simplify Kerberos deployment especially for Hadoop large clusters and alleviate the overhead to employ PKINIT/token-preauth mechanisms like identity sync. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10960) hadoop cause system crash with “soft lock” and “hard lock”
linbao111 created HADOOP-10960: -- Summary: hadoop cause system crash with “soft lock” and “hard lock” Key: HADOOP-10960 URL: https://issues.apache.org/jira/browse/HADOOP-10960 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.2.0 Environment: redhat rhel 6.3,6,4,6.5 jdk1.7.0_45 hadoop2.2 Reporter: linbao111 Priority: Critical I am running hadoop2.2 on redhat6.3-6.5,and all of my machines crashed after a while. /var/log/messages shows repeatedly: Aug 11 06:30:42 jn4_73_128 kernel: BUG: soft lockup - CPU#1 stuck for 67s! [jsvc:11508] Aug 11 06:30:42 jn4_73_128 kernel: Modules linked in: bridge stp llc iptable_filter ip_tables mptctl mptbase xfs exportfs power_meter microcode dcdbas serio_raw iTCO_w dt iTCO_vendor_support i7core_edac edac_core sg bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif wmi mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_m od [last unloaded: scsi_wait_scan] Aug 11 06:30:42 jn4_73_128 kernel: CPU 1 Aug 11 06:30:42 jn4_73_128 kernel: Modules linked in: bridge stp llc iptable_filter ip_tables mptctl mptbase xfs exportfs power_meter microcode dcdbas serio_raw iTCO_w dt iTCO_vendor_support i7core_edac edac_core sg bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif wmi mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_m od [last unloaded: scsi_wait_scan] Aug 11 06:30:42 jn4_73_128 kernel: Aug 11 06:30:42 jn4_73_128 kernel: Pid: 11508, comm: jsvc Tainted: GW ---2.6.32-279.el6.x86_64 #1 Dell Inc. PowerEdge R510/084YMW Aug 11 06:30:42 jn4_73_128 kernel: RIP: 0010:[8104d088] [8104d088] wait_for_rqlock+0x28/0x40 Aug 11 06:30:42 jn4_73_128 kernel: RSP: 0018:8807786c3ee8 EFLAGS: 0202 Aug 11 06:30:42 jn4_73_128 kernel: RAX: f6e9f6e1 RBX: 8807786c3ee8 RCX: 880028216680 Aug 11 06:30:42 jn4_73_128 kernel: RDX: f6e9 RSI: 88061cd29370 RDI: 0286 Aug 11 06:30:42 jn4_73_128 kernel: RBP: 8100bc0e R08: 0001 R09: 0001 Aug 11 06:30:42 jn4_73_128 kernel: R10: R11: R12: 0286 Aug 11 06:30:42 jn4_73_128 kernel: R13: 8807786c3eb8 R14: 810e0f6e R15: 8807786c3e48 Aug 11 06:30:42 jn4_73_128 kernel: FS: () GS:88002820() knlGS: Aug 11 06:30:42 jn4_73_128 kernel: CS: 0010 DS: ES: CR0: 80050033 Aug 11 06:30:42 jn4_73_128 kernel: CR2: 00e5bd70 CR3: 01a85000 CR4: 06e0 Aug 11 06:30:42 jn4_73_128 kernel: DR0: DR1: DR2: Aug 11 06:30:42 jn4_73_128 kernel: DR3: DR6: 0ff0 DR7: 0400 Aug 11 06:30:42 jn4_73_128 kernel: Process jsvc (pid: 11508, threadinfo 8807786c2000, task 880c1def3500) Aug 11 06:30:42 jn4_73_128 kernel: Stack: Aug 11 06:30:42 jn4_73_128 kernel: 8807786c3f68 8107091b 8807786c3f28 Aug 11 06:30:42 jn4_73_128 kernel: d 880701735260 880c1def39c8 880c1def39c8 Aug 11 06:30:42 jn4_73_128 kernel: d 8807786c3f28 8807786c3f28 8807786c3f78 7f092d0ad700 Aug 11 06:30:42 jn4_73_128 kernel: Call Trace: Aug 11 06:30:42 jn4_73_128 kernel: [8107091b] ? do_exit+0x5ab/0x870 Aug 11 06:30:42 jn4_73_128 kernel: [81070ce7] ? sys_exit+0x17/0x20 Aug 11 06:30:42 jn4_73_128 kernel: [8100b0f2] ? system_call_fastpath+0x16/0x1b Aug 11 06:30:42 jn4_73_128 kernel: Code: ff ff 90 55 48 89 e5 0f 1f 44 00 00 48 c7 c0 80 66 01 00 65 48 8b 0c 25 b0 e0 00 00 0f ae f0 48 01 c1 eb 09 0f 1f 80 00 00 00 00 f3 90 8b 01 89 c2 c1 fa 10 66 39 c2 75 f2 c9 c3 0f 1f 84 00 00 Aug 11 06:30:42 jn4_73_128 kernel: Call Trace: Aug 11 06:30:42 jn4_73_128 kernel: [8107091b] ? do_exit+0x5ab/0x870 Aug 11 06:30:42 jn4_73_128 kernel: [81070ce7] ? sys_exit+0x17/0x20 Aug 11 06:30:42 jn4_73_128 kernel: [8100b0f2] ? system_call_fastpath+0x16/0x1b /em and finally crashed crash /usr/lib/debug/lib/modules/2.6.32-431.5.1.el6.x86_64/vmlinux /opt/crash/127.0.0.1-2014-08-10-09\:47\:38/vmcore crash 6.1.0-5.el6 Copyright (C) 2002-2012 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter help copying to see the conditions. This program has absolutely no warranty. Enter help
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
I have updated the binary tar ball to include the docs, by building the docs locally and copying them over. Filed HADOOP-10956 to fix the create-release script to handle this. The RC is here: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ Please note that the binary tar ball is signed by a new gpg key, so please re-import keys. (I lost my original private key). The source tar ball, signature and checksum are untouched. On Mon, Aug 11, 2014 at 1:03 PM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Thank you for dealing with this problem, Karthik. It sounds reasonable to me to update binary tarball without modifying source code. +1(non-binding) to continue the voting. Thanks, - Tsuyoshi On Tue, Aug 12, 2014 at 3:47 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks Akira for catching the missing docs. Let me work on *updating* the binary tarball to include the docs *without* modifying the source. Will send an update as soon as I sort that out. The vote itself is on the source and the binary is just a convenience. I hope we can let the vote continue so long as we don't change the source. Thanks Karthik On Mon, Aug 11, 2014 at 5:51 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: -1 (non-binding). * Downloaded source and verified signature. * Built from source. * Ran tests and some MR jobs. * Ran RM-HA with manual failover mode. - Documents are missing. Good catch, Akira. As he mentioned, the documents including javadocs are missing from the binary tar ball. We should include them. It only includes following docs: $ find . -name *.html ./share/doc/hadoop/httpfs/dependency-analysis.html ./share/doc/hadoop/httpfs/UsingHttpTools.html ./share/doc/hadoop/httpfs/project-reports.html ./share/doc/hadoop/httpfs/ServerSetup.html ./share/doc/hadoop/httpfs/index.html ./share/hadoop/httpfs/tomcat/webapps/ROOT/index.html ./share/hadoop/hdfs/webapps/secondary/status.html ./share/hadoop/hdfs/webapps/secondary/index.html ./share/hadoop/hdfs/webapps/hdfs/dfshealth.html ./share/hadoop/hdfs/webapps/hdfs/explorer.html ./share/hadoop/hdfs/webapps/hdfs/index.html ./share/hadoop/hdfs/webapps/journal/index.html ./share/hadoop/hdfs/webapps/datanode/index.html ./share/hadoop/tools/sls/html/showSimulationTrace.html Karthik, could you create new tar ball with the documentations? Thanks, - Tsuyoshi On Thu, Aug 7, 2014 at 5:59 AM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks -- - Tsuyoshi -- - Tsuyoshi
Re: [VOTE] Release Apache Hadoop 2.5.0 RC2
Can someone please verify the signatures on the new binary and the old source tarballs to make sure it is all good? If it is, I believe we can go ahead and close the vote. On Mon, Aug 11, 2014 at 9:49 PM, Karthik Kambatla ka...@cloudera.com wrote: I have updated the binary tar ball to include the docs, by building the docs locally and copying them over. Filed HADOOP-10956 to fix the create-release script to handle this. The RC is here: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ Please note that the binary tar ball is signed by a new gpg key, so please re-import keys. (I lost my original private key). The source tar ball, signature and checksum are untouched. On Mon, Aug 11, 2014 at 1:03 PM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Thank you for dealing with this problem, Karthik. It sounds reasonable to me to update binary tarball without modifying source code. +1(non-binding) to continue the voting. Thanks, - Tsuyoshi On Tue, Aug 12, 2014 at 3:47 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks Akira for catching the missing docs. Let me work on *updating* the binary tarball to include the docs *without* modifying the source. Will send an update as soon as I sort that out. The vote itself is on the source and the binary is just a convenience. I hope we can let the vote continue so long as we don't change the source. Thanks Karthik On Mon, Aug 11, 2014 at 5:51 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: -1 (non-binding). * Downloaded source and verified signature. * Built from source. * Ran tests and some MR jobs. * Ran RM-HA with manual failover mode. - Documents are missing. Good catch, Akira. As he mentioned, the documents including javadocs are missing from the binary tar ball. We should include them. It only includes following docs: $ find . -name *.html ./share/doc/hadoop/httpfs/dependency-analysis.html ./share/doc/hadoop/httpfs/UsingHttpTools.html ./share/doc/hadoop/httpfs/project-reports.html ./share/doc/hadoop/httpfs/ServerSetup.html ./share/doc/hadoop/httpfs/index.html ./share/hadoop/httpfs/tomcat/webapps/ROOT/index.html ./share/hadoop/hdfs/webapps/secondary/status.html ./share/hadoop/hdfs/webapps/secondary/index.html ./share/hadoop/hdfs/webapps/hdfs/dfshealth.html ./share/hadoop/hdfs/webapps/hdfs/explorer.html ./share/hadoop/hdfs/webapps/hdfs/index.html ./share/hadoop/hdfs/webapps/journal/index.html ./share/hadoop/hdfs/webapps/datanode/index.html ./share/hadoop/tools/sls/html/showSimulationTrace.html Karthik, could you create new tar ball with the documentations? Thanks, - Tsuyoshi On Thu, Aug 7, 2014 at 5:59 AM, Karthik Kambatla ka...@cloudera.com wrote: Hi folks, I have put together a release candidate (rc2) for Hadoop 2.5.0. The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/ The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/ The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1009/ You can find my public key at: http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS Please try the release and vote. The vote will run for the now usual 5 days. Thanks -- - Tsuyoshi -- - Tsuyoshi
